I don't believe that to be true. That is I think that certain materials will run out, meaning less of some stuff, replacement of other stuff, but I don't believe that there's any underlying 'ecological reality' that forces us to accept that your average westerner will have to live at the standard of living of the average person in Bolivia, and that there is little scope for improvement for the latter in the medium term. In the same way that I don't believe it when certain right wingers say the same thing, arguing from 'economic realities'. Again, there are such things as economic constraints, but socio-economic systems are primarily a product of politics, power, technology, and values, rather than the result of some 'natural' economic or ecological order.

As for the rest - no, the kind of urban gardening where people grow some vegetables that they couldn't otherwise get to supplement whatever starches that provide the bulk of their calories doesn't require anywhere near a full time commitment. Nor does it require all that much land, though still more than exists in many cities.  Actually growing enough to provide yourself with enough for a good year round food supply is something else entirely, both in time and land requirements.

People move, and moved, to the cities for a wide variety of reasons. Those include the efforts of capitalist, or, in the case of England, feudal elites, to get more money that Cass and De talk about. But they also include the desire for a better and/or more interesting life, and population pressures leading to either environmental degradation or lack of land. The mass movement away from agriculture in postwar Europe was largely due to the second one - life as a factory worker in Milan or the Ruhr improved much faster than that of a small farmer or sharecropper. In much of the third world today it is all of the above.