Ore grades will never degrade below 4ppm uranium and 12ppm thorium.  That's the concentration in granite, and there's plenty of that.  A 1GW fast breeder[1] will need about 3 tonnes of fuel per year, which at the above concentration amounts to about 500 tonnes of granite per day.  Let's say 1000, allowing for inefficient extraction and some losses.

A coal plant of the same size requires 10 times that amount of coal, leaving a hole in the ground 10 times as large.  This environmental impact is not considered larger than that of radwaste, so the same should go for mining rocks.