I too was thinking the same thing. Then you don't risk damaging a more difficult to replace part, too...
I'd probably chuck it up in my small lathe and use the lathe to spin it while dressing it down with the crocus cloth, checking frequently until it is close. I would try it at .002 to see if it could be done, then at .001 if necessary, not try to go down to exact size in one step.