Considering how cheap a spigot bearing is, and the amount of work needed to get to it if it needs replacing, this is something I would always replace willy nilly.
Having once bought a car with a worn spigot bearing, I have to say that apart from the noise, it puts a lot of strain on the synchromesh. A few pence now to save future wear and tear on the gearbox later seems like a good investment.
Jim