Hmm, difficult to say really! Clutch release bearing comes to mind but you say the noise is there when you depress the clutch which may discount that. If you get noises when you depress the clutch then it could be a pump bearing. I have a water pump bearing which is going and it makes a psycho noise. Im gonna replace it before it goes. If you are finding it hard to change gear then it could be time to change the clutch, in which case you will change the release bearing.
By chance, I had my clutch changed a while ago now and my gearbox makes a strange grindy noise when you leave the car in neutral. Apparently its common, but when were noises like that common?
hope this helps, this is only my opinion!
scott