This sounds like your release bearing is worn out. Its not a big deal, I drove my 98 civic around for like over a year before I bothered to do anything about it. Just make sure you REMEMBER to do it when you do the clutch, I ended up forgetting to replace it when we did the clutch on my girlfriends 87 prelude. We ended up pulling it all apart again just to do the release bearing, cuz guess what started making a grinding noise as soon as we put it back together? DOH!
As Tegger said, it could be your transmission input shaft bearing, but the way you describe the noise, only when the clutch pedal is out, makes me think release bearing.
By the way, the Input shaft bearing went on my civic too, but the shop just replaced it. It wasn't cheap, but leads me to believe it could be done by an ambitious Do-it-yourselfer, they only charged the warranty for like 4 hours of labour, and they did a crapload of other stuff too. Good thing I had Honda Plus.
t