1997 Buick Park Avenue--A/C problem

A/C works fine, no noise from compressor or clutch. When it cycles on and off, still no noise. However, when turned off I hear a shrill, sorta grinding noise, and can see little metal flecks beneath the pulley. It's got to be the A/C clutch but don't understand what's happening. I assume it stays engaged as the thermostat cycles on and off, but why is it self-destructing when the system is completely turned off? ---Bob