That's entirely possible. I used to hear what sounded like gravel
hitting the door while I drove along listening to a specific CD in my
94 LS. I'd turn the radio down, trying to isolate the sound, and
wouldn't hear anything. It happened with a specific CD that had a song
on it with more bass energy than normal.
Anyway, I took the door panels off to inspect the speakers, and the
cones were ripped all the way around the perimeter of the speaker. I
replaced them with some decent speakers, and the stereo immediately
sounded much better, with no gravelly rattling noise.
The best solution for eliminating the noise without paying for new
speakers is to turn the radio off. Running it without speakers will
probably screw it up.