Uneven brakes

1984 Buick Century 4cyl auto, 4 door, disk front, drum rear brakes. Since I've had it, the back brakes (or at least one of them, not sure) will sometimes lock far before you would expect it to. This is most severe when the brakes have not been used in a while, and I can usually prevent the worst of it by pushing the brake pedal a few times while stopped.

Very recently, I've heard a grinding noise for the first stop or two after an hour or two of non-use. Pads and shoes have adequate lining, drums and rotors are not gouged, no leaks. The rear shoes appear to have more wear than I would expect, but I could be underestimating use and miles since the last change. Left front brake had far more wear on the outboard pad than the inboard, but it wasn't worn to the squeaker yet. The sleeve at the retaining allen bolt was very tight, not able to slide evenly, probably accounting for the uneven wear, and I'm guessing unrelated to the other problems..

Any idea what might be causing the premature rear lockup?