If the housing the bearing is in has any play, you're in trouble, as it will get worse. Of course the play disappears with the brake on - the backplate is rigid.
There is some clearance in all bearings, and this will vary slightly in this non critical application between samples.
More likely that will give trouble than the part it's meant to be protecting.