While you've got a significant amount of logic on your side, I've seen this
before. What happens is that the tie rod end holds the rear edge of the
wheel steady but the play in the ball joint allows the front of the wheel
to be pulled in and out.
Regardless, it's all moot until someone actually gets under and sees the
component that is moving around.