True, if you truly decide it on that basis.
But too often, you can't get the trim level/feature sets on the Toyota Camry that you can get on the Lexus ES. That's just how Toyota does things. The ES is, by and large, a higher trim level/feature set.
So, to get the feature set you want, you are forced to go to Lexus. It's not just because there's an L on the back of the thing. If Toyota offered the exact same feature set, I'd buy it. But they don't, so if I want that feature set, I'm forced to take the L with it.