I don't KNOW it. It's my opinion. It's a pretty strong one, but it's still an opinion. Now, I base that opinion on many observations. One of them is that I somewhat rarely see anyone actually use the space or the off-road capabilities -- except for contractors, etc. I also see an awful lot of them running around with only one person in them. That's not an SUV-only phenomenon, but at least with a smaller vehicle you're not eating so much gas. Again, these are only my opinions.
I only had two children so I can't say for sure that I wouldn't need an SUV with three, but my opinion is that I still wouldn't get one.
I'd agree that five children would be a problem with a 'standard' vehicle. Depending on your beliefs, five children is a choice too.
My car requires premium so I am not overly concerned (obviously) about getting the cheapest fuel.
Everyone makes their own choices based on what they want or need. Sometimes, those would not be my choices but them we get more to talk about.