Please let me know wherer I can find the "numbers." I have the magazine and an on-line subscription. I've nver seen raw numbers. It is my opinion that CR does there very best to obsure the actual source of their data and to over emphasize minor differences. If they actually have the raw numbers available somewhere, maybe I would change my opinion.
And then they don't tell you the numbers, instead they feed them to some internal CR process that obscures the raw data and outputs meaningless little circles. Plus, they allow the respondent a lot of leeway in deciding what is minor and what is major.
Thanks for posting this. It confirms my worst fears. CR is making very fine distinction form poorly collected data. The FAQ tries to spin this as being useful, but clearly the little circles are even less meaningful than I thought. In many cases they are giving vehicles a poor rating based on a reported problem rate 4% greater than average. There is no way the CR survey has an accuracy of +/-3% for most of the vehicles listed (the typical vehicle has 200 to 400 responses; they allow data to be reported with as few as 100 responses). This means the little circles are at best worthless for many vehicles. I suppose for high volume vehicles there may be some validity, but still the difference between an excellent and poor rating is at best very small. Probably so small as to be insignificant compared to other factors if people knew how small the difference truly is. My sister just purchased a RAV4, mainly because it had such good reliability ratings. If I had told her it was at best likely to have 4% fewer problems than an Escape, which she could have bought for thousands less, I suspect she might have considered the Escape (especially since my younger Sister has a 7 year old Escape that has been trouble free).
Actually I agree that my concerns apply to JD Powers is well. But at least JD Powers gives you the raw data (problems per 100 vehicles). From that I can infer that most vehicles are very close in quality. CR on the other hand gives you little circles that imply great difference, when in fact they are actually very minor in most cases. I find this to be a misleading approach.
Ed