For brevity, I snipped Gordon's helpful observations. Look
Of course, CR does too, as has been noted.
I too think this is one of the big advantages of the CR
survey. J.D. Power has only three categories (plus
"overall"). CR has 17! It is very important to me to know
whether a tranny has been problematic and whether it is
"major" or "minor" problematic, or is it electrical or
"major engine" or "minor engine" etc. CR evaluates this.
Maybe you saw this already, but for others, here is an FAQ
on the CR survey that I think is very helpful:
It puts the average sample per model-year between 200 and
400, which is less than I estimated, with some model-years
having several thousand samples, and some having less than
100. The latter's results are excluded from publication.
The CR FAQ also notes that it is the differences between
models where there is statistical significance. Again,
that's key. Because fact is a 1% failure rate in a sample
size of 1000 has a margin of error of about +/- 3%. (One
sees this margin of error in political polls all the time.
Political poll takers aim for around 1000 "hits" so they can
report a MOE of about 3%.) So CE White is correct with his
concern about reading any individual chart "too precisely."
But his concern will also apply to the J.D. Power survey.
One has to look at the differences between models, instead,
among other things.
Please let me know wherer I can find the "numbers." I have the
magazine and an on-line subscription. I've nver seen raw numbers. It
is my opinion that CR does there very best to obsure the actual source
of their data and to over emphasize minor differences. If they
actually have the raw numbers available somewhere, maybe I would
change my opinion.
And then they don't tell you the numbers, instead they feed them to
some internal CR process that obscures the raw data and outputs
meaningless little circles. Plus, they allow the respondent a lot of
leeway in deciding what is minor and what is major.
Thanks for posting this. It confirms my worst fears. CR is making very
fine distinction form poorly collected data. The FAQ tries to spin
this as being useful, but clearly the little circles are even less
meaningful than I thought. In many cases they are giving vehicles a
poor rating based on a reported problem rate 4% greater than average.
There is no way the CR survey has an accuracy of +/-3% for most of the
vehicles listed (the typical vehicle has 200 to 400 responses; they
allow data to be reported with as few as 100 responses). This means
the little circles are at best worthless for many vehicles. I suppose
for high volume vehicles there may be some validity, but still the
difference between an excellent and poor rating is at best very small.
Probably so small as to be insignificant compared to other factors if
people knew how small the difference truly is. My sister just
purchased a RAV4, mainly because it had such good reliability ratings.
If I had told her it was at best likely to have 4% fewer problems than
an Escape, which she could have bought for thousands less, I suspect
she might have considered the Escape (especially since my younger
Sister has a 7 year old Escape that has been trouble free).
Actually I agree that my concerns apply to JD Powers is well. But at
least JD Powers gives you the raw data (problems per 100 vehicles).
From that I can infer that most vehicles are very close in quality. CR
on the other hand gives you little circles that imply great
difference, when in fact they are actually very minor in most cases. I
find this to be a misleading approach.
We discussed this already. Look at the key for the circles
in the April issue.
The notion that what the CR circles tell us are the
/differences between/ models, and not a statistically
meaningful problem rate for each model-year, is not easy for
a lot of people to grasp. Yet it's a well-known statistical
concept. Most often it's the /difference/ in two averages
that is most meaningful, and not the averages themselves.
This is like an Internet mythology. Without your citing specific
instances where this is the case, it is pretty hard to respond. As
far as I can see, related vehicles usually have very similar
The average model year had about 7000 responses. A 1% failure rate
represents 70 respondents (typically) who reported a problem. My
guestimate is this is a lot better than a 3% margin of error.
The opinions are irrelevant. The question is, did you have to repair
the transmission last year, yes or no? If the survey is inaccurate,
it has produced some uncanny results. For example: Honda, of course,
has a stellar repair record - traditionally neck and neck with Toyota
for best in the world. Yet one year, CR reported that one feature on
one Honda model had the worst repair record in the survey. That would
seem to indicate that the survey respondents weren't influenced by
Actually I would say that the Yaris is a perfect example of CR bias.
Despite having only one years worth of questionnaire data, they give
it a predicted reliability rating of much better than average. How
many Yaris owners do you figure responded to the 2007 CR survey? In
most cases CR would say the model was to new to be rated. But for a
Toyota, they assume it is great.....despite recent Camry and Tundra
problems. I suppose they may be basing the high rating on the history
of the Echo, but there was a gap of a year between the last Echo model
and the first Yaris model, and the Echo had some problem areas (paint,
brakes) identified for the last year they were sold
It is interesting to read the Consumer Opinions on the CR site for the
Yaris. People would complaint about the ride, or the driving position,
or noise, and still give the car 5 stars. The great majority (~38 out
of 53) gave it 5 stars (and all but a handful of the others gave it
four stars). It seems that most people that buy these are satisfied
because the car gets great gas mileage and they are willing to put us
with a lot of crap to get it. I doubt that many of the respondents to
the 2007 questionnaire had more than 10k miles on their Yaris. Do you
think this is sufficient to say anything? Given the fact that most of
the owners who responded with written opinions on the web site seemed
to mostly care about gas mileage, does it seem reasonable to assume
that more than a few might gloss over a few minor reliability problems
because of their smugness at getting good gas mileage? Lots of cars
get really good rating the first year they are surveyed. For instance,
the 2007 Focus (a recommended car) has really good first year
reliability rating based on the survey results (as it does for 2006
and 2005), yet CR did not provide a predicted reliability rating. They
said it was "new." However, in the road test they referred to the 2007
design as a "freshening" and implied the underpinnings were not much
changed. So why don't they give the Focus a good predicted reliability
rating? Seems to me that they have at least as much basis for giving
it this rating as for the Yaris. Remember, they claim they base the
predicted reliability on the latest three model years of a design -
well unless it is a Toyota, apparently one year is enough for them.
I didn't claim CR openly lied about reliability. Even CR doesn't have
enough guts to make excuses for that turkey. Ditto for the V6 Camry.
Some things are just to bad to cover-up. I was complaining about the
Yaris getting a an excellent reliability rating when it is a new
model, with no substantial history. I see the FAQ covers this (or is
it excuses this?).
But the only reason you know it is a "turkey" is because CR told you
it was. Up until 2006, the Camry has an excellent reliability record.
If there was bias in the system, how was the poor reliability of 2006
and 2007 models spotted so quickly? Same thing with the 2007 Tundra.
Both of the vehicles received high marks in testing. There was no
reason fro the owners to suddenly turn on them. They just
independently reported the troubles they had and when the results were
tallied, they had a relatively poor level of reliability that was very
surprising for a Toyota product. There is no reason to suspect that
the results are not accurate.
AFAIK, no one even suspected that Toyota quality was slipping until CR
reported this. That says to me that they could easily have covered it
up if they chose.
Predicting excellent reliability on a new model when the "manufacturer
has a track record of consistently outstanding (above average)
reliability" doesn't seem unreasonable.
Consumer Reports isn't the authoritative survey, but because it's on
the magazine shelf it has a lot of clout with the typical consumer. If
you know a more authoritative source than JD Powers let me know.
Actually, they probably only have 3 - 6K on the car as the average new
model is only three months old when the survey is completed. As
someone said, you might be able to determine it is terrible that
quick, but it is too soon to know if it is great. That first year
record is the equivalent of the JDP initial quality survey except JDP
mixes reliability questions with fuzzy stuff like how the dealer
Why does Toyota give Yaris a solid prediction based on little
evidence? You would have to ask them but I suspect that they looked
at the first year results and compared them to older Toyotas that had
similar first year reliability and decided that Yaris would likely be
Sounds like a wild theory to me. If the water pump broke, the water
pump broke. Even at the height of their popularity, the monster SUVs
never were rated reliable by their smug owners.
First of all, the first year reliability ratings on Focus aren't as
good as most Toyotas. Second, the prediction is based on all model
years available, not just results for the most recent model year
(which we agree doesn't mean much.) Focus reliability has been
mediocre over the years.
They apparently decided that there were too many unknowns for the new
design. If they had based a prediction on past experience with the
Focus, it probably would have gotten an empty circle.
I think they base it on three years if available. I seem to recall
them noting when a prediction was based on only a single year's
Motorsforum.com is a website by car enthusiasts for car enthusiasts. It is not affiliated with any of the car or spare part manufacturers or car dealers discussed here.
All logos and trade names are the property of their respective owners.