"without peer and unimpeachable" right......
Some years ago (many) CR did a test of soldering irons and accessories.
They tested some of the equipment I was using professionally and they
were totally WRONG.
Example, they tested various electronic soldering iron tips, copper,
gold plated etc. At the time we were using Gold plated tips because of
the superior heat transferring and anti-oxidizing properties, nowadays
iron plated tips are quite common. CR's objective without peer and
unimpeachable rating--- Gold plated tips were a waste of money because
the plating came off when they are filed.
I guess they didn't know how to read or follow directions, the package
the tips came in was boldly marked "Do Not File". Plated tips are not
filed to shape like the old solid copper tips meant for metal soldering.
You see eventually the copper and solder reacted with each other and
caused corrosion of the copper tips requiring dressing to remove the
uneven corroded surface. Plated tips are purchased in the desired shape
and can last many years.
Ever since, I have taken a CR review with a grain of salt, sometimes
they don't have a clue what they are doing. Just my opinion.
Address fake until the SPAM goes away ;-}
I don't recall them ever testing anything as niche market as soldering
equipment. Please supply the issue number. I have files dating back into the
'80's and would love to see this faux pas.If what you are talking about
pre-dates that, even if true, give me a break...I doubt if anybody who wrote
what you claim they wrote is still even with CR.
The density of some people posting such UTTER NONSENSE about Consumer
Reports must exceed that of the inside of a black hole.
Come on people, face it, CR survey results come from responses of people who
bought and own the cars they are responding about. They do not come from
Consumer Reports. All they do is add up the responses. CR survey results are
relied by intelligent folks in making buying choices for exactly that
reason - the credibility of the guy who owns one...of lots of guys who own
one, in fact.
Here's another one for you conspiracy kids who think that CR is out to get
GM. I had a 2000 Yukon XL for five years and 30,000 miles. The thing was
pampered. It was used as a traditional station wagon. It had lots of
problems (fuel pump failure, dead power window motor, defective outside
rear-view mirror (twice), cracked interior trim, rear window defroster
failure, ABS sensor failure, power door lock control failure). The expensive
stuff failed out of warranty and cost a total of $1400. Would have been
$1800, but I replaced the rear window with an after-market instead of GM
part. Yet, if you look at the CR rating for the Yukon XL it is pretty much
average. I would consider it to be worse than average...and compared to our
1992 Lexus LS400 that we have had for 12 years, much, much worse than
average. So as far as I'm concerned, CR went easy on the 2000 Yukon XL.
Except they didn't because most folks have had less trouble than I did and
that's what CR reports. The average of their responses.
We know that, but your missing the part about self-selection.
When CR subscribers get the survey, which ones are the most likely to
turn them back in? I'll tell you, it is the people who are at each end of
the bell curve - the people who are really really happy about their products
and the people who are really really pissed off about their products. The
people who don't care either way are much less likely to respond.
So if you have a manufacturer that makes really bad stuff they are going to
take it in the chops. And if you have a manufacturer who makes really
good stuff they are going to get high rated, right? Not so fast. In order
find out the truth you have to normalize the responses.
Normalization is the process where you cruch the numbers. For example,
lets say you get 50 responses saying Toyota is great. And 50 responses
saying GM is great. And this is out of 200 responses that were sent out
meaning a 50% response rate.
CR takes this data and says GM and Toyota are both equally great. But the
is that in the pool of 200 people that owned cars and were surveyed, 125 of
owned GM products.
So in reality, what you have is 50 out of 125, or 40% of the owners of GM's
liked their vehicles so much as to respond that they were great. What were
the other 60% thinking? Can you assume they hated their cars? If so then
why didn't they reply. Can you assume that they merely thought their cars
OK? You just don't know. And you have a situation where 66% of the Toyota
owners liked their vehicles so much as to respond that they were great.
So the conclusion here is that the Toyota responses were much more accurate
because they were a much higher sample of the total population of CR
that owned Toyotas. So the Toyota ranking in the magazine may be accurate,
the GM ranking is much less accurate. And the way the math happens to work,
what this means is the larger the vehicle manufacturer is, and the more cars
sell, the higher the chances that their rankings are going to be inaccurate.
CR cannot normalize their surveys for the simple reason that they do NOT
the subscribers they sent surveys to who did not respond, to get their
If they actually polled all the CR subscribers - by calling them or some
then their survey might have some value - although those results would have
to be normalized by finding out what deviation from the average of the US
population that the CR subscribers represent, but that also can be
during the polling process.
Anyway, getting back to the car vehicle survey problem. If the CR surveys
showed tremendous differences, like every single Toyota owner loved every
last thing about their car, and every single GM owner hated every last thing
about their car, then the tremendous difference would be so great that
even after normalization, it would still be a big difference. But that is
the case at all. The foreign cars that "won" the survey did so well within
the margin of deviation of a normalized survey would be. If normalization
had been done, it could have easily changed the results. But because
it wasn't done, the results are basically worthless.
It would be interesting to see what the response rate is to the surveys.
I felt the same way about who would respond to the surveys that you
do...until, after some 20+ years of responding I realized that I always
respond and I do it whether a car has been very good, very bad, or just very
average. The same with appliances. And then the question becomes, why do I
think other people are any different than me in this regard...and there is
no reason to think that they are. So I basically convinced myself that my
preconceived notion was wrong. I now think the only pre-selection that goes
on is in terms of who is willing to spend the few minutes it takes to "do"
the survey and spend the money on postage to send it in. Given that people
who subscribe to CR (and it is not cheap) are interested in how things
perform and user ratings (why buy the thing, otherwise), I suspect the
response rate is pretty high. However, I have not been able to find the
response rate in a cursory search.
You misinterpret what normalization is. Normalization, in the case of a
survey where the populations are different (number of responses for a given
model) is basically just converting to percentages of people for each model
who had trouble or did not in a given year. You absolutely have to do that
since the populations are different for different models. It's a given.
Otherwise, you would just be comparing raw numbers and could end up with
some nonsense like saying a low responding model with 100 negative responses
(i.e, there was a problem) giving the same rating result as a high
responding model with 100 negative responses.
Now as to the problem of their survey not being truly random, it's true but
it is also unavoidable. You cannot force people to respond. All surveys are
voluntary. No one can. No one's survey is truly random as long as anyone
asked to participate does not participate. However, I don't think that is a
problem in reality (vs. theory) because all the survey results done by
everyone track the same way. And that's year over year. The Asian brands are
best, with exceptions, the U.S. brands are significantly lower, and the
Euro-brands are slightly worse. Things change. The Korean brands were trash
years ago, but they have improved quickly and it shows in the survey
results. This is actually a very nice little check on CR's survey
methodology as it shows that it can show changes as a manufacturer starts to
make significant efforts to improve reliability. Again, the J.D. power
numbers are showing the same thing.
In the end, you can come up with theoretical reasons why the CR (and other
surveys) MAY not be valid, but in the end the reality is that they just are
because the potential problems just do not materialize and people
cooperate...in their own best interest.
Most surveys get about a 20% response rate; I've seen them as low as 2%
and as high as 40%. We are currently conducting a survey that we
expect to get about a 27% response rate on. None of these surveys had
anything to do with cars but the response rate usually reflects the
respondents interest in the subject (i.e. very angry or very satisfied
about the issue).
Group: alt.autos.gm Date: Sat, Mar 11, 2006, 12:01pm From:
It would be interesting to see what the response rate is to the surveys.
"So why do I drive a big SUV? It's because I have to haul numerous
people and things to places."
~ R. Lee Baxton ~
You need to keep in mind the CR surveys are very broad. The same survey
covers cars and vacuum cleaners. I suspect that that buffers things a bit in
terms of really pissed off or wildly happy people.A car with average
reliability will have counter-balancing nos. in both groups. Stinkers will
have more in the black dot group and vice versa.So even if we take for
granted what you say, the results stay the same and valid.
Anyway, I read that their last survey, the cars part, garnered over a
million responses for the first time. That's a pretty big data set.
I do NOT assume every polled person responds. That would be so far outside
the realm of human nature that you could only expect that response in a
Communist country's election (they normally had 100% turnout, officially).
Nope I assume the response rate is well under 100%. But I do still think the
response is representative of the whole CR readership population and that
they represent the population as a whole - or close enough for the results
to be valid. Note that in their latest survey they got responses on over a
million vehicles. If we assume two vehicles per respondent, that is still
over a half-million responses. Big number.
I disagree with your example of a "random" survey where some of the sample
set refuses to reply and they pick another member as a replacement. That
still means that the responses are self-selected since any participant can
choose not to respond, just like the CR survey. Get this, in the case of the
CR survey, the original sample set is THE ENTIRE CR readership. That would
be the ideal that any representative survey tries to match the results of.
The sample set then self-selects itself down to whoever actually responds.
The same thing happens with your targeted survey except that they start out
with the assumption that the entire population cannot be polled, so they
select a sub-set that THEY HOPE is representative of the whole and parts of
it can/will still self-select out. In either case you run the risk the
erroneous results. However, at least in the case of the CR survey, the
results they get can be looked at as a long term, multi-year set and you
know what, they hold together pretty darn well. There is noise in the data
in terms of anomalous results for a given year (some response facet gets
really good/bad for just a single year), but you expect fine detail errors
that in an imperfect survey.
Want another little data point? If the CR results were really very off, you
can bet that a brand that thought they had been horribly mis-judged would
make a HUGE stink about it because CR survey results are closely watched by
the car manufacturers because many of their potential customers refer to
them when buying. CR (and other polling organizations) have been recently
raking MB over the coals for poor quality. Dieter Zetsche, the fellow who
ran Chrysler has recently been promoted CEO and to running MB as well and
one of his stated objectives (as reported in the car buff books) is to get
MB quality back up to where it was long ago. Now one can be sarcastic and
say that he has been mislead by the erroneous results presented by CR in
their surveys (and other polling organizations that show the same results)
and he is going to waste money fixing something that is not broken. What a
see: http://www.edmunds.com/insideline/do/Features/articleId 8085
Q: How much have recent quality problems hurt the image of Mercedes-Benz?
DZ: "A brand is like a savings account. For most of the past 100 years,
Mercedes has paid into this account and built its brand equity. We have
withdrawn from that account in recent years. But it is still a strong
account and we certainly intend to build it up again in the future.
Yes, there were issues. But I am very grateful for what my predecessors have
achieved in improving quality in the past few years. We have made big
strides, and our internal measurements show our quality now is the best it's
ever been. Clearly we need to prove this to customers. We have to make these
quality improvements sustainable and without exception so that every single
car we offer to our customers proves we are back on top."
"Anne Stevens, Ford's chief operating officer for the Americas, said she was
pleased that the automaker had maintained its place among the domestics in
durability, but was unhappy that Ford had not won any top picks. She said
the magazine's findings on the quality gap between Ford and its Asian
competitors were somewhat more pessimistic than Ford's own research.
"Do we recognize there still is a gap? Yes," she said. "But we also
recognize from our survey data that the domestics, and Ford in particular,
are closing the gaps with the Asians."
GM spokeswoman Janine Fruehan said Chevrolet's quality rankings were hurt by
the launch of some older models, which are still counted in the averages for
She said by GM's internal measures, "we are showing ongoing and consistent
signs of improvement in initial quality, long-term quality and perceptual
Doing statistics with inanimate objects is extremely easy compared with
doing it with people and you can get led astray easily, no question.
(Remember New Coke? Think Coke polled - what they thought were
representative - focus groups selected by statisticians on New Coke for
taste testing before they launched it? Think it succeeded?)
You don't want to believe the validity of what CR gets because the
methodology is not perfect, that's fine. Want to doubt the J.D. Power survey
results that say the same thing. That's OK, too. Reality is reality whether
one chooses to acknowledge it or not. And the car manufacturers, even the
ones that don't do well, believe CR, too. They can't afford to self-delude.
Good day to you.
Motorsforum.com is a website by car enthusiasts for car enthusiasts. It is not affiliated with any of the car or spare part manufacturers or car dealers discussed here.
All logos and trade names are the property of their respective owners.