Re: Screw Consumer Reports

Page 6 of 6  
says...


Some years ago (many) CR did a test of soldering irons and accessories. They tested some of the equipment I was using professionally and they were totally WRONG.
Example, they tested various electronic soldering iron tips, copper, gold plated etc. At the time we were using Gold plated tips because of the superior heat transferring and anti-oxidizing properties, nowadays iron plated tips are quite common. CR's objective without peer and unimpeachable rating--- Gold plated tips were a waste of money because the plating came off when they are filed.
I guess they didn't know how to read or follow directions, the package the tips came in was boldly marked "Do Not File". Plated tips are not filed to shape like the old solid copper tips meant for metal soldering. You see eventually the copper and solder reacted with each other and caused corrosion of the copper tips requiring dressing to remove the uneven corroded surface. Plated tips are purchased in the desired shape and can last many years.
Ever since, I have taken a CR review with a grain of salt, sometimes they don't have a clue what they are doing. Just my opinion.
--
I. Care
Address fake until the SPAM goes away ;-}
  Click to see the full signature.
Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload
right......

Over the years, CR has recommended a lot of crap products. I learned not to trust them in the past.
I have had no epiphany that they are now any better.
Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload
I don't recall them ever testing anything as niche market as soldering equipment. Please supply the issue number. I have files dating back into the '80's and would love to see this faux pas.If what you are talking about pre-dates that, even if true, give me a break...I doubt if anybody who wrote what you claim they wrote is still even with CR.

Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload
Booboo Baker wrote:

There is a difference between "their" and "there."

Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload

Their is no difference at all.
Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload
The density of some people posting such UTTER NONSENSE about Consumer Reports must exceed that of the inside of a black hole.
Come on people, face it, CR survey results come from responses of people who bought and own the cars they are responding about. They do not come from Consumer Reports. All they do is add up the responses. CR survey results are relied by intelligent folks in making buying choices for exactly that reason - the credibility of the guy who owns one...of lots of guys who own one, in fact.
Here's another one for you conspiracy kids who think that CR is out to get GM. I had a 2000 Yukon XL for five years and 30,000 miles. The thing was pampered. It was used as a traditional station wagon. It had lots of problems (fuel pump failure, dead power window motor, defective outside rear-view mirror (twice), cracked interior trim, rear window defroster failure, ABS sensor failure, power door lock control failure). The expensive stuff failed out of warranty and cost a total of $1400. Would have been $1800, but I replaced the rear window with an after-market instead of GM part. Yet, if you look at the CR rating for the Yukon XL it is pretty much average. I would consider it to be worse than average...and compared to our 1992 Lexus LS400 that we have had for 12 years, much, much worse than average. So as far as I'm concerned, CR went easy on the 2000 Yukon XL. Except they didn't because most folks have had less trouble than I did and that's what CR reports. The average of their responses.

Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload

We know that, but your missing the part about self-selection.
When CR subscribers get the survey, which ones are the most likely to turn them back in? I'll tell you, it is the people who are at each end of the bell curve - the people who are really really happy about their products and the people who are really really pissed off about their products. The people who don't care either way are much less likely to respond.
So if you have a manufacturer that makes really bad stuff they are going to take it in the chops. And if you have a manufacturer who makes really good stuff they are going to get high rated, right? Not so fast. In order to find out the truth you have to normalize the responses.
Normalization is the process where you cruch the numbers. For example, lets say you get 50 responses saying Toyota is great. And 50 responses saying GM is great. And this is out of 200 responses that were sent out meaning a 50% response rate.
CR takes this data and says GM and Toyota are both equally great. But the problem is that in the pool of 200 people that owned cars and were surveyed, 125 of them owned GM products.
So in reality, what you have is 50 out of 125, or 40% of the owners of GM's liked their vehicles so much as to respond that they were great. What were the other 60% thinking? Can you assume they hated their cars? If so then why didn't they reply. Can you assume that they merely thought their cars were OK? You just don't know. And you have a situation where 66% of the Toyota owners liked their vehicles so much as to respond that they were great.
So the conclusion here is that the Toyota responses were much more accurate because they were a much higher sample of the total population of CR subscribers that owned Toyotas. So the Toyota ranking in the magazine may be accurate, but the GM ranking is much less accurate. And the way the math happens to work, what this means is the larger the vehicle manufacturer is, and the more cars they sell, the higher the chances that their rankings are going to be inaccurate.
CR cannot normalize their surveys for the simple reason that they do NOT poll the subscribers they sent surveys to who did not respond, to get their opinions. If they actually polled all the CR subscribers - by calling them or some such - then their survey might have some value - although those results would have to be normalized by finding out what deviation from the average of the US population that the CR subscribers represent, but that also can be established during the polling process.
Anyway, getting back to the car vehicle survey problem. If the CR surveys showed tremendous differences, like every single Toyota owner loved every last thing about their car, and every single GM owner hated every last thing about their car, then the tremendous difference would be so great that even after normalization, it would still be a big difference. But that is not the case at all. The foreign cars that "won" the survey did so well within the margin of deviation of a normalized survey would be. If normalization had been done, it could have easily changed the results. But because it wasn't done, the results are basically worthless.
Ted
Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload
It would be interesting to see what the response rate is to the surveys.
I felt the same way about who would respond to the surveys that you do...until, after some 20+ years of responding I realized that I always respond and I do it whether a car has been very good, very bad, or just very average. The same with appliances. And then the question becomes, why do I think other people are any different than me in this regard...and there is no reason to think that they are. So I basically convinced myself that my preconceived notion was wrong. I now think the only pre-selection that goes on is in terms of who is willing to spend the few minutes it takes to "do" the survey and spend the money on postage to send it in. Given that people who subscribe to CR (and it is not cheap) are interested in how things perform and user ratings (why buy the thing, otherwise), I suspect the response rate is pretty high. However, I have not been able to find the response rate in a cursory search.
You misinterpret what normalization is. Normalization, in the case of a survey where the populations are different (number of responses for a given model) is basically just converting to percentages of people for each model who had trouble or did not in a given year. You absolutely have to do that since the populations are different for different models. It's a given. Otherwise, you would just be comparing raw numbers and could end up with some nonsense like saying a low responding model with 100 negative responses (i.e, there was a problem) giving the same rating result as a high responding model with 100 negative responses.
Now as to the problem of their survey not being truly random, it's true but it is also unavoidable. You cannot force people to respond. All surveys are voluntary. No one can. No one's survey is truly random as long as anyone asked to participate does not participate. However, I don't think that is a problem in reality (vs. theory) because all the survey results done by everyone track the same way. And that's year over year. The Asian brands are best, with exceptions, the U.S. brands are significantly lower, and the Euro-brands are slightly worse. Things change. The Korean brands were trash years ago, but they have improved quickly and it shows in the survey results. This is actually a very nice little check on CR's survey methodology as it shows that it can show changes as a manufacturer starts to make significant efforts to improve reliability. Again, the J.D. power numbers are showing the same thing.
In the end, you can come up with theoretical reasons why the CR (and other surveys) MAY not be valid, but in the end the reality is that they just are because the potential problems just do not materialize and people cooperate...in their own best interest.

Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload
Most surveys get about a 20% response rate; I've seen them as low as 2% and as high as 40%. We are currently conducting a survey that we expect to get about a 27% response rate on. None of these surveys had anything to do with cars but the response rate usually reflects the respondents interest in the subject (i.e. very angry or very satisfied about the issue).
Group: alt.autos.gm Date: Sat, Mar 11, 2006, 12:01pm From: snipped-for-privacy@CHARTERMI.NET (GRL) It would be interesting to see what the response rate is to the surveys.
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% "So why do I drive a big SUV? It's because I have to haul numerous people and things to places." ~ R. Lee Baxton ~
Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload
You need to keep in mind the CR surveys are very broad. The same survey covers cars and vacuum cleaners. I suspect that that buffers things a bit in terms of really pissed off or wildly happy people.A car with average reliability will have counter-balancing nos. in both groups. Stinkers will have more in the black dot group and vice versa.So even if we take for granted what you say, the results stay the same and valid.
.
Anyway, I read that their last survey, the cars part, garnered over a million responses for the first time. That's a pretty big data set.
(GRL)

Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload

Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload
I do NOT assume every polled person responds. That would be so far outside the realm of human nature that you could only expect that response in a Communist country's election (they normally had 100% turnout, officially).
Nope I assume the response rate is well under 100%. But I do still think the response is representative of the whole CR readership population and that they represent the population as a whole - or close enough for the results to be valid. Note that in their latest survey they got responses on over a million vehicles. If we assume two vehicles per respondent, that is still over a half-million responses. Big number.
I disagree with your example of a "random" survey where some of the sample set refuses to reply and they pick another member as a replacement. That still means that the responses are self-selected since any participant can choose not to respond, just like the CR survey. Get this, in the case of the CR survey, the original sample set is THE ENTIRE CR readership. That would be the ideal that any representative survey tries to match the results of. The sample set then self-selects itself down to whoever actually responds. The same thing happens with your targeted survey except that they start out with the assumption that the entire population cannot be polled, so they select a sub-set that THEY HOPE is representative of the whole and parts of it can/will still self-select out. In either case you run the risk the erroneous results. However, at least in the case of the CR survey, the results they get can be looked at as a long term, multi-year set and you know what, they hold together pretty darn well. There is noise in the data in terms of anomalous results for a given year (some response facet gets really good/bad for just a single year), but you expect fine detail errors that in an imperfect survey.
Want another little data point? If the CR results were really very off, you can bet that a brand that thought they had been horribly mis-judged would make a HUGE stink about it because CR survey results are closely watched by the car manufacturers because many of their potential customers refer to them when buying. CR (and other polling organizations) have been recently raking MB over the coals for poor quality. Dieter Zetsche, the fellow who ran Chrysler has recently been promoted CEO and to running MB as well and one of his stated objectives (as reported in the car buff books) is to get MB quality back up to where it was long ago. Now one can be sarcastic and say that he has been mislead by the erroneous results presented by CR in their surveys (and other polling organizations that show the same results) and he is going to waste money fixing something that is not broken. What a chump, huh?
see: http://www.edmunds.com/insideline/do/Features/articleId 8085
Q: How much have recent quality problems hurt the image of Mercedes-Benz?
DZ: "A brand is like a savings account. For most of the past 100 years, Mercedes has paid into this account and built its brand equity. We have withdrawn from that account in recent years. But it is still a strong account and we certainly intend to build it up again in the future.
Yes, there were issues. But I am very grateful for what my predecessors have achieved in improving quality in the past few years. We have made big strides, and our internal measurements show our quality now is the best it's ever been. Clearly we need to prove this to customers. We have to make these quality improvements sustainable and without exception so that every single car we offer to our customers proves we are back on top."
or:
http://www.freep.com/apps/pbcs.dll/article?AID=/20060301/NEWS11/60301011
"Anne Stevens, Ford's chief operating officer for the Americas, said she was pleased that the automaker had maintained its place among the domestics in durability, but was unhappy that Ford had not won any top picks. She said the magazine's findings on the quality gap between Ford and its Asian competitors were somewhat more pessimistic than Ford's own research.
"Do we recognize there still is a gap? Yes," she said. "But we also recognize from our survey data that the domestics, and Ford in particular, are closing the gaps with the Asians."
GM spokeswoman Janine Fruehan said Chevrolet's quality rankings were hurt by the launch of some older models, which are still counted in the averages for the brand.
She said by GM's internal measures, "we are showing ongoing and consistent signs of improvement in initial quality, long-term quality and perceptual quality."
Doing statistics with inanimate objects is extremely easy compared with doing it with people and you can get led astray easily, no question. (Remember New Coke? Think Coke polled - what they thought were representative - focus groups selected by statisticians on New Coke for taste testing before they launched it? Think it succeeded?)
You don't want to believe the validity of what CR gets because the methodology is not perfect, that's fine. Want to doubt the J.D. Power survey results that say the same thing. That's OK, too. Reality is reality whether one chooses to acknowledge it or not. And the car manufacturers, even the ones that don't do well, believe CR, too. They can't afford to self-delude. You can.
Good day to you.

Add pictures here
<% if( /^image/.test(type) ){ %>
<% } %>
<%-name%>
Add image file
Upload

Related Threads

    Motorsforum.com is a website by car enthusiasts for car enthusiasts. It is not affiliated with any of the car or spare part manufacturers or car dealers discussed here. All logos and trade names are the property of their respective owners.