Re: Consumer Reports 2007 Auto Survey

The CR true believers cannot be swayed by actual science and proper methods. It's a faith. What really gets me is that they actually think that a lack of advertising makes CR unbiased. As if bias cannot form without advertising money. They have their opinions and biases, it was pretty clear from the articles I read many years ago.

Anyway it's a question of faith.

I think Brent has the right idea. I read CR (at the library, sometimes) with some faith and trust in their opinions, but sometimes I sense a feeling of "we'll let this one skate" but "I didn't like the feeling of the seat in that" in their articles. They seem to ignore rather obvious flaws in auto design, (like a plugugly car) and focus on the fact that the cupholders are mounted at an unfortunate angle for left-handed albino dwarfs. Their opinions are a bit too subjective for my liking. I stopped thinking of CR as gospel about 53 years ago, when they compared a Chrysler Windsor with a Hudson, saying the Chrysler's body was a little torquey, and the Hudson was more rigid. They recommended the Hudson over the Chrysler, totally ignoring the fact that at the time, if you bought a Hudson, you must realize that at trade-in time, you'd better buy another Hudson, because your trade-in value at any other car dealer would be nickels and dimes. CR overall is a good guide to avoid junk, but in a lot of cases, by the time their ratings are published, the products are a year or more old, and some are discontinued at retail outlets, and the models that supersede the products may or may not be similar in features and quality.

You've said that before and you're wrong. It is NOT "clearly an opinion survey." It is a survey which asks SOME questions which elicit answers based on opinions, some questions which require answers based on factual numbers, and some which require answers based on a combination of facts and judgments. To say it is "clearly an opinion survey" shows your own strong bias against their surveys: you're focusing ONLY on the part of the survey that is about opinion and completely ignoring the part which is about facts.

For example, the questions "Was any work done on this vehicle in the past 12 months (excluding tire replacement or accident damage) covered by a warranty or by a recall notice?" and "Did this vehicle get any MAINTENANCE (e.g., oil change/lube/tire rotation) or REPAIRS in the past 12 months?" are not asking for any opinion at all.

Personally, I don't find any use for the opinion portions "are you satisfied/would you buy again" etc. I look at the reliability data based on repair reports. That's the part of the poll to which I refer from this point on.

o> f a specific group who's opinions are influenced by the group

I agree that would be better because of the hard cost data. Since you seem to be implying that CR readers would answer poll questions favorably towards some cars and unfavorable towards others in a consistent enough way to skew the results, it would be better to have a random set of car owners, not because it makes sense that CR readers as a whole would be biased toward or against any particular nation's cars but so that people like you can't say the test is skewed because the owners who were polled all read the same magazine.

`Your opinion means nothing. My opinion means nothing. When you take hundreds of opinions and average them out, you have something. For example, you take a 1.5 pound weight, and ask one person how heavy it is, he might get it right, but he could be way off. However if you ask hundreds of people how much it weighs and average out the answers, you'll have an answer that is remarkably close to 1.5 pounds. Unfortunately I can't remember what book I read about this, so I can't provide any hard evidence. If it makes sense you can accept it, if not then disregard. Naturally, a larger sample would provide more reliable data. That's the biggest caveat I have about the CR surveys--the samples are far too small.

I have three questions.

1. On what evidence is this belief based?
2. How would the survey takers know the opinions about the reliability of a given auto that the CR editors have? As far as I've seen, the CR editors don't state their opinions about the reliability of autos--they simply publish the survey results and let them to the talking.
3. Even if the people answering the surveys knew the opinions of the CR editors and tended to share them, how would that affect their poll answers? Is it like they'd think, "Well, my engine in mt new Toyota fell out, but the CR editors think Toyotas are reliable, therefore it must not have been a serious problem, so I won't report it on the poll." Is that what you're suggesting? Again, I'm referring to the reliability reports, NOT the owner satisfaction part of the poll.

I never read the opinion pieces, unless you consider the reviews opinion pieces. I don't necessarily like what they like, but the information I've gotten from them has always proven helpful. If they say orange juice A is sweeter than orange juice B and therefore tastes better, I ignore the part OJ A being better and buy OJ B if I like my OJ less sweet.

Every time I've read one of their reviews on a product which I already owned or later purchased, they had the facts straight even if I disagreed with their conclusions--and the facts are what I pay attention to.

If it were ever revealed that anyone at CR was skewing their ratings towards what they thought would please their readers and in the process skewing the ratings away from the truth, how long do you think it would take for them to go out of business? There'd be no reason to buy their magazine or send them money for anything after that.

Actually it is likely that most CR readers are independent thinkers like you and when filling out the survey will ignore the temptation to confirm CR editorial opinions.

CR does not consider trade in value when rating cars. Nor do they consider ugliness.

The opinion questions have some value. For example Consumer Reports discovered that people loved their mini's and would buy them again even though they are unreliable cars.

"That Guy" said in rec.autos.driving:

And you're ignoring the facts that a) the sample group is entirely self-selected, and b) even if the sample of CR subscribers were done randomly, it's still not statistically significant because of the limited (and clearly biased) sample population.

Brent is right - the results are no more meaningful than an opinion survey.

But they do rate their depreciation, & make comments on their design.

Cathy

I stopped putting much stock in their car ratings when they went thru a cycle of how they rated cars. For a few years, maybe a decade or more ago, they starting including a cost index of some sort to go along with the trouble index. So that Mercedes that looked like it rarely had any problems suddenly got black dots for it's cost of repair index. Some of their other favorites started to get those black dots too. Some of the domestics that had not quite as good of scores in the trouble index showed up pretty good in the cost index.

That only lasted a couple years and they printed some half assed nonsense about how it wasn't really a good way to evaluate vehicles and dropped that rating. There is absolutely no doubt in my mind that both the editors and the readers of CR have an anti-US bias.

I don't follow that logic at all. They won't be subscribing to CU if they don't more or less agree with the editorial stance of the magazine. For example, the magazine has pushed many social programs for the past decade - to the point where they lost some subscribers, such as myself, who didn't care to be lectured to by some bozo who's supposed to be testing clothes dryers for me. So whoever subscribes is statistically likely to be of a like mind to the CR editors on many subjects. Therefore, their survey WILL be biased. It's been noted many times that if you look at vehicles that are built on the same assembly line but given different "names", they will often have different ratings and virtually always the "American" version will get lower ratings then the "imported" version.

A few "hard data" questions in the survey doesn't turn the overall survey into a hard data gathering exercise. The main questions in the survey are clearly eliciting an opinion. Questions like - "If you had any problems with your car in the past 12 months that you considered SERIOUS because of cost, failure, safety or downtime, select the appropriate box(es)...." or "How satisfied are you with this vehicle with respect to each of the following factors?" The answers to these questions are opinions, not facts. The chief information gathered is all based on the opinions of the reader. This makes it "clearly an opinion survey." When the pollsters come around and ask me how GWB is doing, they also ask me my age and race. Just because they ask me some hard data questions, does that make my answers to the other questions facts?

You mean the data gathered where they asked the respondents to decide what is "SERIOUS?"

No, I am saying that CR readers tend to spit back the opinions of the magazine. They have been told repeatedly that car A is very reliable, so when they are asked to report SERIOUS problems with car A, they tend to think problems with such a reliable car had can't be serious. I am not saying it is a massive case of deliberate misreporting. Because of the limited nature of the survey, very small shifts can appears very significant when the data is reduced to the little circles. In my opinion, the data gathering method is unscientific and the reporting methods can make it appear that there are great differences when in fact, there may be no statistically valid differences at all between component systems of two different cars. This means that very subtle biases can skew the results and make it appear there are significant differences when in fact they don't actually exist.

It is not just that the samples are too small, they are both too small and from a non-random group.

For years Buicks have shown up better in reliability ratings (like JD Powers) than other GM products (at times even better than Cadillac). Why? In most cases the basic parts of the car are exactly the same as Pontiacs and Oldsmobiles and they are built on the same assembly lines by the same workers. There is no reason to believe that Buicks are more reliable than Oldsmobiles, but if you look at JD Powers surveys, Buicks always come out much better. This also used to be the case with CR surveys, but in recent years it appears that CR started blending the data for similar cars sold by different divisions to avoid having to explain this phenomenon.

You have to be kidding. When a new Toyota comes out the CR editors will say they expect it to be reliable. When a new Buick comes out they will say it is a new model and the reliability is unknown.

The reliability reports are based on the respondents deciding what is "SERIOUS." This is an opinion based question. Cars from all manufacturers have very few problems of any type (according to JD Powers the average new car has 1.24 problems, and the average 3 year old car has 2.27 problems). It only takes a subtle shift in determining what is "SERIOUS" to make large differences in the survey. For all I know, Toyota owners might be overly picky and over report problems compared to Buick owners. The CR reporting methods (the little circles) can make tiny differences appear to be very significant.

The reviews are full of opinions. For instance:

"This large, front-wheel-drive four-door sedan replaced the LeSabre and the Park Avenue. The standard power plant is a rough-sounding

3.8-liter V6. A potent 4.6-liter, 275-hp V8 powers the CXS. The four-speed automatic transmission shifts smoothly enough. The Lucerne has a quiet, comfortable ride, especially at low speeds. Handling is not agile and the steering lacks feedback. It has a tendency to fishtail easily at its limits. Stability control is only available on the CXS V8. Braking is unimpressive. The back seat is roomier than the one in the LaCrosse. First-year reliability has been above average."

Opinions: "rough-sounding 3.8-liter V6" "the four-speed automatic transmission shifts smoothly enough" "has a quiet, comfortable ride" "Braking is unimpressive"

or

"The Mercury Milan and similar Ford Fusion are new midsized sedans that are very agile and satisfying to drive. These sedans rate even higher in our testing than the Mazda6 sedan on which they're based. They have a sporty feel; the V6 and automatic transmission make a smooth and responsive powertrain. The four-cylinder is a bit coarse. The interior is well made and space is generous."

Opinions: "are new midsized sedans that are very agile and satisfying to drive" "They have a sporty feel" "The four-cylinder is a bit coarse" "The interior is well made and space is generous"

or

"The redesigned Toyota Camry is roomy, quiet, has a comfortable ride, and is refined. The addition of a telescoping steering column is a plus. Power comes from a strong 3.5-liter V6 mated to a six-speed automatic transmission. It returns 23 mpg overall, just one mpg less than the four-cylinder. The base 2.4-liter four-cylinder is also responsive and relatively refined. A four-cylinder hybrid version returned an impressive 34 mpg overall. Handling is responsive and secure but not sporty. The interior is spacious, with reclining rear seats in the high-end XLE. Curtain air bags are standard, but stability control remains optional."

Opinions: "is roomy, quiet, has a comfortable ride, and is refined" "......a strong 3.5-liter V6" "The base 2.4-liter four-cylinder is also responsive and relatively refined" "Handling is responsive and secure but not sporty" "The interior is spacious"

etc., etc., etc.

The reviews are full of opinions.

You don't think the whole continual retesting of the Suzuki Samurai wasn't an attempt to skew the ratings? Every newspaper / magazine / TV news program / etc that I am aware of skews their content to please / attract the readers / viewers. You are naive if you don't think CR does as well.

People keep buying Motor Trend, and I am sure most of the people who buy it understand that "Car / Truck / SUV of the Year" is up for sale to the highest bidder. People keep watching NBC's Dateline, yet most people know they faked the whole Chevrolet exploding gas tank report.

60 Minutes has told so many lies it is hard to imagine they have any credibility, yet it remains a very popular program. I think most people know WWE is scripted, but it still remains very popular. I think CR is biased towards certain makes, but I still enjoy reading it. Heck I often even agree with their opinions. I thought the Samurai was a death trap, but I also thought CR when over the line in trying to prove it.

Ed

Particularly when considers the Jeep, from day one, had the exact same problem, during extreme maneuvers, because of its similar short wheelbase. ;)

mike

Yep. I'm not saying they don't have value to anyone, only that they don't have value to me.

I'm not ignoring that at all. It isn't relavent to the fact that this isn't simply an opinion poll.

It is statistically significant because the results are consistant enough to prove it. You failed to show any "clear bias" on the part of the poll-takers.

Brent didn't say that. He said "It [sic] clearly an opinion survey." Even if he had said that, the results show the statement to be untrue.

I didn't say much of anything in this thread and did not even mention the freaking survey, both of you are confused.

What I did post on was the religious following of CR where the faithful just ignore the obvious flaws in method and buy into the nonsense that people at CR have some god-like lack of bias.

CR at best is like a testing department run amuck that doesn't have to correlate its tests to actual use. At worst it's about as good as usenet in gathering reliablity information.

I never said it was a hard data gathering exercise, just that it's not simply an opinion poll.

How do you determine which questions are the "main" ones?

I said so in my post.

The only thing that makes it "chief information" is your opinion, which is clearly biased. I mentioned that I was referring specifically to the reliability data--which means the only relavent questions in the poll are factual ones.

There's a problem with your analogy. If the poll results included the average age and racial data of the persons taking the poll, and some people were only interested in the average age and race of people who were polled rather than what they thought of GWB, then absolutely those are hard data questions pertaining to the information at which those folks are looking.

Yes. I trust the law of averages because as I mentioned before, the results have enough internal consistency to be believed. When two different makes and models of a car get rated, the results are usually pretty similar if not always exactly the same. If they were at different ends of the spectrum, that would be another thing altogether. I expect some variation because of the small sample sizes, however if you look at general trends you get a picture based on more data.

Your basic argument is that the owners of certain companies' cars will use a significantly more restrictive criteria for determining whether or not a repair problem is "serious." You haven't provided any evidence or reasoning that support your claim, IMO.

Does CR express the opinion that certain cars are very reliable? From what I've seen, they don't. They might mention that a car is expected to be reliable based on the poll results, maybe that's what you mean. But then we get to a case of the chicken or the egg. Did CR first tell its readers that certain cars are more reliable and then start sending out polls? Or did they send out the polls and then mention the results? I don't have any issues from before they started collecting repair data from readers, so I don't know.

That's your opinion, but it doesn't make sense to me. It would make more sense to me that if people shell out money for a car that they are EXPECTING to be ultra-reliable and then end up having to repair it, they'll report even the most *minor* incidents as serious.

I would agree with you there and to some extent what you are saying based on that point. Expecting people being polled to use their own judgement on "serious" repairs is a bad way to collect data, because when you ask judgement-based questions, the smaller samples are bound to be less accurate. However I disagree with your other conclusion, which is that the results would be skewed in favor of certain brands and/or against others. I think they would just have a wider range of accuracy (meaning less reliable.)

Yep, I agree again--I want to see:

1. A much larger survey response.
2. No relying on opinion or judgement questions! They should be asking specific questions like "What was repaired" and "How much did the repair cost."
3. The results should reflect the actual numbers. For example, what percentage of owners of 1997 Corollas with 100,000 to 130,000 miles on the vehicle had to repair their vehicle's transmission last year, and on average how much was spent per repair job?

I'm saying the polling and reporting procedures are flawed, not that they don't provide a reasonably reliable (if somewhat vague) indication of a car's dependability.

Can you give me an example of a Buick and its twin Olds vehicle that this happened with? No offense, but I would like to look it up myself--I am pretty sure I can find past issues of CR frombefore they started combining the data for similar models.

By the way, are you sure they did start blending differently-branded instances of the same basic vehicle? I notice that the Vibe is separate from the Matrix in the reliability records, and from what I understand they are basically the same vehicle.

I haven't seen that, but then I don't usually pay attention to statements like that, so they very well could. However, I believe they used to say that about Mercedes-Benz autos as well, because they usually topped the CR reliability records... yet now, you find Mercedes cars at the bottom of the reliability charts, while Toyotas have sonsistently remained at the top. Unless CR started telling Mercedes owners that their cars were very unreliable, this would indicate that people don't tend to minimize their repair problems based on CR reliability reports, otherwise Mercedes owners would still be considering most or all their repairs to be "not serious" and not reporting them.

Good point about the goofy circles. However, they also list percentages--at least on their web site.

Yep. I never said they weren't. However, they aren't "opinion pieces" because they also contain factual data, like "the rear seats fold all the way down to make a completely flat cargo area" etc. An opinion piece would not have any use for statements like that. They are reviews, which contain a mix of factual data and descriptions of their experience, which have to be opinions. For example, you can measure how loud engine noise is, but you can't measure how an engine *sounds* or how a shifter *feels* in any way that would make sense to people. You have to use your best judgement, which is of course opinion. As I've said, I have found that their descriptions of such things tends to agree with what I have found. If they say one car has an engine sound that's "rough" and don't say that about another car, it's obviously a judgement but again I have in the past agreed with most of their judgements so I

That's yet another opinion. I'd say you're naive if you think they do, because CR isn't any newspaper/magazing/TV program etc--their existence is based on the trust that they are truthful and objective. If they break that trust, no one has any reason to buy their magazine, subscribe to their web site, etc--and this isn't the case with most newspapers, magazines, TV programs, etc--it's the opposite for them in fact. People pick which news station, newspaper, fashion magazine, etc based on it telling what they want to hear--but with CR it's different. The purpose of CR is to give the reader information which will help them spend their money on purchases they won't later regret making. And that's pretty much all it's for.

Some do, some don't, most don't care--they just want to see their own car praised so they can feel smart for buying it. If their car isn't one of the favored ones, then they don't get Motor Trend. I suppose some people might base their buying decisions on what MT says, but I feel sorry for them.

Right, because it's entertainment. It's supposed to be true, but even if it isn't, it's still interesting. CR is about the most boring, driest read you can buy. I have trouble forcing myself to even read the reviews of things I'm interested in. Without impeccable credibility, I'd pay quite a bit if I had to just to *avoid* reading CR.

You actually enjoy reading CR. I am astounded, flabbergasted, amazed and befuddled. I guess we all have our own tastes. I won't even consider that there enough people who enjoy reading CR that they could make it without credibility. I mean good grief, it would be like reading weather reports for places that don't exist.

I look forward to your response.

Except those vehicles built on the same assembly line sometimes ARE different. GM's Prizm, for example, didn't include suspension components that the Corolla did have. Same car but different. Does the Vibe come with AC-Delco parts or Nippon Denso parts?

And, once you've purchased it, your GM car is serviced by a GM dealer and your Toyota is serviced by Toyota. That could have an impact on satisfaction and it might have an impact on frequency of service, too.

There's also no reason to believe your contention, that the outlook of CR does anything besides select whoever reads the magazine. If a CR subscriber buys a Ford, he does it for a reason and he's just as invested in having a positive outcome to the transaction as a Toyota buyer. In other words, he's as likely to overlook problems as the Toyota purchaser.

The percentage of CR readers who own domestic cars might be lower than the percentage of domestic owners across the US as a whole but as long as they get a decent sample size, they can still evaluate satisfaction.

And, for those that were whining about statistical sampling, determining sample size is just plain math. If sampling wasn't a valid way of assessing quality, pretty much every manufacturer's QC operation would turn out the lights and go home.

The Detroit Fan Club can piss and moan about CR and other surveys all they like but the fact of the matter is, Toyota and Honda are doing SOMETHING to make their customers believe in them and it's paying off in $$$.

The selection is very important, not just the sample size. The CR survey is asking a bunch of more or less like-minded people (afterall, those who don't like CR for whatever reason are not subscribing to it generally speaking) what they think if they want to share their thoughts. It is like sending out a questionaire to neo-cons asking how Bush Jr is doing as president. They can fill it out or just toss it in the trash. Odds are the results are going to be skewed.

I apologize for attributing to you a quote that you didn't make.

I'm sure there are people like that. Obviously they are wrong--everyone is biased. CR reviews are like any other. It doesn't matter whether or not the reviewer will like the product being reviewed. What matters is whether or not the person reading the review gets an accurate picture of how they'll feel about the product.

CR can rave about a product all they like, but if it doesn't excel at the things you want it to excel at and in the correct way, you won't be happy with it. I'm sure you know this, but I'm hoping that if any of the CR faithful are reading this it might switch on a light bulb so to speak.

I like CR because it gives me information which I find useful in helping me decide which product to buy. For example, I bought a certain camera not because they recommended it but because it had the features I wanted, such as low shutter lag (which they measured to the tenth of a second at .3 seconds, and as far as I can tell that's accurate.) It saved me the trouble of having to go to several different stores and asking them to unchain each of the demo models, put batteries in, supply it with a memory card if necessary, and let me take some photos. That would have taken hours at least, and I would have had to deal with salespeople, which I often find frustrating and sometimes infuriating. Some stores probably have a "you-cram-it-up-someone's-ass, you-bought-it" policy, and I don't want a camera that's been up anyone's ass, even if I'm the one who put it there.

Well, every time I've used their info, I've found it to be accurate. However, I've only used their info about probably 30 times or so, and they have done thousands and thousands of reviews--so maybe I've been lucky with the thirty I used.

I don't trust CR implicitly, though. If they start providing information that I find to be incorrect, I won't hesitate to let as many people know about it as I can. I will give *specific* examples that they can look up for themselves, though, rather than making vague statements about "the religious following of CR" because I don't want to look like a fanatic member of the anti-CR religion.