I just completed my 2007 Consumer Reports Subscriber Survey. I though people who don't participate might like to see the sort of questions that are used to create the results reported in the magazine..
*********************** If you had any problems with your car in the past 12 months that you considered SERIOUS because of cost, failure, safety or downtime, select the appropriate box(es) for each car. INCLUDE problems covered by warranty. DO NOT INCLUDE: 1) problems resulting from accident damage; 2) recalls; or 3) replacement of normal maintenance items (brake pads, batteries) unless they were replaced much sooner or more often than expected.
Considering all factors (price, performance, reliability, comfort, enjoyment, etc.), would you get this car if you had it to do all over again?
How satisfied are you with this vehicle with respect to each of the following factors?
Did this vehicle get any MAINTENANCE (e.g., oil change/lube/tire rotation) or REPAIRS in the past 12 months?
Was any work done on this vehicle in the past 12 months (excluding tire replacement or accident damage) covered by a warranty or by a recall notice? Select ALL that apply.
Which, if any, of the following performed the maintenance and repairs? Select ALL that apply.
New car dealer (which sells your make of car) Independent local service station Repair or service chain such as Jiffy-Lube, Midas Other
How satisfied were you with the service for each of the following?
New car dealer (which sells your make of car) Independent local service station Repair or service chain such as Jiffy-Lube, Midas Other
***********************
If your cars actually had a problem (and neither of mine did), you get additional question about the area affected, but it all depends on what the respondent considers significant and they don't collect cost data. Yet from this limited questionnaire, that is only sent to Consumer Reports subscribers, CR generates "data" that million use to make buying decisions.
I consider the information generated by Consumer Reports more reliable than a news poll of 1200 people deciding who the leading candidates are in each party or 50 people interviewed after voting to decide which way that polling place has voted. Consumer Reports has never let me down, but some of these other pollers sure have!
Who best able to determine whether they would be willing to buy that car again? All of those questions are subjective indeed but it ultimately comes down to the impression that vehicle made on the owner. Remember that the data is viewed in a cumulative fashion, a few stupid responses will not skew the results for the vehicle's overall reliability rating.
If you are suspicious of the questionaire's methods how do you explain the good ratings that some Ford and GM vehicles received when the overall impression of those brands has been in the dumps for the past many years? Everyone "knows" Japanese cars are superior in technology and reliability these days yet we are seeing the domestic brands sans Chrysler begin to hold their own in the ratings and in the general opinion of the consumer.
Considering my personal experience with domestic brands I would never buy a Ford or Chrysler again but the recent reports from Consumer Union has me actually thinking I may look at domestic brands in the near future. I certainly want them to do well but I will not sacrifice my hard-earned money to buy something that will nickel-and-dime me constantly over the next ten years. Pretty soon you and Ford/GM will begin spouting CR ratings as evidence of how well American cars are made to appear credible to the average consumer.
I've relied on CR for years & have been happy. A friend's daughter interned at their headquarters a couple of years ago & was super-impressed with the thoroughness of their various & sundry testings. Their info is not funded in any way by the corporations which produce the items which they test, & the people who own the items give extensive feedback. Personally, I don't understand your complaint. If, OTOH, you think it's a waste & unreliable, then quit subscribing to CR. You'll be happier.
All I did was post the questions asked in CR's annual auto survey and comment that "from this limited questionnaire, that is only sent to Consumer Reports subscribers, CR generates "data" that million use to make buying decisions." Anything false in that statement? Is it even a complaint?
Do you believe that the few questions they asked of a very specific group (CR subscribers) provides hard data? It clearly an opinion survey of a specific group who's opinions are influenced by the group doing the surveying. If you want a real survey, with better data, you need to survey a random group of car owners and collect hard cost data. Depending on people to determine what problems are serious leaves a lot up in the air. I consider a problem "serious" if I have to have it fixed no matter how little it costs and I want almost everything wrong with a car "fixed." I consider a rock strike in the windshield "serious." My Sister has been riding around with a cracked windshield for three years. She doesn't think it is serious. The paint has fallen off the bumpers of her car - I'd be freaking out if mine did that - she doesn't care. One of her sun visors fell off - again she didn't care. A guy I work with constantly complained that the rear end of his pick-up truck hummed. I rode with him and never heard a thing. Eventually he got rid of the truck because it was so annoying - to him.
I guess the point I am trying to make is this - People who read CR and follow CR's advice have a tendency to share the opinions of the CR editorial staff. Surveying that group is likely to confirm the opinions of CRs editors.
I enjoy reading Consumer Reports. The OPINIONS of the editorial staff are interesting. Sometimes I agree with them, sometimes I don't. I think they do a good job of collecting hard data (0-60 times, braking distances, interior volume, etc.), but I have a lot less faith in the subscriber opinion survey results.
Even though CR "is not funded in any way by the corporations which produce the items," they still need to generate revenue. To do this they must create a magazine that attracts subscribers and contributors. They have a constituent they need to please just as surely as if they did accept funds from corporations.
Why do you think expressing an opinion is complaining? And why would I be a lot happier if I did not express my opinions? You do not have to agree with me, but I think it is a good thing to listen to and considered the opinions of others. I actually consider the results of the CR opinion survey, but I don't consider it to be hard data. To me, the survey results are just another opinion to be considered.
The CR true believers cannot be swayed by actual science and proper methods. It's a faith. What really gets me is that they actually think that a lack of advertising makes CR unbiased. As if bias cannot form without advertising money. They have their opinions and biases, it was pretty clear from the articles I read many years ago.
I think Brent has the right idea. I read CR (at the library, sometimes) with some faith and trust in their opinions, but sometimes I sense a feeling of "we'll let this one skate" but "I didn't like the feeling of the seat in that" in their articles. They seem to ignore rather obvious flaws in auto design, (like a plugugly car) and focus on the fact that the cupholders are mounted at an unfortunate angle for left-handed albino dwarfs. Their opinions are a bit too subjective for my liking. I stopped thinking of CR as gospel about 53 years ago, when they compared a Chrysler Windsor with a Hudson, saying the Chrysler's body was a little torquey, and the Hudson was more rigid. They recommended the Hudson over the Chrysler, totally ignoring the fact that at the time, if you bought a Hudson, you must realize that at trade-in time, you'd better buy another Hudson, because your trade-in value at any other car dealer would be nickels and dimes. CR overall is a good guide to avoid junk, but in a lot of cases, by the time their ratings are published, the products are a year or more old, and some are discontinued at retail outlets, and the models that supersede the products may or may not be similar in features and quality.
You've said that before and you're wrong. It is NOT "clearly an opinion survey." It is a survey which asks SOME questions which elicit answers based on opinions, some questions which require answers based on factual numbers, and some which require answers based on a combination of facts and judgments. To say it is "clearly an opinion survey" shows your own strong bias against their surveys: you're focusing ONLY on the part of the survey that is about opinion and completely ignoring the part which is about facts.
For example, the questions "Was any work done on this vehicle in the past 12 months (excluding tire replacement or accident damage) covered by a warranty or by a recall notice?" and "Did this vehicle get any MAINTENANCE (e.g., oil change/lube/tire rotation) or REPAIRS in the past 12 months?" are not asking for any opinion at all.
Personally, I don't find any use for the opinion portions "are you satisfied/would you buy again" etc. I look at the reliability data based on repair reports. That's the part of the poll to which I refer from this point on.
o> f a specific group who's opinions are influenced by the group
I agree that would be better because of the hard cost data. Since you seem to be implying that CR readers would answer poll questions favorably towards some cars and unfavorable towards others in a consistent enough way to skew the results, it would be better to have a random set of car owners, not because it makes sense that CR readers as a whole would be biased toward or against any particular nation's cars but so that people like you can't say the test is skewed because the owners who were polled all read the same magazine.
`Your opinion means nothing. My opinion means nothing. When you take hundreds of opinions and average them out, you have something. For example, you take a 1.5 pound weight, and ask one person how heavy it is, he might get it right, but he could be way off. However if you ask hundreds of people how much it weighs and average out the answers, you'll have an answer that is remarkably close to 1.5 pounds. Unfortunately I can't remember what book I read about this, so I can't provide any hard evidence. If it makes sense you can accept it, if not then disregard. Naturally, a larger sample would provide more reliable data. That's the biggest caveat I have about the CR surveys--the samples are far too small.
I have three questions.
On what evidence is this belief based?
How would the survey takers know the opinions about the reliability of a given auto that the CR editors have? As far as I've seen, the CR editors don't state their opinions about the reliability of autos--they simply publish the survey results and let them to the talking.
Even if the people answering the surveys knew the opinions of the CR editors and tended to share them, how would that affect their poll answers? Is it like they'd think, "Well, my engine in mt new Toyota fell out, but the CR editors think Toyotas are reliable, therefore it must not have been a serious problem, so I won't report it on the poll." Is that what you're suggesting? Again, I'm referring to the reliability reports, NOT the owner satisfaction part of the poll.
I never read the opinion pieces, unless you consider the reviews opinion pieces. I don't necessarily like what they like, but the information I've gotten from them has always proven helpful. If they say orange juice A is sweeter than orange juice B and therefore tastes better, I ignore the part OJ A being better and buy OJ B if I like my OJ less sweet.
Every time I've read one of their reviews on a product which I already owned or later purchased, they had the facts straight even if I disagreed with their conclusions--and the facts are what I pay attention to.
If it were ever revealed that anyone at CR was skewing their ratings towards what they thought would please their readers and in the process skewing the ratings away from the truth, how long do you think it would take for them to go out of business? There'd be no reason to buy their magazine or send them money for anything after that.
Actually it is likely that most CR readers are independent thinkers like you and when filling out the survey will ignore the temptation to confirm CR editorial opinions.
The opinion questions have some value. For example Consumer Reports discovered that people loved their mini's and would buy them again even though they are unreliable cars.
And you're ignoring the facts that a) the sample group is entirely self-selected, and b) even if the sample of CR subscribers were done randomly, it's still not statistically significant because of the limited (and clearly biased) sample population.
Brent is right - the results are no more meaningful than an opinion survey.
I stopped putting much stock in their car ratings when they went thru a cycle of how they rated cars. For a few years, maybe a decade or more ago, they starting including a cost index of some sort to go along with the trouble index. So that Mercedes that looked like it rarely had any problems suddenly got black dots for it's cost of repair index. Some of their other favorites started to get those black dots too. Some of the domestics that had not quite as good of scores in the trouble index showed up pretty good in the cost index.
That only lasted a couple years and they printed some half assed nonsense about how it wasn't really a good way to evaluate vehicles and dropped that rating. There is absolutely no doubt in my mind that both the editors and the readers of CR have an anti-US bias.
I don't follow that logic at all. They won't be subscribing to CU if they don't more or less agree with the editorial stance of the magazine. For example, the magazine has pushed many social programs for the past decade - to the point where they lost some subscribers, such as myself, who didn't care to be lectured to by some bozo who's supposed to be testing clothes dryers for me. So whoever subscribes is statistically likely to be of a like mind to the CR editors on many subjects. Therefore, their survey WILL be biased. It's been noted many times that if you look at vehicles that are built on the same assembly line but given different "names", they will often have different ratings and virtually always the "American" version will get lower ratings then the "imported" version.
A few "hard data" questions in the survey doesn't turn the overall survey into a hard data gathering exercise. The main questions in the survey are clearly eliciting an opinion. Questions like - "If you had any problems with your car in the past 12 months that you considered SERIOUS because of cost, failure, safety or downtime, select the appropriate box(es)...." or "How satisfied are you with this vehicle with respect to each of the following factors?" The answers to these questions are opinions, not facts. The chief information gathered is all based on the opinions of the reader. This makes it "clearly an opinion survey." When the pollsters come around and ask me how GWB is doing, they also ask me my age and race. Just because they ask me some hard data questions, does that make my answers to the other questions facts?
You mean the data gathered where they asked the respondents to decide what is "SERIOUS?"
No, I am saying that CR readers tend to spit back the opinions of the magazine. They have been told repeatedly that car A is very reliable, so when they are asked to report SERIOUS problems with car A, they tend to think problems with such a reliable car had can't be serious. I am not saying it is a massive case of deliberate misreporting. Because of the limited nature of the survey, very small shifts can appears very significant when the data is reduced to the little circles. In my opinion, the data gathering method is unscientific and the reporting methods can make it appear that there are great differences when in fact, there may be no statistically valid differences at all between component systems of two different cars. This means that very subtle biases can skew the results and make it appear there are significant differences when in fact they don't actually exist.
It is not just that the samples are too small, they are both too small and from a non-random group.
For years Buicks have shown up better in reliability ratings (like JD Powers) than other GM products (at times even better than Cadillac). Why? In most cases the basic parts of the car are exactly the same as Pontiacs and Oldsmobiles and they are built on the same assembly lines by the same workers. There is no reason to believe that Buicks are more reliable than Oldsmobiles, but if you look at JD Powers surveys, Buicks always come out much better. This also used to be the case with CR surveys, but in recent years it appears that CR started blending the data for similar cars sold by different divisions to avoid having to explain this phenomenon.
You have to be kidding. When a new Toyota comes out the CR editors will say they expect it to be reliable. When a new Buick comes out they will say it is a new model and the reliability is unknown.
The reliability reports are based on the respondents deciding what is "SERIOUS." This is an opinion based question. Cars from all manufacturers have very few problems of any type (according to JD Powers the average new car has 1.24 problems, and the average 3 year old car has 2.27 problems). It only takes a subtle shift in determining what is "SERIOUS" to make large differences in the survey. For all I know, Toyota owners might be overly picky and over report problems compared to Buick owners. The CR reporting methods (the little circles) can make tiny differences appear to be very significant.
The reviews are full of opinions. For instance:
"This large, front-wheel-drive four-door sedan replaced the LeSabre and the Park Avenue. The standard power plant is a rough-sounding
3.8-liter V6. A potent 4.6-liter, 275-hp V8 powers the CXS. The four-speed automatic transmission shifts smoothly enough. The Lucerne has a quiet, comfortable ride, especially at low speeds. Handling is not agile and the steering lacks feedback. It has a tendency to fishtail easily at its limits. Stability control is only available on the CXS V8. Braking is unimpressive. The back seat is roomier than the one in the LaCrosse. First-year reliability has been above average."
Opinions: "rough-sounding 3.8-liter V6" "the four-speed automatic transmission shifts smoothly enough" "has a quiet, comfortable ride" "Braking is unimpressive"
or
"The Mercury Milan and similar Ford Fusion are new midsized sedans that are very agile and satisfying to drive. These sedans rate even higher in our testing than the Mazda6 sedan on which they're based. They have a sporty feel; the V6 and automatic transmission make a smooth and responsive powertrain. The four-cylinder is a bit coarse. The interior is well made and space is generous."
Opinions: "are new midsized sedans that are very agile and satisfying to drive" "They have a sporty feel" "The four-cylinder is a bit coarse" "The interior is well made and space is generous"
or
"The redesigned Toyota Camry is roomy, quiet, has a comfortable ride, and is refined. The addition of a telescoping steering column is a plus. Power comes from a strong 3.5-liter V6 mated to a six-speed automatic transmission. It returns 23 mpg overall, just one mpg less than the four-cylinder. The base 2.4-liter four-cylinder is also responsive and relatively refined. A four-cylinder hybrid version returned an impressive 34 mpg overall. Handling is responsive and secure but not sporty. The interior is spacious, with reclining rear seats in the high-end XLE. Curtain air bags are standard, but stability control remains optional."
Opinions: "is roomy, quiet, has a comfortable ride, and is refined" "......a strong 3.5-liter V6" "The base 2.4-liter four-cylinder is also responsive and relatively refined" "Handling is responsive and secure but not sporty" "The interior is spacious"
etc., etc., etc.
The reviews are full of opinions.
You don't think the whole continual retesting of the Suzuki Samurai wasn't an attempt to skew the ratings? Every newspaper / magazine / TV news program / etc that I am aware of skews their content to please / attract the readers / viewers. You are naive if you don't think CR does as well.
People keep buying Motor Trend, and I am sure most of the people who buy it understand that "Car / Truck / SUV of the Year" is up for sale to the highest bidder. People keep watching NBC's Dateline, yet most people know they faked the whole Chevrolet exploding gas tank report.
60 Minutes has told so many lies it is hard to imagine they have any credibility, yet it remains a very popular program. I think most people know WWE is scripted, but it still remains very popular. I think CR is biased towards certain makes, but I still enjoy reading it. Heck I often even agree with their opinions. I thought the Samurai was a death trap, but I also thought CR when over the line in trying to prove it.
MotorsForum website is not affiliated with any of the manufacturers or service providers discussed here.
All logos and trade names are the property of their respective owners.