Bad Statistics II

Tops in the Dick Envy category

Simply, in a word, wow. Rarely has Blipverts seen such piss poor research methodology. Too bad Blackberries from Detroit to Nashville to Torrance must be in full meltdown mode after Consumer Reports ‘ automobile brand perception rankings (HT: The Truth About Cars).

Money observation:

Among new-car shoppers, safety (63 percent) is the most important consideration, followed by quality (58 percent).

However, Consumer Reports’ survey has nothing to do with sales. If it did, Volvo would hold the top selling brand spot.

According to Consumer Reports, 77% of the 1,720 respondents listed Volvo as the top brand in safety. Toyota was named by 21%. A look at the 2007 American passenger vehicle sales reveals that Volvo sold 106,213 vehicles. Toyota, by contrast, sold 2,291,648. Toyota out sold Volvo by a roughly 22:1 margin. For every one, ultra safe Volvo car, Toyota pushed 22 cars and trucks off the lot.

Safety is neither the most important consideration nor much of a consideration when Americans buy a car.

Consumer Reports chose not to provide many details about its methodology.

To learn about consumers’ car brand perceptions, the Consumer Reports National Research Center conducted a random, nationwide telephone survey from Dec. 6-10, 2007, contacting 2,037 adults. The survey data was collected from the 1,720 adults whose households own at least one car.

We are not sure where those 1,720 adults live. Imports such as Volvo sell better on the coasts than in the Midwest. Stratified sampling represents a better approach. In this approach, the sample includes more drivers from California and New York and fewer drivers from Iowa and North Dakota because of population differences. This approach assures the sample includes respondents from Iowa and North Dakota and in proportion to their respective percentage of the American populous.

Overall brand perception is an index calculated as the total number of times that the particular make was mentioned as an exemplar across all seven categories, divided by the total unaided mentions.

Hmm… a telephone survey that relies on unaided recall and possibly aided recall. At Blipverts, we guess the respondents were asked:

• When you think of a safe car, what brands come to mind?
• When you think of a car with many safety features, what brands come to mind?

Unaided recall represents a bogus response format that magazine and advertising agency folks love. The logic follows something like this: If people can think of the responses off the top of their head, then those responses represent what consumers know. Maybe consumers saw an advertisement for Toyota while they were completing the survey. Of, maybe they or their neighbor recently purchased a Toyota. Who knows? Many uncontrollable, external factors affect recall.

Category scores reflect the number of times that the particular make was mentioned as an exemplar of the particular attribute, again adjusted for awareness.

Huh? This last paragraph makes absolutely no sense. At Blipverts, we’re pretty sure Consumer Reports has resorted to the “baffle them with bullshit” approach.

Yes, Consumer Reports’ methodology represents a whole lot of feces piled 10 feet high. And it stinks.

Consumer Reports plucked seven categories, or dimensions, out of their assthin air. Why these seven? Why not include categories like Popularity or Dick Envy (Porsche finishes tops in this category)?

Do these seven factors even correspond to anything? By performing factor analysis, Consumer Reports may have determined if these dimensions appear relevant or provide insight.

The magazine lists the top 5 brands for each category. Why do the percentages add up to more than 100? Does this result from the overreliance of the average unaided exemplar mentions?

The Design/Style category brought gales of laughter from the Blipverts staff. How can a brand have a design or a style? Individual cars maintain a unique design and style. An entire brand? We call shenanigans.

Cadillac somehow finished third. Granted the XLR represents some interesting design work. The Escalade EXT represents more of GM’s outstanding use of badge engineering (see Chevy Avalanche).

Up until the current design for the Jetta, Volkswagen maintained some interesting lines. Now, the Jetta appears as if a German created a Corolla.

If anyone says they know or can even recall Saturn’s design or style, then they are lying. No one knows what a Saturn looks like.

How can you ask questions over the telephone about a visual? Consumer Reports used a medium that prevents the use of visual aids to ask a series of questions about visuals. What smells in here?

In addition, how were people anchoring their response when they were asked, “Name a car brand that has strong design and (or?) style?” What adjective or series of adjectives did the magazine use with design? Interesting? Pleasant? Did the magazine phrase it as “design AND style” or “design OR style”? Phrasing matters in survey methodology.

This survey has nothing to do with sales and everything to do with promotional activities. Of those surveyed, 49% ticked off Toyota and 26% rattled off Honda as tops for the Environmental Friendly/Green dimension. Toyota does produce the fuel sipping Prius (45 city / 48 highway) and the gas hogging Tundra (13/16). Similarly, Honda pushes the Civc Hybrid (40/45) and the Ridgeline (15/20) with a straight face. How will the environment fare when those batteries from Priuses and Civics end up in the landfill?

Those Blackberries must remain off. This survey does not reflect future sales or good research methods.