Sherrill Mane: December 2010 Archives

The Very Visible Consequences of Bad Methodology

| | Comments
metrics-blog-header.gif

For the record, interactive media are growing and transforming the way people communicate with brands, with content and with each other. What interactive media do not need is badly executed and poorly reported research. The media themselves already resonate with consumers and the marketing ecosystem.

What specific piece of poorly executed research prompted this? On December 13, Forrester released its North American Technographics Report to great fanfare. Much was made of a finding that simply does not synch up with everything we know using other data sources. According to Forrester, consumers reported that they spent 13 hours a week watching TV and 13 hours a week using the internet in January and February of 2010. Fanfare surrounding the purported parity was nearly matched by the furor over the fact that all other data sources show nothing of the kind. For the same time period, comScore reported that people spend 7 hours and 24 minutes a week using the internet.

The IAB does not need to shill for TV so the fact that Nielsen reports that TV viewing levels are in excess of 30 hours per week is not the point. The point is that we, the IAB and the overwhelming majority of members, stand for transparency and consistency in measurement. We sincerely believe that good measurement must be the barometer against which all media are evaluated.

Putting out numbers that confuse the already confused marketplace—numbers that are based on a methodology that every expert would say is inferior for the purposes of measuring media usage—serves no one. Self-reported usage data rely on consumers’ memories and on their desire to disclose what they do with media. Everyone knows that most people are using the internet at work and spending significant time doing so. We all know that being a “couch potato” is not perceived to be as cool as using new media. We know that people watch TV content on digital media. And, we know that most people are not sitting around making many of the distinctions about media that the professionals do. So why would we expect that people can accurately report their media usage using our artificial business distinctions?

A really thorough review of the Forrester study would also require thinking about the sample. Is it representative of broad population groups? Can the findings be generalized beyond the sample? These complexities are beyond the scope of today’s blog.

Planning and buying media for the purposes of marketing to consumers relies on what people do, not on what they say. To those who continue to confuse the marketplace, we say please consider the disservice you do to all of us. To those who accept counterintuitive information, we say caveat emptor.

Sherrill Mane is Senior Vice President for Industry Services at the IAB.