Metrics: April 2010 Archives

Will the Real Top 20 Please Stand Up?

| | Comments
metrics-blog-header.gif

Interactive media are blessed with a plethora of metrics: a wide variety of measures, and also a wide variety of measurers. Of course, as everyone also knows, blessings can also be curses. The curses of complexity and confusion that are digital media’s daily measurement diet spring up in everything we do as sellers and buyers of media.

The chaos is particularly blatant when one looks at unique user rankers. We don’t often see a pair of rankers from comScore and Nielsen side by side, and the view is a little disquieting. For example, take top news and information sites for March 2010. Only 7 of the top 10 properties are common across both comScore and Nielsen, and eight of the properties that appear on the comScore top-20 list do not appear on Nielsen’s.
Slide1.PNG Differences in site roll-ups and how sites are categorized have a huge impact in the rankings. For example Nielsen splits About.com and NY Times.com apart, while comScore rolls them up into New York Times Digital. Nielsen considers Wikipedia a “news and information” site (and the top one at that), while comScore does not. News is a particularly problematic category in terms of fluidity of definitions. But pick any category and stack the rankers side by side, and you’ll see a similar pattern of disparity.

Why does this matter? Syndicated audience measurement is one of the most common tools for planning and buying ads—and as this example shows, planners thinking about a basic news reach buy would make significantly different decisions depending on which vendor’s data they’re looking at.

What’s the IAB doing about this? The IAB’s Nomenclature project is a longstanding effort to establish common definitions around site categories, and some guidelines regarding rolling up sites into larger entities, which all measurement vendors could follow. Nothing will bring the numbers perfectly in line with one another, but this effort should reduce the level of the discrepancies to one that is less chaotic and easier to live with. It’s been tough to establish compromises around something as competitively sensitive as rankings. But we hope to see progress this year, and we’re grateful to everyone, vendors and publishers alike, who has contributed to the process.

What else is the IAB doing about this? The IAB is working across the ecosystem to tackle the problem of measurement in a very fundamental way, above and beyond fixing nomenclature. Measurement is a business process that is implemented by research scientists. Unless and until the business leaders from across the ecosystem define their business requirements and engage with each other on precisely what they need to know in order to manage their businesses better, the dichotomy of blessing and curse and attendant measurement confusion will continue to plague digital marketing and advertising.