Recently in Metrics Category

If you follow the world of media and advertising research, you may have heard something about a debate set off by last week’s release of an IAB commissioned report entitled, “An Evaluation of Methods Used to Assess the Effectiveness of Advertising on the Internet.” If you’re not a media or advertising researcher, you probably skipped over the Twitter stream, the trade press and the numerous blog posts about the paper. As a digital salesperson or business development executive or agency chief or CMO, you likely figured it had nothing to do with you.

That assumption would be incorrect. This methodological review, a completely independent one, at that, conducted by the esteemed social scientist, Dr. Paul Lavrakas, has everything to do with you and your company. Here’s why…

With permission from Dan Murphy, SVP Interactive Research & Ad Traffic, Univision Interactive Media, we cite the following. “The IAB study was initiated a while back because publishers were being inundated with large numbers of studies – in some cases so many, users were getting multiple survey requests on the same User Session. (Pop-Up blockers anyone?) If the publisher didn’t accept the study, they were told – No Campaign. Putting the issue of recruiting sample with a pop-up window (i.e., a variation of a pop-up ad) to the side, many had questions about sample frame issues from Power-Users, Shared Computers, Users of various bandwidths and so forth. Concurrently, many of us (both buy and sell-side) were seeing peculiarities in the output.”

Simply put, the independent review was taken on to solve a business problem that had become more pressing over time. Should the industry be forced to continue to use research that may or may not be doing the job in order to be included on a buy? Does the research need to be improved or changed in order to preserve the quality of the consumer experience? After all, if as an industry, we do not make the consumer the center of our universe, what kind of businesses will we have? And, lastly, what guidance is being given to the agency staffers who are forcing these studies upon the sellers?

No research for any medium is perfect. We at the IAB did not expect perfection when we commissioned Dr. Lavrakas. We got what we expected: an objective evaluation that clearly expresses confidence in what is good, critiques what is bad and goes on to suggest steps for improving site intercept ad effectiveness research. In addition, the IAB and our member companies are committed to using the paper as the foundation for Best Practices work that force both buyers and sellers to be wiser about how the studies are used.

It is time that as an industry, we invested in making ad effectiveness research better; in moving from ad hoc solutions that sell well to buyers who may not understand of the limitations of studies; in truly building a body of work that proves how well the online media build brands. We cannot get there if we continue to ignore the need to think big, get out of our silos and recognize that we must raise the standards, the practices, the discourse itself to a new level.

Let’s embark upon learning and applying the learnings of Dr. Lavrakas’s work to solutions that make online media attractive destinations for consumers and for brand advertising dollars. And, in the spirit of collaboration shown by the vendors who let us look under the hood for this report, let’s move forward together so that we really can get to the next level.

Interactive media are blessed with a plethora of metrics: a wide variety of measures, and also a wide variety of measurers. Of course, as everyone also knows, blessings can also be curses. The curses of complexity and confusion that are digital media’s daily measurement diet spring up in everything we do as sellers and buyers of media.

The chaos is particularly blatant when one looks at unique user rankers. We don’t often see a pair of rankers from comScore and Nielsen side by side, and the view is a little disquieting. For example, take top news and information sites for March 2010. Only 7 of the top 10 properties are common across both comScore and Nielsen, and eight of the properties that appear on the comScore top-20 list do not appear on Nielsen’s.
Differences in site roll-ups and how sites are categorized have a huge impact in the rankings. For example Nielsen splits About.com and NY Times.com apart, while comScore rolls them up into New York Times Digital. Nielsen considers Wikipedia a “news and information” site (and the top one at that), while comScore does not. News is a particularly problematic category in terms of fluidity of definitions. But pick any category and stack the rankers side by side, and you’ll see a similar pattern of disparity.

Why does this matter? Syndicated audience measurement is one of the most common tools for planning and buying ads—and as this example shows, planners thinking about a basic news reach buy would make significantly different decisions depending on which vendor’s data they’re looking at.

What’s the IAB doing about this? The IAB’s Nomenclature project is a longstanding effort to establish common definitions around site categories, and some guidelines regarding rolling up sites into larger entities, which all measurement vendors could follow. Nothing will bring the numbers perfectly in line with one another, but this effort should reduce the level of the discrepancies to one that is less chaotic and easier to live with. It’s been tough to establish compromises around something as competitively sensitive as rankings. But we hope to see progress this year, and we’re grateful to everyone, vendors and publishers alike, who has contributed to the process.

What else is the IAB doing about this? The IAB is working across the ecosystem to tackle the problem of measurement in a very fundamental way, above and beyond fixing nomenclature. Measurement is a business process that is implemented by research scientists. Unless and until the business leaders from across the ecosystem define their business requirements and engage with each other on precisely what they need to know in order to manage their businesses better, the dichotomy of blessing and curse and attendant measurement confusion will continue to plague digital marketing and advertising.

As we close out the year, take stock of accomplishments and prepare for better days ahead, one of the few constants in the rapidly evolving interactive media world remains measurement troubles. Quite recently (Monday, December 7th), Ad Age featured a piece about a proposed change to Nielsen’s @Plan service and the vast data inconsistencies that change is causing compared to the old methodology.

The @Plan methodology change, now in beta, should be heralded as an improvement. It is fairly common knowledge that unobtrusive measures of behavior are more accurate than recall based measures. However, the reported inconsistencies tell us that something is really wrong here. Most likely, the current method is completely wrong in which case 10 years of data use and decisions based on that data are questionable. Maybe the new methodology is wrong or maybe it is actually right. Or, the beta is so full of bugs that the new data cannot be relied upon to provide an indication of what the numbers will look like. The uncertainty points to fundamentally flawed business approaches.

Buggy business, you say? Yes. One of the two syndicated measurement services introduces a beta of a new methodology and its clients are already leaking results to the press. The proposed @Plan change is sowing negativity even before it is released. What resonates is not the beta and not that it is likely an effort to improve measurements and not that the @Plan clients are leaking data from a test. What resonates is that interactive measurement sucks. Once again, as an industry, we are showing the advertising community that we are our own worst enemy.

The IAB represents the entire industry and advocates transparency among all parties. Clearly, some of the old rules of media measurement simply do not apply in a transparent world. Research vendors and their clients are accountable to all of the many sites that are not purchasing data when they change methodologies and talk to the press.

It is time that IAB members took advantage of the power of a community of business people who truly want to make interactive media live up to its promise of being most measurable and most accountable. We are that community. We, at the IAB, are who we serve and until the members, publishers and vendors alike, avail themselves of the leadership and power of community, we likely will continue to see public relations fiascos about flawed measurement well into 2010 and beyond.

This inaugural edition of IAB Metrics Blog invites you to dialogue with us, to praise or critique, to propose solutions. Most of all, we invite you to make the IAB the standard bearer in metrics that it is for many other areas of interactive advertising.

Anyone who still thinks that contextually targeted online video doesn't have impact ought to have been in my shoes last week. My appearance on "3 Minute Ad Age" beneath the headline "IAB CEO Rants Against Audience-Measurement Complexity," tore up my email box as well as the advertising-obsessed quadrants of the blogosphere.

While enormous credit goes to Ad Age editor Hoag Levin -- who knew the hot word "rant" would be a magnet to blogging, traffic-driving controversialists -- I believe my generally mild charge struck a chord because it reflected the daily reality of marketers, agencies, and publishers who are weighed under by the numbing complexity of selling, planning, buying, placing, and measuring advertising.

You can watch the video, but here, in essence, is what I said:

Measurement is not just a science; it is also an essential business process. Yes, measurement aims to uncover truths about the brands, products, services, and consumers with stakes in an advertising campaign, but those truths are worthless unless they improve -- unless they add value to -- the companies and customers on either end of the chain.

Our problem is that, right now, the marketing-media ecosystem has been so consumed by the equivalent of an angels-dancing-on-the-head-of-a-pin debate, that we make the transacting of advertising business more difficult than it needs to be. Measurement, as I told the crowd at the IAB's Audience Measurement Leadership Forum last week, isn't the cause of this complexity crisis, but it's certainly a major ingredient.

Physician, Heal Thyself

An apt analogy is modern medical science. Advances in molecular biology, diagnostics, neuroscience, and technology and instrumentation have revolutionized our understanding of how the body works and malfunctions. But were it not linked to the practice of medicine in ways that allowed physicians to cure peoples' ills, we'd consider it a failure.

That's unfortunately where we are with so much of modern media measurement: The science is diverging from -- even divorcing itself from -- the practice. We can measure so many things, in so many ways, that the actual customer -- the marketer -- is tuning out. The McKinsey & Co. study I referenced last week, "How Poor Metrics Undermine Digital Marketing," is sad proof of this undeniable reality. "Companies have failed to crack the code for measurement," the consulting firm concluded from its survey of 340 marketers around the world. Comparing its findings to a similar study done in 2007, McKinsey found that "most companies have made little progress in this area."

Who's at fault? Let's ask, rather, what's at fault. Interactive media is an industry with historically low barriers to entry. Moreover, its delivery is based upon the almost continual exchange of bits and bytes of information. Pretty much by definition, that means the more media we create, the more data out there that can be measured -- and the more opportunities to measure them. Pageviews, hits, clicks, time spent, pre-roll completion -- the media glut almost pre-determines a measurement glut.

The IAB was created to bring form to this chaos, and has done a fine job of coalescing all parts of our industry around standards and guidelines that create the definitional consistency that make business simpler to conduct. Last week, we issued for public comment our Audience Reach Measurement Guidelines. Forty-four companies -- publishers such as the Wall Street Journal and the Weather Channel, researchers such as Scarborough and Millward Brown, auditing experts such as Price Waterhouse Coopers and the Audit Bureau of Circulations, giants like Google and Microsoft, and newer innovators like YuMe Networks -- came together to find the simplicity on the other side of complexity.