George Ivie, Executive Director and CEO,
The Media Rating Council
George Ivie, Executive Director and CEO of the MRC
Nielsen//NetRatings and comScore, the two largest panel-based Internet audience measurement firms, have started undergoing audits of their methodologies. This represents a key milestone on the path to a better understanding of the metrics that drive the interactive industry. Both NetRatings and comScore selected the Media Rating Council to perform those audits.
IAB Director of Research Joe Laszlo recently sat down with George Ivie, Executive Director and CEO of the MRC, for some thoughts on the implications of the audits, and an update on their status.
JL: How do the audits of NetRatings and comScore relate to the larger issue of reconciling panel-based and server-based measurement?
GI: A big topic today is how panel measurement compares to census measurement, and how ad serving compares to publisher measurement. We’ve got all these differences in counts and measurements going on around the Internet. How does this world fit together and why are these all different? If we don’t know how the systems work and we haven’t validated that they do in fact work that way, then it’s extremely difficult to reconcile. We need to reach a phase in the audit where we have confidence in the knowledge that we have.
These numbers are not going to be in line with each other, necessarily. But if you can take the monkeys—the errors and inconsistencies—out of the process, and you can also understand the natural variability built into things like sampling, you can begin to reconcile why the numbers are different.
JL: What’s the current status of the audits?
GI: Both companies now have completed what’s called a pre-audit. A pre-audit is a preliminary evaluation that our CPAs make of how their systems work, not based on auditing but based on walkthrough and inquiry. It’s a quicker process than an audit that efficiently enables the rating service and MRC management to address known gaps in compliance with our standards.
With both companies through their pre-audits, we’re on to the next stage, which is the real audit. We’ve formed audit committees of our membership, which are the groups of organizations that will be receiving the audit results, reviewing them, and working with the MRC to determine whether accreditation is granted or not.
JL: What information are you looking for in the audit process?
GI: There are two types of information gathered in the audit. One type is the audit and the audit report itself, that’s conducted by CPAs. Each audit consists of multiple phases.
For example the phase that’s going on right now at both of the organizations is testing of the meter technology. There’s an audit scope that’s in writing, that’s been shared with the companies, about how we’re going to be testing their meter technology. In part we’re going to be testing it in the lab: we’re going to be seeing what operating systems it’s compatible with, the granularity of the data captured, how well that works, and how the data communication works.
Then we also have a set of testing that occurs in panel households: we’re testing what you might think of as production meters, not in a laboratory. For example, when we contact these panel households we might ask them to execute certain things on their computer and we trace back through the systems to see if their particular meters are communicating properly.
So there’s both lab and field testing going on in this first phase related to the meters. That’s the side of the audit done by the CPAs.
There’s also a whole different level of study as well. We’ve established what’s called a research study plan that’s unique for each of the two companies, since their methodologies differ. We put this plan in front of the rating company and we say we’d like you to do these studies, and report the results of these studies directly to the audit committee.
There’s a whole list of projects for each company in that regard, and those are being worked on now. In fact, Nielsen NetRatings has completed one of the very significant studies, and comScore is beginning work on this.
JL: What’s your sense of the timetable for these audits?
GI: My estimation is these audit processes will take over a year to execute. They’re very, very complex. And when you think of all the things we have to look at, the different panel structures, work versus home, weighting, the sampling aspects, how they maintain their panel and things like that, there’s a lot of complex stuff to absorb. The good news is where we see issues we communicate them in a timely way, and we expect that the measurement services, both of them, will work on these things in a timely way, too, so we don’t wait over a year for any improvement to take place.
And then when all the phases are completed, that’s when an accreditation vote will take place.