Would you trust research that ignored the views of a third of a country? If we had a general election and prevented 34% of the country from voting would that be fair and representative?
Of course not. It would be massively prejudiced and unreliable and you would insist on a rigorous method that leads to a trustworthy and robust result.
So why don’t we have the same high standards when it comes to media research?
Too many media organisations now pay only lip service to the goal of giving clients and the wider market the right information by spending the time to do the necessary research. Instead they are being seduced by the ease of the quick and dirty online survey.
Online surveys are the quickies of media research: hurried, meaningless and ending in dissatisfaction.
But the majority of media research is now conducted via online surveys. This is despite – according to UKOM’s most recent figures – 34% of the UK not using the internet in most months. That is an incredibly large audience – and, by extension, collective wallet – to ignore.
This is not a methodological issue, not an ideological one. I have nothing against online research done and used properly; indeed we use it at Thinkbox. It certainly has its place in research. When you are interested in what people online are doing it can be excellent. But as a rigorous way of revealing what the nation is up to or thinks, it is dangerously unrepresentative, not just because the nation isn’t all on there to ask, but also because of a number of alarming factors that affect the representation of samples drawn from online panels.
Firstly, regardless of not representing the 34% not online, online surveys don’t even represent the 66% who are online. Not everyone who is online is invited to take part in a survey or panel, and not everyone who is invited then actually responds. Those who ultimately do join tend to be heavier internet users with time to kill, so there’s an in-built bias.
Ipsos estimates that the industry level for responses from invitations to join are typically 0.5% and below. Once that hurdle is cleared, participation levels are low (below 20%) and dependent on the task.
There are also huge and largely unknown areas of bias as panel providers use various sources of recruitment lists. This explains why different panels sometimes produce widely varying results.
And if this isn’t enough to put you off, researchers then put too much strain on weighting the data they get back from online samples so that it looks more like what they think it should look like to be representative.
Advertisers and agencies need to understand their consumers and markets. This is the only way to ensure they make and market things that people will want to buy and use. We ignore large swathes of the population because it is easier and cheaper to do so at our peril.
I worry that media research is in a crisis of convenience where headlines win and reality is irrelevant. It is too often unprincipled and unfit for purpose. It is with this in mind that we have been working with Ipsos on face to face interviews with a properly representative national sample to find out their opinions of TV that aren’t available through any existing robust research source. We’re going to do this every year from now on and we’re calling it TV Nation. We’ll share the results soon. Until then, please treat research making grand claims for the country based on online surveys with a sackful of salt.