The Forgotten 34%

Would you trust research that ignored the views of a third of a country?  If we had a general election and prevented 34% of the country from voting would that be fair and representative?

Of course not.  It would be massively prejudiced and unreliable and you would insist on a rigorous method that leads to a trustworthy and robust result.

So why don’t we have the same high standards when it comes to media research?

Too many media organisations now pay only lip service to the goal of giving clients and the wider market the right information by spending the time to do the necessary research. Instead they are being seduced by the ease of the quick and dirty online survey.

Online surveys are the quickies of media research: hurried, meaningless and ending in dissatisfaction.

But the majority of media research is now conducted via online surveys. This is despite – according to UKOM’s most recent figures – 34% of the UK not using the internet in most months. That is an incredibly large audience – and, by extension, collective wallet – to ignore.

This is not a methodological issue, not an ideological one. I have nothing against online research done and used properly;  indeed we use it at Thinkbox. It certainly has its place in research.  When you are interested in what people online are doing it can be excellent.  But as a rigorous way of revealing what the nation is up to or thinks, it is dangerously unrepresentative, not just because the nation isn’t all on there to ask, but also because of a number of alarming factors that affect the representation of samples drawn from online panels.

Firstly, regardless of not representing the 34% not online, online surveys don’t even represent the 66% who are online. Not everyone who is online is invited to take part in a survey or panel, and not everyone who is invited then actually responds. Those who ultimately do join tend to be heavier internet users with time to kill, so there’s an in-built bias.

Ipsos estimates that the industry level for responses from invitations to join are typically 0.5% and below. Once that hurdle is cleared, participation levels are low (below 20%) and dependent on the task.

There are also huge and largely unknown areas of bias as panel providers use various sources of recruitment lists. This explains why different panels sometimes produce widely varying results.

And if this isn’t enough to put you off, researchers then put too much strain on weighting the data they get back from online samples so that it looks more like what they think it should look like to be representative.

Advertisers and agencies need to understand their consumers and markets. This is the only way to ensure they make and market things that people will want to buy and use. We ignore large swathes of the population because it is easier and cheaper to do so at our peril.

I worry that media research is in a crisis of convenience where headlines win and reality is irrelevant. It is too often unprincipled and unfit for purpose.  It is with this in mind that we have been working with Ipsos on face to face interviews with a properly representative national sample to find out their opinions of TV that aren’t available through any existing robust research source. We’re going to do this every year from now on and we’re calling it TV Nation. We’ll share the results soon. Until then, please treat research making grand claims for the country based on online surveys with a sackful of salt.

  • Koen Smeets

    Has TV viewing data not been – and still is – entirely based on panel research? It sounds hypocrite – and incorrect – to say that only face-to-face research can be used to produce nationwide facts. It all depends on the subject matter, the target group you are after and how you are going to interpret the data. Let’s not shoot wildly at non face-to-face research, Neil. Every technique has its pitfalls and strengths…

  • Neil Mortensen

    Koen, you
    misunderstand me. I’m not shooting wildly – that is what online surveys often


    I do not
    say face-to-face research is the only way to produce nationally representative
    results; I say online surveys are not able to do it for all the reasons I give.
    I also acknowledge in the blog that online surveys have their uses. I’ll go
    further and happily say that online surveys can show which way the wind might
    be blowing in terms of public opinion, but they can’t reliably measure the
    behaviour of the nation. Unless you know something I don’t, how can ignoring a
    third of the population ever be nationally representative (and that’s before we
    get to how unrepresentative online surveys are of the online population)?


    viewing in the UK is measured by BARB, widely regarded as the most rigorous
    media measurement in the world. It is indeed a panel, but not an online one and
    it is recruited to be as nationally representative as it possibly can be. Have
    a look at its website. Are you seriously comparing BARB to an online survey?