How to understand why people are doing things not just what they’re doing

This essay is a write up by Matt Locke from a conversation with Kenyatta Cheese about
audience engagement and measurement.

Way before you start thinking about the technologies you use to measure your audience, you have to have a model for what your audience is doing. Admit it, most of us don’t do this – we start with the tools – Google Analytics, Parse.ly, Chartbeat or whatever – and then we build a story from the information we see coming out of their dashboards. We’re doing it exactly the wrong way round.

What our audience is doing, what we can measure, and the value or impact we want – these are not always the same thing.

The problem is this –  what our audience is doing, what we can measure, and the value or impact we want – these are not always the same thing. For a lot of the projects we work on at EA1, the audience is in one place, but the opportunities to monetize or produce value for them are somewhere else. And so we start off by looking at how we measure audiences in three layers – platforms, products and business goals.

We think of it as layers of a stack, and the decisions made at each layer is going to affect the other layers. For example, let’s say you have a subscription-based business where you’re measuring the number of downloads of an app, and the number of people who are paying customers. There’s one approach where you can treat all audiences the same, and assume that if you can just get in front of enough eyeballs, a certain percentage of those people will care enough to click through, download the app and pay. But if you look at the audience with a slightly different lens, you start to see that there are people who are more likely to want to buy, subscribe or engage with your media, and folks who are less likely to. It’s easier to get these people through the funnel than others.

Asking people to commit their attention over a long period of time is actually a form of effort. And you can measure that effort, but it requires looking at all three of these layers – platforms, products and business goals. If your business goal is to get subscriptions, then you need to be able to measure the impact your product is having on that goal, and the impact your chosen platform is having on your product. So you need to have a model of the user journey and user experience in your head when you’re designing your audience measurement strategy. You’re not just looking for raw numbers, but to understand the range and diversity of audience behaviors, the diversity of reasons why some people complete the journey in your model, and why some people don’t.

The role of audience measurement technology is not just to provide data for an audience model or funnel, but to show when that model doesn’t fit with what people are actually doing.

A lot of what we do at EA1 involves working with fandom communities around popular culture, and these have their own kind of dynamics, behaviors and communities. One of the challenges that brands have in engaging with fandoms is they assume that fans want to do the things that brands have in their audience models. We often have to explain to clients that their models of behavior are actually very different to what we’re seeing on the ground. So even if you start with a good model of audience behaviors that support your business goals, that model might not match up with real life. The role of audience measurement technology is not just to provide data for an audience model or funnel, but to show when that model doesn’t fit with what people are actually doing.

The data we get from social media platforms can sometimes serve as a proxy for understanding user behavior. Where our clients almost always get into trouble is when they start mistaking platform behaviors for actual user behaviors. Measuring likes or retweets just tells you how many times people have used a certain platform feature, and that leads to assumptions about what that feature means. A like doesn’t necessarily mean that someone is interested in the content for themselves – they might be bookmarking it or sharing it with others. So you can’t always have a model that uses likes in one part of the funnel, and deeper engagement like subscriptions further down the funnel, because you might be measuring two completely different, and separate, things.

The biggest mistake people make when setting up audience measurement tools is just relying on the dashboards that come out of the product box. You have to build custom dashboards that reflect the audience models you have. But even then, there are two things you need to do. First, go back and periodically question the assumptions about what the data means. Then you have to look beyond the dashboard to pay attention to what’s happening in culture. This is harder to measure because so much of audience behavior is actually closer to culture than it is to commerce – it’s about communities and behaviors, not data and transactions.

It’s a good idea to have someone in your team with more of an ethnographic background – use the dashboard as a map to understand where an audience might be, but then parachute in a human to see and understand what their interaction actually means. For example, we were working on a project for a TV client around the time that people started using animated gifs as a way to communicate with one another. They weren’t using these for the content, but as ‘reaction gifs’ –  as a way to express an emotion.

There was a sudden explosion on Facebook and Reddit of a reaction gif of a character from the client’s TV show. The network we were working with took this as a sign to change their audience strategy – they thought these people must love the show if they’re using this gif. So they ended up routing a big lump of their media budget into making more content for these platforms.

They were expecting to see an increase in views and engagement with the show, but it just didn’t happen. It turned out the audience were using the gifs because the actor had a really expressive face – it had nothing to do with the show at all. What the network should have done was not just promote the show, but build new content around that character. They needed to look at that user behavior and understand that the ties to the show were actually fairly weak, but if they built a bunch of other small ties around it, they could get people to the point where they might consider watching and downloading the show.

They see audience activity on a platform, assume this directly relates to their business goals, and then go and put additional ads or content onto that platform without actually understanding the context of the audience activity in the first place.

This is where a lot of people get into trouble with dashboards. They see audience activity on a platform, assume this directly relates to their business goals, and then go and put additional ads or content onto that platform without actually understanding the context of the audience activity in the first place. The great thing about these analytic platforms is that they do a really great job of looking for what is quantifiable in an audience, and then showing you those numbers. That might give you a sense of overall scale, or trends in sentiment or behaviors, but it doesn’t give you the cultural context of these behaviors. To understand this, you need something more than just numbers – something more than ‘Big Data’ – what the digital ethnographer Tricia Wang calls ‘Thick Data’: deeper ethnographic research into the context of your audience behaviors that go beyond the raw numbers and help you understand why people are doing things, not just what they are doing. 

I’m fascinated by organizations that have put a lot of work and resources into big data, because it seems to them like it’s the future, it lets them look at things at scale. People like the idea of scale because it’s easily commodifiable and gives you a clear number. But it’s easy to fall into the trap of thinking, ‘well, if I can just get even more data, if I can turn my data lake into a data ocean, that’s going to help me make a better decision.’ Some of the people we work with are finding this isn’t the case however – having too much data slows down decisions, especially if you’re a small organization.

For us at EA1, it’s always important to pair up big data insights with smaller stories that show what the data actually means in context. 

For us at EA1, it’s always important to pair up big data insights with smaller stories that show what the data actually means in context. We have a rule that we never present numbers on their own. We always accompany data with an insight and, most importantly, a recommendation for what to do next. The numbers aren’t an end in themselves; they’re part of a story that helps us create a shared model of audience behavior, and how this relates to the project goals.

The last really important point is to understand the difference between competitive and comparative audience data. If we are trying to understand audience attention, we have to find ways to measure it across multiple areas of culture and multiple tech platforms. For a show like Dr Who, the competition isn’t just other TV shows –  it’s social platforms, the latest Animal Crossing game, in fact it’s everything else that the target audience is paying attention to in wider culture. So we need to not just look at competitive data – how did we do against our industry peers? – but comparative data – how did we do against everything else the audience could be spending their attention on?

In short, a good audience measurement strategy is like an hourglass. You start with a wide-open problem of all the possible ways you could measure your audience across multiple platforms. Then you refine that into an audience model – a set of assumptions about the behaviors you want to track around your content. This will create a funnel that helps create a more manageable data flow, like sand flowing through the narrower middle of an hourglass. Once you start to get live data, you need to open out your model again, and start asking questions about the broader cultural context of the data.  What’s the context of the user experience? What’s the context of the social platforms that they’re using? What’s happening in the broader cultural milieu around your project?

In this hourglass model of data, your data and dashboards are in the middle, but the really important things are the stories you can tell before and after the dashboards – the stories that explain your audience model to stakeholders, but also contextualize the insights within a much broader understanding of the world our audiences are living in. If you just look at the data, it’s a bit like trying to drive a car just by looking at the speedometer. You really need to be looking out the window, not looking at the dashboard.

Kenyatta Cheese (@kenyatta) is from the internet and has been making and talking about digital culture since before he can remember. He is Co-Founder of Everybody at Once, a social media and audience engagement agency, whose work includes running the Doctor Who Tumblr for BBC America, which is the second biggest Tumblr in the world. He is also Co-Creator of Know Your Meme, a resource for understanding web culture. He has previously worked with the Eyebeam Center for Art and Technology, Manhattan Neighborhood Network, and the online video network Rocketboom.