Are Social Engagement Metrics a Proxy For Awareness?

Awareness is a means, not an end, so even if social engagement metrics are a indicator for it, the benefit is not necessarily established - or even apparent. JL

Katie Paine curates a discussion in The Measurement Advisor:

Beware of spurious correlations. I have difficulty understanding where the physical act of clicking or the writing a few thoughts ‘engages’ me fully cognitively or affectively in an issue, brand, or whatever.

Here’s the transcript of a fascinating discussion by IPR Measurement Commission members on whether social engagement is an indicator of awareness. Spoiler alert: Social engagement is probably not equal to awareness.

The participants (in order of appearance):

• Katie Paine, CEO, Paine Publishing
• Allyson Hugley, Measurement & Analytics President, Weber Shandwick
• David Geddes, Principal, Geddes Analytics
• Don Stacks, Professor, University of Miami
• Mark Weiner, CEO, Prime Research
• Sean Williams, Owner, Communication AMMO
• Tim Marklein, CEO, Big Valley Marketing
• Pauline Draper-Watts, Executive Vice President, Edelman Berland
• Lou Williams, Retired, The Lou Williams Companies
• David Dozier, Professor, San Diego State University
• Fraser Likely, President & Managing Partner, Likely Communication Strategies
• Brad Rawlins, Dean, College of Media & Communications, ASU
• Forrest Anderson, Planning & Evaluation Consultant, Forrest W. Anderson Consulting
• Elizabeth Rector, Senior Manager of Strategic Marketing, Cisco

Katie Paine: I’ve frequently given the advice: “Never ask a question for which you may not want to hear the answer.” In the case of the Institute for Public Relations Measurement Commission (IPRMC), if you decide to ask them a question, you better be prepared to hear more answers than you thought possible… and all of them thoughtful and provocative. For example, here’s what happened when we got onto the subject of ROI.
Last week, before I started a fight with a fellow speaker who had issued what I considered blasphemy to her attentive audience, I posed the following question to my fellow IPRMC members: I’ve heard a number of speakers at various conferences tell their audiences that engagement with a brand on social and digital media is evidence of awareness. We’ve always maintained that awareness can only be measured by a survey. It does seem that you must be aware of the brand when you click or share its content. Is it time to change our stance on this?
The debate:
Allyson Hugley: I don’t think so. With surveys, you benchmark awareness and track the change over time, pre/post campaign, etc. With measures of social engagement, we don’t know if people just became aware of the product/organization/issue or if they were previously aware and communications activity prompted them to start engaging via social channels at a specific point in time (different from when they became aware). Agree that audiences must be aware to engage, but they can be aware and not engage. I don’t feel this distinction can be teased out without directly asking about awareness via a survey.
Also, in many cases engagement on social is a reflection of the expanding options for where people can interact with organizations and brands rather than increased awareness (activity migration). Example, U.S. adults have always (one would hope) been aware that they have local representatives but, now, rather than calling or writing a letter they engage with their representatives via Twitter. The method of interacting/engaging has changed and engagement on social channels is increasing, not awareness. I’m open to debate on this, but that’s my view: awareness is measured via surveys. Same reason social chatter is not a proxy for awareness. Just because people decided to comment on something doesn’t mean they weren’t aware of it prior to being prompted to publicly express an opinion on Facebook/Twitter/Instagram.
David Geddes: Consider my example of a blog headline “Monsanto, Greenpeace, and GMOs: The Truth Behind the Debate.” Does a click arise because of either organization — (i) Monsanto, (ii) Greenpeace — or because of the topic, (iii) GMOs? Furthermore, click is a very minimal measure of engagement. A survey is required to measure and refine our understanding of the individual’s awareness – knowledge – understanding, etc.
Don Stacks: And let’s not forget that one click cannot be tested for reliability or, as David notes, validity of measurement.
David Geddes: This exemplifies the poverty of measurement when developed outside of an understanding rounded in communications/public relations theory. Start with a conceptual framework of how communications actually works, and derive measurements at each stage along the communications process. I see a tendency to gather low level data, such as clicks, rather than meaningful information, or to measure what is easy to measure rather than what should be measured.
Mark Weiner: The question was about “engagement” not “clicks.” Engagement indicates there’s a dialogue. There’s unstructured text which can be structured and qualified. As social media research evolves, it will continue to emerge as a surrogate, an alternative, and even a replacement to surveys because social data is linear rather than a moment in time, the unstructured text can be structured like qualitative research but the volume of responses can be in the tens of thousands and thus quantitative. What’s more, it satisfies the needs of most PR people because it’s fast, accessible, and inexpensive. And despite its drawbacks—limited samples, unrepresented parts of the population who operate outside of social networks, etc. etc.—it’s often good enough for many or even most situations where speed and cost are important.
As a group and among “people like us,” we evangelize that surveys/outcomes are some higher form of research when compared to media analysis/outputs research. In my opinion—and I recognize the natural assumption that I have a vested interest in the primacy of media analysis—outputs research, especially as reflected in social media, is already a powerful instrument to represent outcomes and a viable alternative to surveys (when done right). What’s more, social tells you not just what’s on people’s minds; it tells you how you can respond now. It’s not just me: I’m working with a paper now that incorporates the academic citations to support this notion so this is not a commercial. We never sound older and more out of touch with the common practitioner than we do when we make judgments like what I’m reading in this string. I love you all. Now let me have it.
Sean Williams: LOL Mark = I first come to praise Caesar… Some time ago I wrote an IPR blog post that averred that observations of social data such as what we describe here (and unscientific “polls” on intranets) can still be useful just for the notional insights. That is, we can quickly try things out and see what the response is, at very low cost of either time or treasure. This doesn’t replace the more formal research, but it can help organizations be flexible and nimble. Surely that’s a valuable thing. As for the specific question about whether social comments or clicks represent actual straw people for awareness—we’d stipulate that because it’s not representative, we should not draw too many conclusions about projecting the social results. What we can do is say that it can help establish a proxy for awareness or engagement within the specific group being examined.
I do this to some degree in my work with independent schools—we know that a fair number of parents are on Facebook, so we tailor campaigns on Facebook to that audience, and use social data (clicks, shares, comments) in a line to web traffic (to specific landing pages) that carries specific calls to action. Thus, we’re looking at the line of sight from admission inquiries back to the source. That doesn’t tell us how aware our communities are of the school—but it tells us how engaged our parents are in promoting the school to others (including their FB friends of friends…) This starts organically, then goes paid, and we have great metrics around it. I like this example because it doesn’t require scale—we don’t care if our cost per prospect winds up at $5+ —we need only (for example) 10 families of middle schoolers to decide to come to the school. Yours in good harmony…
Tim Marklein: What a fascinating debate to open the day! Exactly why the Measurement Commission is awesome. A few thoughts:
On an individual level, awareness is a precondition to engagement—and thus Katie’s characterization (via other speakers) that engagement is “evidence of awareness” is a reasonable statement. However, I’m inclined to think of awareness and engagement in a non-linear path for a few reasons:

1. We’re typically evaluating different objects, since awareness is “of” something and engagement is “with” someone. An individual might engage with Katie’s Lobster Shack to download a coupon, and thus we know they’re aware of Katie’s Lobster Shack, but we don’t know if they’re aware of Katie’s food or hospitality or pricing, etc.
2. We typically know very little about an engagement online beyond the action itself, whether that’s a like, share, click, comment and/or download. We can count those, but the individual engagement action doesn’t tell us more—unless there’s text/image/video commentary to go along with it, at which point the power of media analysis kicks in.
3. Testing awareness requires much more information, since as Allyson points out, we’re typically investigating changes in awareness not just evidence of awareness.
4. In the same way that awareness can be seen as a precondition to engagement, more active engagement can be used as a precursor to increase an individual’s or group’s awareness, knowledge, understanding, etc. The goals/metrics thus interact with each other, but not always one way.

Beyond viewing them as non-linear, I would argue that the increased ability to drive and measure engagement decreases the value of simple awareness as a goal/metric. Organizations have traditionally wanted to increase awareness as a precondition to increasing some other outcome (sales, donations, etc.), so marketing and communication leaders focused a lot on measuring awareness as the first step in a chain of impacts. However, we can now compress and/or circumvent that cycle via digital and social and other means—moving engagement and advocacy earlier in the process, making it more visible, and making it easier to measure. Engagement and advocacy don’t necessarily replace reach and awareness as metrics/goals, but I would argue they are potentially more important.
Pauline Draper-Watts: Coming late to this but it does remind me of the early days of social listening when some providers were talking about social listening being a replacement for focus groups. They provided a different purpose from a group of individuals who were engaged with social media and the conversation was free-flowing rather than directed to gain answers to specific questions. In a similar way, I think it would be very easy for people to see the plausibility of engagement equating to awareness or considering it as a proxy without contextualizing or considering relevant checks and balances. Much of the time we are considering aided and unaided awareness for a target group of individuals on something specific. If we are looking at engagement metrics it can lead to very misleading data and conclusions. Think back to the subservient chicken when much attribution was given to KFC rather than Burger King. Engagement metrics may look wonderful but they can be off target. I am not being negative to engagement metrics as they are very valuable for measuring just that—the engagement with the content—but they should not be confused with awareness. Loved reading this thread of conversation
Sean Williams: Good point, Pauline. Again, I don’t think it’s an actual proxy for general awareness—although at scale it can be easy to draw that conclusion. Even the engagement by itself isn’t very useful unless it links to some sort of change in behavior or attitudes, or confirmation of them. It’s a moment in time, among a certain group, and thus shouldn’t be generalized, n’est pas?
Lou Williams: Ok, I can’t resist getting my two cents in: I’ve always considered awareness merely a first step on a continuum. It’s sort of a (vicious or not so vicious) circle that includes attention, listening, assimilation, arousal, sympathy, empathy, fanaticism, lost perspective and, ultimately, ignorant bliss. From which one must start over. The reason I’ve never used the word “engagement” is that it is a fuzzy concept. With many meanings. The truth is we first need to get someone’s attention… awareness. Good PR programs are planned around the location of an audience on the circle. Knowing the location gets us a better understanding of what we need to do to sell, get the votes, get the confidence, etc. And, each and every step is measurable as a benchmark and for movement. Awareness is only great if it leads you someplace: cooperation, support, or aggressive (positive) action.
David Dozier: I’m with Lou with regard to engagement. The concept needs careful explication, or so it seems to me. I’m working on a project right now for my school. We’re interested in measuring organizational engagement of alumni with and between our school, as distinct from organizational identification. I believe the concept implies behavior, rather than some cognitive “sense” of engagement. I think we dilute the concept when we equate clicks with engagement. Visiting a website may be a relevant indicator if clustered with other behavioral indicators of engagement.
Mark Weiner: Which naturally leads to ROI, wouldn’t you agree? 
Fraser Likely: Katie asked two questions really:
1. How are the terms engagement and awareness defined; do they mean similar or different things (such as whether in the Dictionary or the Conclave definitions or in the definitions of other groups in the SM sphere)? Some MC folks in this string have implied that engagement precedes awareness, while others suggest that awareness precedes engagement. Is it one, or the other? We seem to be of mixed minds about these terms. The use of the term ‘engagement’ in social media measurement has always left me scratching my head. In fact, at one of Katie’s SM Conclaves recently, I argued against using the term. When I think of engagement, first I naturally refer to its use in management, HR and employee comms literature. Different than its SM use, certainly: confusing to have it used differently in two realms of PR, certainly. Second, I have difficulty understanding where the physical act of clicking or the writing a few thoughts ‘engages’ me fully cognitively or affectively in an issue, brand, or whatever. I would propose that another term be used to measure what Mark calls the “click-stream.” I think the term engagement, as typically used in SM, is misleading at best. 

2. How do you measure engagement and awareness?

That said, I’m certainly in the survey camp for awareness, if we define awareness as we have traditionally. Awareness I understand. Then Mark added tracking the language used in SM. That makes sense to me, and adds a whole different methodology between counting and surveying. It’s a precursor to actually surveying awareness in my mind, but it is not actually measuring the full extent of awareness. This methodology may indeed compliment an awareness survey – and may lead to a change in the type of questions in an awareness questionnaire.
Brad Rawlins: This has been a fun discussion to follow. Here’s my question: Why use engagement as a proxy for awareness? I think we all agree that they are two separate concepts, whether you agree on a particular definition of engagement or not. If engagement is a richer measure than awareness, then label the measure as engagement and don’t use it as a proxy for awareness. We know how to measure awareness, as a starting point in Lou’s continuum. There is obviously some disagreement about what engagement is and how we measure it. So, I think the solution is simple and obvious: Keep them as separate measures.
Mark Weiner: Way back when in this string, Tim made a great point about the diminished value of awareness now that you can go more directly to the outcomes which are usually more valuable in uncovering the behavioral sequence. Nowadays, awareness may just be the gate attached to which there is no fence. Why unlock the gate when one can simply walk around it? Fine and provocative observation, Tim. I was unable to express it late yesterday but you already succeeded earlier.
Sean Williams: This whole conversation points out the warrens and burrows of what it is we all do. There seems to be a market demand to “measure” things that we, as experts here, don’t see or believe the “true” value of. This is to some degree reinforced by the “social media guru” apparatchiks, who command a share of discussion in this space. WOMMA has a great investment in the concept that its members are moving the needle (somehow) in marketing. The Content Marketing Association has mastered its self-marketing (3,600 conference attendees) repackaging public relations activity. PR is now commonly seen as merely a piece of the marketing mix (a subordinate position often seen as passe, as the apparat declares news media irrelevant despite consistent data proving otherwise, and that consequentially limits the definition of PR to media relations.)
These points are made relevant for me as regards this discussion because feeding the top of the funnel (baseline awareness) is still an important metric in these discussions. Engagement in the social media sense cannot depend on deeper research, as when that’s done, there seems to be a tenuous link between social engagement and revenue. Witness Coke’s retreat from Facebook, or the continuing challenges of GM to keep and win market share despite “excellent” social activity. In internal comms, there is a strain of practitioners who reject the idea that employee communications outcomes include employee engagement; the sense seems to be that engagement for its own sake is not very useful. Instead there’s the question of “so what?” with engagement: To what end do we desire it? Is it possible that the research opportunity here is to establish conclusive relationships (not anecdotally) between social engagement and business outcomes OUTSIDE Of sales? It’s the difference between doing social marketing and social PR, no?
Mark Weiner: Yes and true. But a lot of external PR still strives for sales and as such is a contributing stream within the marketing mix.
David Geddes: Let me return to my initial comment that measurement must be built upon theory. Flashback to phlogiston chemistry, the dominant concept of combustion and chemistry in the 18th century. Phlogiston chemistry was a dominant chemical concept of the time because it seemed to explain so much in a simple fashion. Phlogiston accounted for observations in very narrowly defined contexts. Proto-Scientists made their careers based on the disparate observations and measurements to understand the nature of phlogiston. The metrics were correct. Then came Antoine Lavoisier who, in the late 18th century, developed a comprehensive foundation for modern chemistry. Phlogiston theory went up in flames.
Fortunately for public relations measurement, we have a theoretical foundation, called cognitive psychology and communications theory, with abundant scholarly research and support. We have a lot of phlogiston measurers latching on to things that they can measure, and trying to build something upon these data points (data… not information). Let’s try to understand what engagement is (and today the term is used for a variety of purposes) and what clicks are (since our discussion has linked the two) within the theory of communications. Then we can develop meaningful measurement constructs. For now, most social media measurement is at the state of phlogiston chemistry, or Lamarckian evolution if you prefer a biological analog: A lot of things that can be measured easily, lacking a conceptual foundation.
Mark Weiner: Thanks David. I didn’t know about Phlogiston chemistry until now. For me, this conversation has diverged into two paths: the “engagement path” and the “the role and purpose of ‘awareness’ research path.” I agree on the prevailing opinions here that engagement means a lot of things to a lot of people. It’s another overused measurement term that’s kept purposefully vague and perpetuated in the absence of anything better and in an attempt to obfuscate (like “impressions”). And I appreciate and agree with your point about people looking for easy ways to explain things. Sometimes that’s me and it’s always a temptation. I’m also thinking about how market mix modeling (one of my favorite subjects) seems to go from outputs to business results without necessarily stopping at outtakes and outcomes. The models work backwards from “business result” to “outputs” to draw their conclusions. So putting engagement aside, what role does outtakes/outcomes measurement hold for those who a) focus on the business result and b) have the data and resources to undertake such models? What would Lavoisier say?
Forrest Anderson: It’s hard to speak for Lavoisier. But he might want to understand why the business result occurred and wish to examine outtakes and outcomes as a part of knowing how best to manipulate that business result to improve (or ameliorate) it. This goes back to that long discussion we had on what was the best thing to evaluate. My take on that discussion was that in the larger picture the more parts of the model, or the communications campaign and its environment, you can examine and evaluate, the better you will understand what effect it has and how it has it. What this leaves out, of course is all the things outside communications such as product (or voter or shareholder) appeal, distribution, pricing/tradeoffs, initial reputation, etc. Great discussion, by the way!
David Geddes: Forrest, this is a productive comment, but I suggest that Lavoisier’s first interest would be to understand how the communications process works, in its broadest sense, and covering everything leading to the formation of attitudes, beliefs and perceptions… and then leading to behaviors. This interesting debate has turned around the difference between what we can measure and what we should measure.
Elizabeth Rector: Thanks for a great discussion. From my perspective, we have to be open to some of these proxies for awareness. Cisco is conducting primary research (that includes awareness measures among many other things) every six months and have increased it to quarterly, but if anyone has read the articles about the new CEO, you will see that he is always quoted as he is going to make things go “faster, faster, faster”, so we are being asked to have more “real-time” data, even if it isn’t perfect. This is the new reality that I am living in and I’m guessing other companies are moving that way too.
David Geddes: Selection of valid, reliable proxies is straightforward. Use data to show a high correlation, and you have a potential proxy metric. Beware of spurious correlations. Has anyone seen a correlation between a metric of engagement and a measure of awareness? See my paper that won the Holden Ruler Award. We can develop a metric to forecast awareness or anything else based on media or social media data.∞