So What? AI-enabled Share of Voice: Halloween Special - Trust Insights Marketing Analytics Consulting

So What? Marketing Analytics and Insights Live

airs every Thursday at 1 pm EST.

You can watch on YouTube Live. Be sure to subscribe and follow so you never miss an episode!

In this episode of So What? The Trust Insights weekly livestream, you’ll learn about calculating share of voice and sentiment analysis using generative AI. You’ll discover the importance of requirements gathering and learn how to use generative AI for this process. You’ll also uncover the limitations of traditional share of voice tools and learn how this new approach overcomes those challenges. Finally, you’ll see a practical application of share of voice analysis to Halloween candy.

Watch the video here:

So What? AI-enabled Share of Voice: Halloween Special

Watch this video on YouTube

Can’t see anything? Watch it on YouTube here.

In this episode you’ll learn:

How to gather and compute share of voice within a topic
How to use generative AI to improve sentiment analysis
When share-of-voice is or is not appropriate

Transcript:

What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.

Katie Robbert – 00:27
Happy Halloween, everyone! Welcome to “So What?”, the Marketing Analytics and Insights Live Show. I’m Katie, joined by Chris and John. How’s it going, fellas? Alright, so what do we have? We have Chris as a spooky cat, which totally makes sense. That’s on brand for you. Oh man, John, there are so many guesses. The bullet holes—Marvel.

John Wall – 00:52
So the big punchline to this is my son is going to be Deadpool tonight for Halloween. And so he’s like, “You can be Wolverine”. And I’m like…

Katie Robbert – 01:01
You know what you’re missing? The mutton chops. That’s what threw me off.

John Wall – 01:04
I know. If I had the hair and then, of course, the closet. I’m like, “No, we have $20 to work with. So it’s going to be sweatshirt and we’re done.”

Katie Robbert – 01:13
And I—of course you can’t see it because I definitely need a haircut—but I have my Jason teeth from Friday the 13th t-shirt on and my Oogie Boogie socks from The Nightmare Before Christmas. So we are all representing Halloween. It is a favorite holiday around the Trust Insights offices. Chris, thank you for our spooky opening. That was fantastic.

Christopher Penn – 01:34
Very tight.

John Wall – 01:35
It totally hit with the zoom in to you.

Christopher Penn – 01:41
Thank you, thank you.

Katie Robbert – 01:42
So, keeping in the spirit of Halloween today. These guys are so over me keeping the spirit of Halloween today. We’re talking about AI-enabled share of voice. It’s our Halloween special. So what we wanted to do today is talk through how you can use generative AI to measure share of voice. But we’re doing it with a Halloween twist. So Chris, where would you like to start today?

Christopher Penn – 02:08
Oh gosh. We’re going to do the Halloween twist. And I want to provide the caveat and caution that to do this… well, most of the work is not AI. Most of this problem—solving this particular problem—is not an AI problem. It is a data gathering problem. It is a data analysis problem. There is AI involved, but ultimately, it’s mostly a technical problem to solve. Now, the flip side of that is that you can use generative AI to help you build most of the infrastructure and pieces you need to solve the problem. But that’s the hard part. So let’s start with the 5 Ps. Katie, when you and I were talking about this last week, what went through your mind in terms of why in the world we would even want to do something like this?

Katie Robbert – 03:09
Well, and I feel like we need to take one step back, even from the 5 Ps. And first, define what share of voice is because it’s not a metric that everybody uses or is familiar with, or they are familiar with it, but they’re not using share of voice as the descriptor. So I learned what share of voice was when I first started working with you, Chris, at the PR agency. I’d never heard the term before. So my understanding—and correct me if I’m wrong—is that share of voice is basically your digital footprint. How much people are talking about you. And then you can go deeper to find out: are they talking about you in a positive way, or are they talking about you in a negative way?

Katie Robbert – 03:57
And so a lot of social listening tools, for example, are capturing data similar to what we would consider to be share of voice: how many mentions did you get? How many conversations are you appearing in? There’s a lot of tools on the market that do social listening. But then you’re missing other data sets that aren’t necessarily social, which is what we wanted to do today. So share of voice, in a nutshell, is how big your digital footprint is. How many people are talking about you and what are they saying about you—good, bad, and different.

Christopher Penn – 04:36
Correct. Now, where almost everyone goes wrong with share of voice is they assume it’s sort of a global analysis. What percentage of the conversation is about us? Share of voice has to be done within a constrained context, a defined context. Because, for example, right now in the USA, we are a week away—less than a week away—from a presidential election. Globally, your share of voice is zero. No one is talking about anything else except that and maybe some AI. But effectively, if you were to measure the total amount of news and conversation, any brand’s share of voice is effectively zero at the moment because everyone’s talking about that other thing. We’re not going to talk about the thing.

Christopher Penn – 05:17
But if you were to constrain it to a specific set of sources or conversations, then you can say, “Okay, within this set, what is the share of voice?” So for example, we used to have this one client, a tech company, and they said there are five trade publications that our buyers read. “We only care about our share of voice in these five publications. We don’t care what anyone’s saying on Twitter. Twitter doesn’t matter. These five paper magazines are what our senior buyers read.” And so our share of voice is measured within this. And this was, in fact—this was almost eight years ago now. It was the same trip that you and I, Katie, went out to visit this client in Santa Clara and actually had the idea of founding Trust Insights. That’s how to think about share of voice: within a specific data set.

Christopher Penn – 06:11
What is your share of voice? So for the first part of this is you have to figure out: A, why do you care? And then B, what is that data set? So if we think about the 5 Ps, it’s measuring the share of voice within the media that your buyers care about.

Katie Robbert – 06:29
Right. And so if you go through—now that we have defined share of voice—we can walk through quickly the 5 Ps so that we can get to the analysis that everybody is looking forward to. So in this case, share of voice is really just a very broad term. If you narrow it down using the 5 Ps, it could be brand awareness, it could be social conversation, it could be a variety of things. And so for us, for Trust Insights, I would lean towards share of voice being… my purpose being: I want to understand our brand awareness. Not only how much we’re coming up in conversations that we’re starting, but how much people are talking about us when we’re not even there.

Katie Robbert – 07:21
And that, to me, would be a measure of brand awareness. So we would use technology to build that. So the purpose would be, “I want to understand brand awareness.” The people would be our external audience, but also just sort of the digital landscape as a whole. Who’s out there? Those people—what are they talking about? And then us internally would be the people. The process would be what Chris is going to describe today on the live stream platforms. Again, what Chris is going to walk through. And then performance is: I want to understand how strong our brand awareness is outside of us starting the conversation. Are people basically talking about us? And that’s then leading to some kind of engagement transaction.

Katie Robbert – 08:09
So I mean, I could look at referral traffic. I could look at other conversations and mentions. There’s a lot of different ways I would do that. But first, I’ve defined my purpose. So knowing that Trust Insights today is not the topic, it’s a Halloween special. Before we get into what you’ve cooked up for us in your cauldrons today, John, in your… you’ve had a slightly different career path than Chris and I have. Have you come across share of voice as a metric that’s mattered in your previous roles?

John Wall – 08:43
Yeah, we used to do a lot of that with event management stuff. Because you have attendees and exhibitors, and you have a relatively tiny number of purchases that happen. But you really need to know: what’s the… how much chatter is out there? And what topics are the hottest? And where should… should you be going? But the stuff that we were doing was of no comparison to the tools that you guys have been using as far as machine learning. We could do surveying—some email surveys—but most of it was just interviewing people out in the field, which is just like the caveman tools compared to the ability to grab huge piles of data and sift through it to get insight that… you’d get at 1/1000th of the cost.

Katie Robbert – 09:27
Yeah, that makes sense. Alright, Chris, where do we start today?

Christopher Penn – 09:33
We start uncomfortably—at least for me—with requirements gathering. Because to do this sort of thing, you need to sit down and say, “Well, what is it we’re trying to do?” And so there’s two forms of requirements gathering. One is domain requirements, which is: what’s the big picture? And then two is: what are the functional requirements? What are the specific things that the technology has to do? And this is an important process that you have to go through. The good news is you can use generative AI to do it. So when I was… when I got started with this, we talked… Katie and I talked through the idea, and then I said, “I wrote it out in a couple of paragraphs of text. I want to do this.”

Christopher Penn – 10:12
“I want to get a bunch of data from the news sources, maybe the last 30 days of news sources. I want to get some conversations from Reddit. I want to get some videos and comments from YouTube. I want to get… try and get as much data as I can from reasonably credible sources. And I want to see Halloween candy. Which Halloween candy is mentioned the most? And how is it talked about? What sort of sentiment? And maybe even what are some of the emotions that people express around this?” So I gave those couple of paragraphs and had a conversation with Google’s Gemini model about, “Well, what would it look like? What would the requirements be to build out a piece of software to do this?” And what ended up happening was we ended up building four different pieces of software.

Christopher Penn – 11:02
So one is an article extractor to get news articles and just to download them and download the text from them, removing stuff like navigation and ads and all that stuff. Two was a tool to get YouTube videos and their comments. Three is a tool to get Reddit, a subreddit, posts and comments. And then four is to process all this data through generative AI. So part of the remit was to say, “Here is an article. What candy has mentioned it? What is the sentiment around this candy? Is it from a minus ten—’I hate it’—to, ‘I would eat this if it was… you know, I had to pick my last meal before execution. It would be a bowl of this.'” And then there’s a concept in behavioral psychology called Plutchik’s wheel of emotions, which essentially has six core emotions.

Christopher Penn – 11:59
And then, like a color wheel, combinations of those create more complex emotions. And we said, “Okay, we don’t want to just do sentiment. We want to decompose sentiment into the emotions that people express around their language.” So the architecture of the system was: grab all the data, clean it up, get it into a database. And then article by article, post by post, hand to a generative AI model. We used Google’s Gemini 2 model and come up with the analysis. There’s a lot that goes into baking a system like this. If you were to look… and I’ll just bring up—just so you can see—just the processing came out to about 12 or 14 different pieces of code. Because when you use today’s modern generative AI tools, they implement best practices if you tell them to.

Christopher Penn – 12:56
I told it: “I want to implement best practices. I want you to modularize this. I want you to make it flexible. I want it to be object-oriented. I don’t want repetitive code”—all the things basically I do wrong. It came out and said, “Okay, here’s your system diagram.” And then let’s write each piece of code. And so for the better part of a week in the evenings, I’ve been having two different models work on this. So Claude from Anthropic was writing the code and then, after every iteration, I would open up Google Gemini and have Gemini QA the code. So, Katie, one of the things you say often is: a developer should never QA their own code.

Katie Robbert – 13:39
It’s true.

Christopher Penn – 13:41
And AI should never QA its own code either. You should have a different model doing the QA.

Katie Robbert – 13:48
So I have a couple of things. Number one, I’m now going to have to start saying, “I’m feeling my whole color wheel of emotions.” I feel like that’s just something I immediately need to adopt. But more importantly, my question, Chris, is: how is what you’re creating different from what somebody could maybe get in an off-the-shelf product? For example, there are tools that specifically do what we’re calling share of voice. How is what you’re building different from what people can get in an off-the-shelf piece of software?

Christopher Penn – 14:24
Share of voice numbers only? It’s no different, because you’re just counting mentions. You just see how many times does this word appear in the text, and you count it. That technology is what—60 years old? There’s nothing new there. What’s different is when you get to sentiment and emotion. Typically, most legacy analysis packages, particularly ones that analyze social data, they use an old Stanford approach from Stanford University—which was great at the time, 20 years ago—called “bag of words.” And they say, “When these words appear, it’s negative sentiment. When these words appear, it’s positive sentiment.” “I hate this.” The word “hate” is negative. The word “love” is positive. That bag of words approach is… it’s better than nothing, but it’s wrong a good amount of the time because it can’t… it does not understand the context at all.

Christopher Penn – 15:17
What language models are really good at is language, and in particular, understanding the context of a statement. One of the things that is in the code is: it takes a mention, it takes the two sentences before, the two sentences after it, plus the sentence it’s in, and says, “What is the context of this brand mention?” So saying, “I made myself sick again because I can’t stop eating Reese’s Pieces.” And that sort of sentence, in context, goes, “They really like this.” They’re making themselves ill. If you were using a bag of words approach, that would be negative sentiment, saying, “Oh, you’re sick, you’re ill.” But the meaning behind that in human language is: “Oh, you really like that?”

Katie Robbert – 16:07
Well, and what’s interesting… Well, first of all, you can tell that it was named by developers because it’s called “bag of words”. But the second is, so… you’re describing it at a very simplistic level. Love is positive. Hate is negative. And it can’t understand concept because people will say things like, “I love to hate it.” And so, like, what? Like, how do you score that? And so that’s… that’s what’s interesting to me is the lack of context. And I know we’ve talked before about sentiment. It’s really hard for these tools to pick up on sarcasm. For example, “Oh, I love it so much,” means “I really hate it.” But it can’t hear me saying it. It’s just reading it.

Katie Robbert – 16:54
So it’s going to rate it positive when I’m really saying something negative. And so I’m really fascinated to hear more about the tool, or the analysis, that you’re working on versus what we can get off the shelf today.

Christopher Penn – 17:10
Right. And that’s the big difference: is by taking a moving window of language through the text, you’re going to understand the sort of sentiment and the tone around the particular mention that you’re talking about. And what’s challenging is, with a lot of articles, and even Reddit posts, because people sure do love to foam at the mouth, you will have multiple mentions in a piece of text that have different sentiment. So you have to catalog each of them individually. Your starting paragraph might be talking about, “When I was a kid, I used to hate Reese’s Pieces, and now as an adult, I love them.” Those are two very different things and you have to account for both of them because they are both valid mentions. They have different sentiments.

Christopher Penn – 17:59
So part of the thing that you have to figure out is: you need to have a language model help to interpret that. Hence the moving context window. None—to my knowledge—none of today’s off-the-shelf software packages do this level of analysis. And here’s the reason why. To do just 6,000 articles, which is what we started with, it took close to 24 hours of processing time where my computer was basically half usable during this time. Now, could I have used Google Gemini? Of course. I could have sent it straight to the Gemini cloud. And Katie, you would have a bill for $3,000 today.

Katie Robbert – 18:37
Do you want to guess what my sentiment would be? I don’t think you need a tool for that one. I am not using my entire color wheel on that one exactly.

Christopher Penn – 18:50
Because it may seem like a simple thing, but when you have a document that is multiple mentions, you have to send each mention and the two sentences around it to AI for it to evaluate. So a document may have 25, 30, 50 API calls, and suddenly you are racking up the bills. And as a result, I ran this locally using Gemini 2, because I know that my MacBook can support that model. And it did, but at the expense of… it was churning away all night. And I had to… I had to tone it down for a conference call earlier because I was like, it would otherwise… the conference call would not have worked.

Christopher Penn – 19:30
So those are some of the architectural things that, from a generative AI perspective, you need to know before you go into building something like this. Because you need to say, “What either… what is my budget that I’m willing to spend on API calls? Or what processing power can I bring to the party to do this kind of thing?”

Katie Robbert – 19:48
John, you have a couple of teenagers at home. I feel like you could use this tool that Chris is building to try to decipher what the heck they’re saying to you half the time. Like, what is the sentiment of your 15-year-old? What is he trying to tell you? Is he being sarcastic or is he being honest?

John Wall – 20:05
Is it good that I am Skibidi Bop? Yeah, these are things that we need to figure out.

Katie Robbert – 20:11
Well, but that’s a really good… I’m actually glad you brought that up because I feel like, Chris, that might be something that these standalone, off-the-shelf softwares might struggle with, is keeping up with how quickly language and jargon and slang are changing. Whereas generative AI theoretically can do that faster now. Is it total nonsense? Absolutely. But to that generation it means something. And so… but we need tools to be able to decipher: is “Skibidi Bop” a positive thing or a negative thing? If you’re asking me, I have no flipping idea what it is. But hopefully my generative AI tool can tell me. Like, is it a positive thing? Is it a negative thing? I don’t know. Then you add in the word “toilet” and all bets go out the window.

John Wall – 21:05
I was dying, though. You totally had me stringing along. I thought your punchline was going to be, “Can we use this to make sure your kids don’t eat too much candy tonight?” which is going to be the number one disaster in my house as they chow down five bags of candy that they pull home. But yeah, we will not get to the bottom. I guess there’s no clue, no cure for that one.

Katie Robbert – 21:25
No, unfortunately not. There’s no AI that’s going to prevent your teenagers from ingesting too much sugar.

Christopher Penn – 21:32
Exactly. So we have your traditional ETL pipeline: Extract the data from where it lives. We use the AHREFS tool, Reddit and YouTube, extract the data from those sources, clean it, transform it, load it into a… in this case, I used an SQLite database because it’s local, it costs nothing, et cetera. Then row by row, a Python tool to feed the data to a language model in segmented pieces. Take that data and then put it back into the database. Here’s the problem. There’s one part that we didn’t do here, which is: the end result of this process is a database. And that is not a visualization, that is not a report, that is not analysis. That is just a big old pile of data.

Christopher Penn – 22:26
So the third and final component of the system is you have to take the data out of the database and do something with it. The tool of choice there is the R programming language, because R is natively… is one of the best languages for doing advanced data visualization. You have to go back to the database, get the data out of the database, and then start doing stuff with it. What you end up with is charts like this, where we see the total number of mentions by candy in 2024 Halloween candy content. And we start off with Snickers with 2,700 mentions, candy corn, 2,600—those are the top two, by far. Skittles is a distant third at 1,000, Sour Patch Kids at 826 and Twix at 702. So those are the top five mentioned candies.

Christopher Penn – 23:22
Now, one of the other things that was a bit challenging here was we had to, first, up front, figure out what candies are going to be mentioned. So to do that was a relatively straightforward NLP analysis where I took 15 or 20 different articles from the last three years about popular Halloween candies, put that into just straight-up Google Gemini, and said, “Give me an ordered list of names, just the names of the candy that we should be looking for.”

Katie Robbert – 23:53
Which makes sense because, theoretically, you’re doing this on behalf of a client or for your company itself. So you already know what it is you’re looking for. You’re looking for Trust Insights. You’re looking for, Chris Penn, John Wall, Katie Robbert. You already have those things. Whereas in this we’re doing it, outside of the context, if somebody asked us to do this. So to your point, we had to get that list first because you can’t just say to the tool, “Get me all the top candy.” That’s too vague. What does that mean? That’s not good prompt engineering. You need to first define your constraints, like: “I want to look for the following things.”

Christopher Penn – 24:36
Exactly. So this is the total number of mentions. From there… let’s actually… let’s go through. Let’s look at the share of voice. We can convert that by basically summing up everything and then dividing each line by 100 to get a percentage. So Snickers has 17.8% share of voice. Candy Corn, 17.5% share of voice. Skittles has 6.7% share of voice. Sour Patch Kids has 5.5%, and Twix has… sorry, Twix has 4.7%. So if you were measuring share of voice, if you were the brand manager for, like, Lemon Heads, you have one percent share of voice. You are not a substantial part of the Halloween conversation. Although your candy is quite tasty… well…

Katie Robbert – 25:19
And I think that this is a good benchmark because you can imagine—and again, this is me sort of speculating—that especially this time of year, these brands are spending a lot of advertising dollars. And so this is a good way to check: “Is my advertising dollars doing anything?” Because you’re doing advertising at Halloween for your candy so that people will buy it and talk about it. So obviously you can look at revenue. That’s going to be a really good metric. But in addition, you also want to look at share of voice. It’s not enough for people to just buy. You also want them to be aware of it, talk about it, bring it up in conversation. “I had a really good Snickers. Have you tried it?”

Katie Robbert – 26:03
“If you haven’t tried Snickers, you should totally try it.” Whatever the conversation is. Candy Corn… it’s funny, it’s such a polarizing candy that, when we get to the actual sentiment, I’m interested to see… and I’m not surprised because every year you see news articles about people are completely on either side of the, “Is Candy Corn any good?” conversation. So that doesn’t surprise me. What I am surprised to not see higher up is Reese’s. So I see that’s, what, around like ten or 12. I personally would think that it would be higher, but that’s me being an N of one.

Christopher Penn – 26:45
Reese’s Peanut Butter Cups is up there, and then there’s Reese’s Pieces further down.

Katie Robbert – 26:50
Man, they’ve got to step up their game.

Christopher Penn – 26:53
Exactly. Part of what we also had to do—and this was something that was suggested by the generative AI while it was generating the code—was trying to account for all of the variations in the way people reference these things. So some people will talk about “Reese’s Peanut Butter Cup.” Some people call them, “Reese’s PBCs,” which… I guess that’s just the way people talk. Sometimes there’s an apostrophe, sometimes there’s not an apostrophe. Sometimes different things are capitalized. So part of the natural language processing, even just for entity recognition, is accounting for the ways people talk. And again, that’s when we’re building the tool—that’s something that generative AI can help you with.

Christopher Penn – 27:36
You would give it examples of 20 or 30 or 40 posts and articles and say, “Try to identify and calculate the different ways people will do this.” And then let’s encode that into the system so that it can detect those correctly. There were a couple of misses earlier. Katie, you had mentioned the Take 5, and Reese’s Take 5 are denoted separately here, and they shouldn’t have been. So the QA process has to be ever ongoing.

Katie Robbert – 28:05
Well, and this brings me right back to when we used to use share of voice tools at the agency and those really long Boolean string queries that we would build. It would be like, “This and this and this,” for one topic. So like, let’s say it was Trust Insights. It would be, “Trust Insights,” and “Trusted Insights,” and “Truth Insights.” And for all the different variations, prior to having tools like generative AI where you could do that research upfront to see what are the variations and what should I actually put into the search? We were just kind of guessing. We weren’t totally guessing because we had some data, but we weren’t catching all of the variations of misspellings and things that aren’t applicable. So, I don’t know, just pick, like, Oreo, for example.

Katie Robbert – 29:00
Let’s say that “Oreo” was borrowed from a different word that contained “Oreo” that meant something else. You would have to exclude that from your search query as well, because that messes up the results really fast.

Christopher Penn – 29:14
Yep, exactly. So part of the process behind the scenes, of course, is, as the articles are being downloaded and processed, they have to be all converted to lowercase. That way you account for some of the issues you’ll run into. Or sometimes people will capitalize “Eminem,” sometimes people won’t. Or capitalize one of the “M”s. By forcing everything to lowercase, you do a bit of text cleaning. Alright, let’s look at sentiment. So this is weighted sentiment. Weighted sentiment is a weighted average. And what you do there is you basically take the number of mentions, and you apply that as a scale to the… the actual sentiment. Because otherwise you could have one mention of, Lemon Heads, that was a positive ten, and it’ll be at the top of the chart. And it’s clear that’s…

Christopher Penn – 30:00
…that literally is an N of one. So by applying weighted mentions, weighted sentiment, we see what people like and don’t like. The top five for things people like: M&Ms, Heath Bars, Caramel Apple Pops, Hershey’s Miniatures, and Oreos. At the bottom of the list, though: Swedish Fish, Hot Tamales, Jolly Ranchers, Blow Pops, and Sugar Babies. Sugar Babies is the one that stands out the most. People really don’t like that.

Katie Robbert – 30:27
They are terrible. I agree with that. Sugar Babies are not good, but… again, that’s… it’s not representative of everybody’s opinion. It’s just… the majority of people online, this is what they’re saying about it. I would tend to disagree that Caramel Apple Pops should be so high, but that’s my opinion. And so if I really want to change that, I need to get online and start a campaign to get everybody behind how terrible Caramel Apple Pops are. But that’s how you change the sentiment. It can’t just be, “Well, this is my opinion. So obviously the data is wrong.” It’s like, “No, it’s me against all of the rest of the people having this conversation.” I don’t know, John, does any of this come as a surprise to you?

John Wall – 31:17
Well, yeah, I had to Google “Caramel Apple Pops”. I’m not familiar with that. And I am a junk food junkie, so for that not to be on my radar is pretty significant.

Katie Robbert – 31:28
Oh no, we added something to his list, Chris.

John Wall – 31:31
The Sugar Baby hate is a little weird to me, too. I mean, they’re just balls of sugar. It’s not like they… they’re not Hot Tamales. Hot Tamales are just straight-up terrible. We can pretty much agree on that. And it is funny. There’s… there’s a couple candies not mentioned. Necco Wafers doesn’t make the list anymore. Those are beyond horrible. Those are just like Alka-Seltzer with some sugar on them. But… not surprising to see M&Ms and Heath totally dominating the conversation, and Reese’s Pieces way up there. That’s interesting, too. Ahead of the Peanut Butter Cups, even. I mean, given, again, the N of one. But when I see the pile come back from the kids, Reese’s and Snickers definitely dominating. How about… what do you feel about the fact that it seems like it’s all…

John Wall – 32:13
…of course, I guess it’s candy. Because it’s like 85% is positive. The amount of hated candy is far smaller than… I don’t think I was expecting it to be that small.

Christopher Penn – 32:25
Yeah, the distribution… obviously people like candy. So for you to be that negative, you have to really be terrible. What’s interesting is Candy Corn has a sentiment of about 1.84. Now remember, this is minus ten, positive ten is the range. So Candy Corn is pretty much near the middle of the range. It’s a mildly positive sentiment because… this… what effectively it says is the number of people who hate it and the number of people who love it are probably about equal. And so it averages out and you end up with this distribution.

Katie Robbert – 32:58
I mean, that sounds about right. Based on… we did a very informal, non-scientific poll in our free Slack group, Analytics for Marketers, which you can certainly join for free. You can join the conversation. And I asked about, “What side of the conversation are you on with Candy Corn?” And we definitely had a bit of a split where people were like, “No, it tastes delicious like caramel,” and whatnot. And obviously, those people are psychopaths. Because then you also have the normal people who are like, “It tastes like a ball of wax.”

John Wall – 33:36
Put it right there with Sugar Babies. It’s just a wad of sugar. Like, there’s not much going on there.

Katie Robbert – 33:41
Yeah. And I think, what’s interesting here is we’re talking about a mix of brands and then types of candy. So you have a little bit of both. So Candy Corn itself is not a brand. Candy Corn is a type of candy that people don’t say, “Oh, I’m looking for homemade Candy Corn. Or I’m looking for Brach’s Candy Corn.” They don’t associate that. So Candy Corn, to me, is a little bit of an outlier because it could depend on the brand of candy corn in terms of the taste. Whereas everything else on this list, for the most part, is the brand.

Christopher Penn – 34:23
Exactly. So if we look at the emotional, weighted emotional intensity, what’s interesting is the wide array of emotions. Now this is zero to… this is zero to ten. And the intensity is how much the audience feels about these things. And it’s interesting that pride came up as the number one, followed by love, fear, hope, et cetera. This is a very odd distribution of the way people feel about their Halloween candy.

Katie Robbert – 34:57
And it would be interesting… and obviously we didn’t do it for this episode, but if you were doing this analysis for a brand, or for a competitive analysis for your brand against others, you would want to see specifically these emotions per brand. So I’m going to take just a wild—just totally wild guess, not scientific—that the pride might be from people who enjoy having the polarizing opinion about certain candies. So they take pride in maybe starting those online arguments of “Candy Corn is the best, you know, absolute… Yeah, I… I win. I got it right.” So we did intensity because this is the intensity just for Candy Corn.

Christopher Penn – 35:37
So I ran that exact analysis because I was curious if we dug dove just into Candy Corn: pride, curiosity, and love. Whereas pride, love, and fear for candy as a whole.

Katie Robbert – 35:49
I would like the audience to know that I did not know that this analysis existed. So it was pure speculation. And I’m pretty stoked and prideful—using my entire color wheel of emotions—that I got it right because… but it makes sense because pride, in the context of talking about something like candy, is like, to me, that’s like you’re bragging about something. And so you’re… maybe you have a lot of pride of how much candy you’re giving out, or how much candy you got from trick-or-treating. Or pride in starting arguments online with polarizing opinions.

Christopher Penn – 36:26
Exactly. And so that’s the dig into candy. Now here’s the thing: is this data is reusable over and over again because you can see there’s the candy name, the sentiment score, the emotion name, the intensity. You can see in the mentioned database the type of candy: Is it chocolate? Is it chewy? Is it gummy? Is it sugar candy? The number of mentions. And you can see a lot of the different URLs there. So the takeaway for a brand would be you would want to be doing this kind of analysis of the publications and the places that your audience speaks and using generative AI as part of the process. What I can’t underscore enough is that you cannot do this just with generative AI. You cannot open a ChatGPT browser and create this analysis because 85, 90% of this is not AI.

Christopher Penn – 37:27
85 to 90% of this is good, old-fashioned coding.

Katie Robbert – 37:34
So I guess this… naturally, the question our friend Brooke Selles was asking on LinkedIn the other day is: what are things that people keep saying are dead that aren’t really dead? And so I think you’ve convinced me, Chris, that share of voice is not dead. We just need to rethink how we approach gathering that data.

Christopher Penn – 37:57
Exactly. Because you want to be thoughtful about the boundaries of your data. If, for example, that Twitter is not where your audience is, but LinkedIn is, then Twitter data should not be in your measurements of share of voice. It should be LinkedIn. If Threads is where your community is, if Instagram, if Discord, wherever your community is, that should be part of it. You should be looking… if people are trying to figure out how to use your product, you might be looking at not just YouTube videos, but a subset of YouTube videos—say of the “how to” instructional videos—”What is our share of voice within that segment?”

Christopher Penn – 38:39
And so you have to go back and really think through your 5 Ps, the people in particular: where are the people? And what are the boundaries where the people are spending their time or getting their information to do this better? I was just doing this earlier today for a client. We’re looking at some data about a particular healthcare topic and saying, “Okay, well, where… what is the weighting of some sets of data versus other sets of data? Should one be given more authority than other in the analysis? Because the people are in different places and in each place the people are different.

Katie Robbert – 39:20
And that is definitely something that you need to take into consideration. I would say the opposite is also true if you’re using it as an opportunity to figure out what your competitors are doing. So if you’re doing a competitive analysis and you’re trying to do the share of voice, I would say you would want to look at channels that you are… that you don’t already have a good handle on. So let’s say you don’t think your audience is on Twitter, so you don’t look at that data. You would want to include that data as you’re looking at your competitors to say, “Well, are they there? Do we need to try harder to be there? Do we care about being there?” That all comes through those requirements gathering up front. Because, as Chris just outlined, it’s not…

Katie Robbert – 40:02
…it is a straightforward process once you’re organized, but it’s something that takes time to put together. So you really want to make sure you’re clear about what it is that you’re doing. John, have any of the results of… the great candy debate come as a surprise to you?

John Wall – 40:20
The pride thing was… yeah, completely out of left field. I was not expecting that. And the fear ones being at the bottom are right on the mark. So, but yeah, it’s just interesting to see. I think it highlights how everybody has their own bias. You kind of think about what stuff is popular and isn’t, but when you look at the data, you get a different picture and it definitely gets you… it’s a good reality check.

Christopher Penn – 40:43
One other caution I will advise people: if you want to do this yourself, the prompting is going to be wildly different based on the model you are using. You should not be writing these prompts yourself. You can come up with the prompt idea, but then each model has its own particular way of doing something. And if you just write a generic prompt like, “Oh, I want to know the sentiment of this whatever,” you’re going to get some peculiar results. So one of the tricks of the trade, if you will, is when we’re assembling this software, when we write these prompts, we start with our intent. And then we ask the model, “Rewrite this prompt for Gemini 2,” or Google Gemini or ChatGPT, or whatever the engine is you’re using under the hood.

Christopher Penn – 41:35
If you don’t do this step, you will get really bad results. This is a mistake I have seen a lot of people make—I made early on—of trying to write the prompt. Do not write the prompt yourself. Come up with the prompt, and then have the machine rewrite it to be specific to that model. And you cannot take it from one model to the next. If you have Claude write a prompt, it will not work in Gemini. In fact, Claude, the first time… when I was developing the software, rewrote my prompt for me. I put it into the Gemini 2 model and it just crapped the bed. The results were all over the place. And I… I basically said to Claude, “Do the rest of the coding around this.”

Christopher Penn – 42:16
“Do not touch my prompt because it has to be exact for this particular model.” This is an open model—this is Gemini 2 is an open model. It’s a small model, which means that the precision and prompting has to be very precise as opposed to using ChatGPT, which is a much bigger model, which can handle more ambiguity.

Katie Robbert – 42:37
It makes sense. I mean, and that’s all the stuff that… if you open up a model today and you’re not used to it, those are the mistakes that pretty much everybody is going to make. So I think that those are really good pro tips. And the other thing is: if you are interested in share of voice, or something like that, that’s certainly something we can help with. Go to TrustInsights.ai, contact, and you can talk to Wolverine over here. He’ll get you set up with all the right stuff.

John Wall – 43:06
That’s right. That’s what we X-Men do.

Christopher Penn – 43:13
But yeah, this is an interesting adventure because we ended up building four apps, which was not the intent. What sounded like a simple thing… And again, 90% of it was not AI. One of the questions, Katie, that you often like to ask me is: “What does a non-technical person do if they want to do this?” The answer is: you hire someone. There is no low-code/no-code way to accomplish this task. Sorry, that’s just the uncomfortable truth.

Katie Robbert – 43:44
Well, I would respectfully disagree with you, but the answer is not a scalable one. Because I’ve been on the side of doing this is you do it manually. You have human coders. I did a clinical trial that was, I believe, called sentiment analysis, where we took a lot of forum data and we had to hand-code, “Is this positive? Is this negative? Is this neutral?” And so, to respectfully disagree with you, yes, you can do this yourself, but it’s going to take you forever. By the time you’re done hand-coding, you’ll already be into the next quarter and your data will be completely out of date given how quickly conversations move and move on the internet. So I agree with you, Chris. You definitely want to hire someone if you don’t have the skill sets that you’ve talked about in this episode.

Christopher Penn – 44:38
Yeah, I mean, you could manually paste each article—all 6,500, one at a time—at ChatGPT. But it’ll be next Halloween.

Katie Robbert – 44:47
But to your point, then you’re talking about computing time, and bills and invoices and costs that you weren’t aware were going to happen. So definitely get yourself organized, use the 5 Ps, do your requirements and then say, “What of this can I do myself? And what do I need to bring in someone like Trust Insights to do for me?”

Christopher Penn – 45:08
Exactly. All right, any final thoughts before we go and cause egregious harm to our insulin systems?

Katie Robbert – 45:17
Happy Halloween, everyone.

Christopher Penn – 45:20
Happy Halloween. We’ll talk to you all on the next one. Thanks for watching today. Be sure to subscribe to our show wherever you’re watching it. For more resources and to learn more, check out the Trust Insights podcast at TrustInsights.ai/TIpodcast. A weekly email newsletter at TrustInsights.ai/newsletter. Got questions about what you saw in today’s episode? Join our free Analytics for Marketers Slack group at TrustInsights.ai/analyticsformarketers. See you next time.

Need help with your marketing AI and analytics?

Leave a Reply Cancel reply

Subscribe to our Weekly Newsletter

Pin It on Pinterest