So What? Generative AI Analytics

So What? Marketing Analytics and Insights Live

airs every Thursday at 1 pm EST.

You can watch on YouTube Live. Be sure to subscribe and follow so you never miss an episode!

In this week’s episode of So What? we walk through what you can and cannot do with generative AI analytics.

So What? Generative AI and Analytics

Watch this video on YouTube

In this episode you’ll learn:

How generative AI applies – and doesn’t apply – to analytics
What tools to use for generative AI analytics applications
The (not-at-all) secret to use generative AI for any analytics use case

Upcoming Episodes:

Have a question or topic you’d like to see us cover? Reach out here: https://www.trustinsights.ai/resources/so-what-the-marketing-analytics-and-insights-show/

AI-Generated Transcript – NOTE – This is machine generated and may not be entirely accurate:

Katie Robbert 0:36
Well, hey everyone. Happy Thursday. Welcome to SWOT the marketing analytics and insights live show. I’m Katie joined above me by Chris and John. How’s it going, guys?

Christopher Penn 0:45
Hello!

Katie Robbert 0:47
I love the enthusiasm.

Christopher Penn 0:50
It’s Thursday!

Katie Robbert 0:55
It is cold. Oh, excuse me, oh my goodness. This week, we are talking about generative AI analytics. So how generative AI applies and doesn’t apply to analytics, what tools to use for generative AI analytics applications. And the not at all secret to use generative AI for any analytics use case. I feel like this is a really timely topic, because analytics is still at the core of what we do. And generative AI is the shiny new object. So how do we bring the two together to make all of our analyses more straightforward? And so Chris, I’m guessing you are going to blow our minds today and just make us dizzy.

Christopher Penn 1:40
I’m gonna irritate you.

Katie Robbert 1:42
I mean, that’s any day ending in Y.

Christopher Penn 1:47
Ha! Okay, so I think we should preface this by saying that there are definitely specialty vendors that are trying to build tools in this space, and we will not be talking about them. Today, we will be talking about using the base tools that people are familiar with ChatGPT, Claude, etc. And not individual vendors. So I just want to put that out there in front, because otherwise I will say, Well, this company does like I’m sure they do. I have not tested this software. So I don’t know. That’s fair. Okay. So let’s talk about analytics. Probably the, the most common data source that everyone’s used to is good old fashioned Google Analytics 4, the the bane of, of marketers all over. By the way, a quick reminder, if you have not backed up your data from Google from Universal Analytics, you should do that sooner rather than later that will Google at some point, we’ll be taking that away, I believe it was July one of this. And I think they move that back to January one or 2025. But either way, you should export that data and save it because you will go away permanently, after a certain amount of time, and you will never ever get it back.

Katie Robbert 2:58
I would say if you have questions about that reach out to us trust insights.ai/contact.

Christopher Penn 3:06
So Google Analytics is probably the most common form of of digital market analytics and people are aware of the challenge with it is that there are no direct hooks to getting its data from here into a general AI system. So you need to essentially get your data out of here first, then you can take it to different language models, and ask them questions about your data. Now, here’s the the other challenge. language models, you know, for example, are French ChatGPT, or Claude or any of these things are really good at language. They’re good at processing and predicting the next sequence of words or something. They are not good at math. They don’t know how to do math. What to the extent that they can do math in their bare bones version? It is because they have seen that math somewhere before. So if you ask a language model, this is particularly in the early days, you asked it like two plus two it could answer for because it’s seen that example, a bunch of times, if you asked you 176 plus 1212 13, it would hallucinate because it wasn’t sure what to do with those numbers cuz it can’t add it, he can’t do math.

Katie Robbert 4:17
Which is really weird. And I say that because I feel like that’s the misunderstanding of what these tools are meant to do it. You know, if you ask, you know, a Google search bar, what that same equation, it’s going to give you an answer. And I think that there’s a misunderstanding that these tools are identical, if not more advanced. And I think that that’s a really good distinction to call out.

Christopher Penn 4:46
You know, they’re not their language machines. The core models themselves are just word prediction machines. They’re really good at predicting sequences of words that they have seen before. Sometimes math is expressed like that. But not that often. The work around that some tools ChatGPT, in particular, and many other tools in the analytics space is that they write code, they write code behind the scenes, that and because that is language, and then they run the code in a virtual environment that can then do the math. So let’s take, let’s take a pretty straightforward example of some data. So I’m gonna take some data out of my Google Analytics instance here. And this is real simple stuff, I said, I just want the number of sessions by channel for the, for the past two months, and the two months prior. So this is from my personal website. And we can see here I had, for example, 18,000 visits, sessions from email in that two month period at 37,000, in the most recent two month period, and so on and so forth, relatively straightforward. The way that you would typically use this kind of data is you would you would put it into a prompt, and he would ask it of language models. Does that seem clear so far? Katie?

Katie Robbert 6:05
Yes. I’m curious to see where this is going. Because in my mind, I’m like, Well, you have everything you need, why do you need to bring in generative AI, because you wanted to know session for the past few months and sessions for the past 30 days, or previous period? And you have that data? And you have percent change? And you have absolute change, I’m guessing is the next one.

Christopher Penn 6:29
Yeah. So columns II through I added those, right, those did not come out of Google Analytics.

Katie Robbert 6:37
But that’s a basic calculation. So yes, it makes sense so far. But I’m curious to see where this is going. Because in my mind, I’m like, Okay, you’re done, you got the answer.

Christopher Penn 6:49
And I think that’s actually a good. sidebar to this is, when you’re, when you’re thinking about generative AI and analytics, you probably should be thinking about what your user story and use case is that you’d want to use this first. Because if you’re just to do math, you don’t, you don’t need general AI to do math.

Katie Robbert 7:11
So as a friendly reminder, if you’re unfamiliar user stories, a simple sentence comprised of three parts as a persona, I want to so that the persona being who you are, or who you’re representing the want to being the action, and the so with that being the outcome. And so in this case, it sounds like you started with, I want to know, the sessions to my website for the past two months, and then the previous periods, and how much it is either grown or not grown.

Christopher Penn 7:37
So that I can decide what do I need to do to make my website better? Yeah, exactly. That’s, that is the use case. So the way that you would take this data in would be through using a prompt. So I’ll give you an example prompt that says, you act as a Google Analytics expert, you have knowledge of Google Analytics, 4, Google Marketing Platform, blah, blah, blah, all these different terms. Here’s some background information. This is following. If you’re not familiar, the what we call the race framework, roll action, context execution that you can get this for free on the Trust Insights website. If you go to trust insights.ai/prompt sheet, it’s a PDF. And so I have my role, I have my action, I have my context, which is all this information, which are things that I think are important about analytics. So I have some some experience with analytics. And then here’s the data itself. Here is the execution statement, do this thing. So let’s go ahead and take this. What I want to do is I’m going to bring up a very interesting service. This is a service called chat, chat bot arena, Spy LM systems. And I want to bring this up for a couple of reasons. One of the things chat bot arena allows you to do is allows you to compare models side by side to see which models handle this sort of information. So I’m gonna go to side by side arena. And let’s say let’s start with a OpenAI is GPT-4 Turbo, which is their their biggest best model. And then next to it, we have our choice of different models that we could look at here. Well, let’s take let’s say Google’s Gemini, the one that powers Bard, right, that’s a that’s a good choice. These are two very popular models, actually. Well, is Gemini good. Gemini is more of a, well, they’ll sleep with them.

Katie Robbert 9:25
Well, it sounds like you first need to understand what all the different tools are. So you know, as I’m looking at this list, I know what GPT-4 is. So I’m obviously not a power user, like you are more of a casual user. I didn’t know what half of those are. I recognize GPT. I recognize llama. I recognize a couple of others. But I don’t know that Gemini for the casual user is a common model.

Christopher Penn 9:51
That’s true. That’s your Yeah, you would only know it as Google Bard stays Google Bard. Yeah, so there’s a bunch of other ones. So this is an open source models like MCs drawl in there, there’s OpenAI, as GPT-3, point five, there’s Anthropos, Claude, a few versions of that. And then there’s a bunch of open source models. I would say for for what we’re doing today, we’ll stick with the commercial providers, for example, GPT-4, Gemini from Google and a cloth from anthropic, because those are the same models of power, the very common interfaces that everybody knows, but you would use, you can and should use this particular system if you want to evaluate which of these different models would be good for a specific kind of Prop, which is exactly what we want to do here today. Okay. All right. So I’m gonna take my massive prompt that has all my data in it, because I took those those two tables right out of Google Analytics. And I said, summarize these findings, taking into consideration the background information, comparing the previous period to the current period, include percentage and absolute numeric changes from period to period and the analysis. So we’re gonna send these two. It says, you hit that really great limit on this, this model for GPT-4 Turbo.

Katie Robbert 11:04
That’s a pretty Thunderdome. I mean, if it’s over in like, point two seconds, you know, I’m here for the show. What about you, John?

John Wall 11:15
Yeah, no, exactly. If you Tina Turner would not approve, obviously.

Christopher Penn 11:20
So Gemini says direct traffic saw significant increase from 5696 to 80 to 65. Sessions, representing a 45% increase could potentially indicate issues of tagging and data governance on a site traffic increased to 11.5 to 20 to 70, email traffic grew substantially from 1867 to 37 to 94, it was representing an impressive 100% increase, organic search traffic remained relatively stable. referral traffic was a 70%, this decline, and so on and so forth. So this is what Gemini has said. Now. Here’s something funny. Let’s see. Gemini is saying. So Gemini is getting a math correct. 45% increase for direct traffic from period to period. On a sign went up, 98% Yep. on assignment 98%. So Gemini, at least can do basic counting, which is good going email going from 18, six, or seven to 37294, which is 100%. Increase. So far, so good. Next, let’s take our second follow on prompt, which is now build a marketing strategy with recommendations based on this information. Because you’re right, just doing the the math Yeah, you can absolutely do that and Excel. Well.

Katie Robbert 12:37
And so that’s it. I’ve been sort of on the edge of my seat going, where’s this going? Because that first prompt, now it’s starting to make sense, like, Okay, you really just want to first validate that the model can read the data correctly. That’s number one. So you have to go through that step. You can’t just say, here’s some data, create a strategy, you need to validate first, that it can actually read it, then you can move on to what the heck do I do with this information?

Christopher Penn 13:06
Exactly. And this, there’s a very subtle gotcha in there as well. language models get better at generating results, more more verbose they are, the more time they have to talk out loud. And the reason for this is a mathematical reasons for it, especially with industry we should have or sad distribution tokens. But the simplified version is if you ask a model to to go through things step by step and talk through its thinking, it’s going to assemble more words that are related to the initial prompt. And because of the way the Transformers architecture works, it takes into account all the words generated in addition to your original words. And so it becomes stronger at writing about that topic, the longer it goes on. If I gave it a table that just create a marketing strategy, there’s not enough words there for it to, to come up with a great answer, as opposed to talking through everything first to create more words that are relevant, that you can then refine, that’s why short prompts deliver, typically deliver worse results and longer prompts because the more longer problem is, the more words there are for to consider. And the easier it is to find the probabilities of what words to say next.

Katie Robbert 14:22
Do you think that people who are using these systems to try to create strategies, this is what they’re doing? They’re basically saying, here’s some data. What do I do with this? And it’s it’s very generic and short sighted.

Christopher Penn 14:36
I’m not going to name any names, but a prominent marketing influencer, got called on the carpet by a number of folks recently, for very generic bland short prompts that were going to generate terrible content. And it’s because fundamentally, the person in question doesn’t actually understand AI. They don’t understand the gender of AI and so the advice they’re giving is very generic and it’s going to deliver subpar results compared to someone like you who knows the some of the architecture behind the systems to say like, yeah, they need to talk. They are talkers they need that they need to be for good or ill they need to mansplain a bunch of self stuff even to themselves, just so that they get the right words in context.

Katie Robbert 15:24
Because, you know, I need another thing mansplaining to me on a daily basis was this prominent influencer named John J. Wall chief statistician of Trust Insights, not at all.

John Wall 15:39
Okay, long prompts, it’s all about that.

Christopher Penn 15:43
so OpenAI is modeled did decide to get working can see here GPT-4 Turbo has presented the same math, it can count, which is good. Just quickly compared to the spreadsheet that I have, yes, is correctly counted, which is great. Now let’s see what OpenAI is marketing strategy recommendations. email continue to optimize email campaigns have a B testing segment email is more graphically as an award director traffic review and correct any tagging and tracking issues, organic search, audit SEO practices on assigned checker tracking, strengthen relationships with referral sites organic social reassess your social media strategy, organic video, invest in higher quality video production, three to three tactical tips, improve tracking and tagging, SEO optimization and social media engagement. Now we head on over to Gemini, and it says email marketing. So Gemini just went through my channel with animals well, you continue to invest in email marketing traffic, great conduct AV testing, yep. Do segmentation, SEO, do an audit, create relevant content, referral traffic partner with more and better websites, social media strategy, reevaluate your strategy because you suck at it. And develop a video marketing strategy for the most important things conducted comprehensive SEO audit influenced email marketing campaign and develop a video marketing strategy. So that’s both these two models side by side and their assessment of the same data.

Katie Robbert 17:18
I’m underwhelmed by both? Well, I guess it also looks like Gemini doesn’t know where the spacebar is.

John Wall 17:26
Yeah, I was gonna ask him about that on the titles. That’s bizarro? It is this is good, though. Like, because I’ve seen this in a number of things where if you have a pile of stuff, and you’ll look at it, you’ll say, Okay, I think we need to do these 10 things. And then you run it through the model, and it’ll give you a list of 20 things. And you will probably find like three or four you like, oh, yeah, I should have thought of that one. Like that’s a good one. Like it casts a decent net. But yeah, it is funny to how these are all kind of like all the generic info. Like I’m sorry, but AB testing is not just going to solve all your email problems, even though that’s you know, the number one option.

Katie Robbert 18:01
Yeah, it feels very much like alright, what is the most bland vanilla advice that I can give? That’s not incorrect. But also not super specific, because I only have this one tiny data set. So let me at least say something. It feels very much like I brought my data to an overpriced consultant and this is what they gave me back and I’m underwhelmed.

Christopher Penn 18:26
Now of these two results, which one is better?

Katie Robbert 18:30
They feel the same.

Christopher Penn 18:34
I’d vote for even just because OpenAI can find the spacebar.

Katie Robbert 18:38
Yeah. Well, yeah, ease of readability, OpenAI is better. But yeah, I mean, I honestly, I feel like they’re both equally underwhelming

Christopher Penn 18:49
They are. And again, this is partly because A is partly because it’s relatively simple data set. Like if I hand this, if I hand this spreadsheet to you, as a human, I could do the first thing you’re probably gonna say is, I ask them questions. There’s not enough information to make a judgement here.

Katie Robbert 19:05
Yeah. What do you want to do? What? You know, where do you have, you know, budget? Who are your resources? Who’s on your team? You know, I know, like, I could ask you a bunch of questions. But I know you don’t have the bandwidth to focus on social. And I know, you don’t have the bandwidth to focus on partnerships for better referral traffic, you have the bandwidth to focus on email, and maybe some organic search through content creation. And so I would say, Okay, so the other channels aren’t terrible, or do you really not care about social and video, etc. So where can we refocus and get you to boost some numbers?

Christopher Penn 19:45
So let’s do another example. Let’s go back into Google Analytics was going to our explore hub this time, and let’s create an exploration and let’s do less 28 days is good for dimensions. I want to do the page Of the of my content. So let’s do page title, import that. For our metrics. Let’s do sessions. And let’s do events. Where’s my event count, event count. And let’s do conversions.

Katie Robbert 20:24
While you’re pulling that up, I will do a quick plug. I know that a lot of people have given the feedback that Google Analytics 4 reporting is a nightmare. And they’re not wrong. But if you want help with that, we still are foundational services include Google Analytics. So if you want some support with your reporting, please give us a shout. And John will take care of you. Maybe, depending on his mood.

John Wall 20:47
Bring a bag of money and no job too big no fee to big.

Christopher Penn 20:56
Thank you, Ghostbusters. Alright, so we now have a much more thorough piece of pie of data. So this is the pages on my website, the number of sessions events and conversions for those pages. So let’s take this now. And let’s feed this into a prompt and see what happens when I restart my my thing here. And I’m gonna take a prompt. I’m gonna take let’s choose who do you want to choose for our let’s go with GPT-4 Turbo. What do Claude this time? Yeah, right, Claude 2.1. This time? Let’s start with our prompt basics here.

Katie Robbert 21:39
Now that you’re using this chat Thunderdome thing, is this a paid service? And? Okay, and so I know that if you’re using, like, ChatGPT, to use the analysis portion, that’s a paid service. So how does that work here? Well, that’s

Christopher Penn 21:56
so that’s not in here, we’re gonna get to that in just a bit. Okay, because this is just using the language model portions. Got it. Have pages and metrics associated with the pages on my website, produce some useful insights about what content is working for me. Okay, so let’s take that let’s take our data set itself. And go.

Katie Robbert 22:31
So interesting, you didn’t clean up the data at all? Nope. You didn’t remove any of like the headers or the junk at the end? You’re just saying, Here you go. Good luck.

Christopher Penn 22:43
And part of the reason for that is that’s what someone who maybe is not necessarily skilled at data analysis might do. The whole reason you might be thinking about using generative AI with your analytics is because you’re like, I am not an analyst. So machine, you do the work for me.

Katie Robbert 23:00
Which is fair, which is fair. I do think that, at the very least, if you’re exporting your data from a system like Google Analytics, or other systems, just give your data and eyeball for us, because you’ll see that a lot of these systems will generate unnecessary information like headers. Yeah, so what Chris has up like the first five rows, you don’t need that in the analysis. It just basically says, you know, here’s the account. Here’s the type of report and here’s the date range. You don’t exactly in there.

Christopher Penn 23:34
One of the things that solves up virtually every system with Google Analytics data is they always put a summary row in somewhere weird, they don’t label it, and that throws off everything. Thanks Google.

Katie Robbert 23:44
Super helpful.

Christopher Penn 23:44
All right, let’s see how our two contestants did. So we have GPT-4 Turbo and we have Claude it’s says here’s some insights that can provide your newsletter top performing content your My Account page has the highest number sessions events versions its newsletter content Yep. Newsletter attraction pages related to the almost have a newsletter should you buy customers up to consistently seeing good engagement rates? content related gender and marketing the strategy of performing well well no, no, that’s all the content I’ve been doing lately. It’s not surprising the page thanks for taking my poll has comparatively high number of conversions Yes, slick thank you page subscribe to has new some of the sessions and count the conversion rates lower Yes, because that’s a landing page. content with lower engagement to receive fewer sessions and conversions may need to be optimized short presence is not set. There’s no direct data set. Of course that’s it’s only it’s only page data, strategic recommendations enhance your newsletter signup process did that develop more AI and marketing content doing that data collection? Okay, so not bad from from GPT-4 lets you know Claude, did your newsletter are doing 50% and conversions are going to these two pages? Yeah. blog content will Ralston she’s driving decent traffic conversion rates are lower. That’s not too specific social media traffic is unavailable will draw several generative AI focus posts ranking well for traffic, I’ve lower conversion rates, comments and account matching pages are very high churn rates and overall Traffic Conversion bots are healthy. Okay, so that’s what the two different models spit out from this fairly long dataset. How are you feeling sad about this?

Katie Robbert 25:27
I’m once again underwhelmed, because I’m like, okay, is that this is the recommendations. This, again, I could look at the data and know all of that, I don’t need to put it into a machine learning model, to know that your email newsletter is the highest converting for your website, or that you know, the variety of different things that it says like, I get all of that from the data when I’m looking at it in the spreadsheet. So I’m still sort of stuck on why I need this step. And maybe it’s just me personally, maybe I’m an n of one. And because I know how to analyze data, I’m sort of trying to figure out where this comes in useful.

Christopher Penn 26:11
Put yourself in the shoes of maybe the a junior person on our team, you just you’re fresh out fresh out of college. And Chris, the annoying, you know, tech guy says, Hey, here’s what it looks like, Did it go analyze it? And that’s all the direction Chris gives us. Chris is a terrible manager. In that scenario, would this be an appropriate response for that junior person as opposed to staring at the screen? But I don’t know what to do?

Katie Robbert 26:42
I see. Yeah, I guess it’s a good starting place to at least get a baseline analysis of the data. And then you can take it further. So you know, for example, if I’m looking on the left hand side of the GPT model, and I’m seeing content audits and SEO, perform content audits to identify refreshed underperforming pages use SEO best practices. So that’s a good starting place. What I would then say to the junior associate who is doing this, like, Okay, let’s go let’s take it a step further and come up with more specific recommendations for SEO. So how can we dig into this data? And figure out what is the most useful SEO, you know, set of tactics for Chris, specifically? So it is it’s a good conversation starter to that point, but it’s not the full picture. It’s not the full analysis by any means.

Christopher Penn 27:38
So our junior employee says, Wait, how do I do a content audit?

Christopher Penn 27:47
Yeah. So Claude is saying set some goals inventory, you’re just in content, analyze individual performance, traffic engagement, metrics, search rankings, content, quality, depth, accuracy, formatting, optimization of top and bottom content items, look for content gaps, and identify actionable recommendations. That stuff again, for that junior person who is not super skilled. That’s not bad.

Katie Robbert 28:14
Well, and I guess we should stop saying junior person, because it could be any marketer who just this isn’t their core competency. So for me, for example, like I’m not a content marketing expert. So it’s helpful for me. If you say, you know, Katie, our content is underperforming? Where should we start? I’d probably be like, Oh, that’s a really good question, not what I focus on. So let me think about, you know, what a content audit would look like. So I can see how this is useful to at least get started. Because even that I’m looking at the clawed recommendations, they still feel really generic and not instructive. It’s just more of like a, and theoretically, here’s all the things you could do. But it’s not saying, okay, when you say inventory, all of your content, what does that mean? What do you actually need to know about the content to consider it inventoried? Like, do I need to say, I have five pieces on this topic and six pieces on this topic? Okay, now what?

Christopher Penn 29:14
So I’m going to say here that, that GPT-4 is certainly more aligned with what we would consider Best Practices than Claude is here, because we would start with the five P’s the purpose being like, what are you trying to do? Like, what what’s the what’s your goal? And here it says, Before diving into the audit to find what you want to achieve, you want better rankings, increase user engagement, and so on and so forth. Whereas clause like, hey, just get going and do all this stuff.

Katie Robbert 29:43
So it sounds like instead of just saying, What should I do? A better prompt would have been for us to go through the five P’s. include that information with this data that we pulled from the website and say, Okay, help us put together or a strategy based on this information instead of just asking the system? What do you think I should do, which I’m guessing a lot of people are still doing like, Well, okay, that’s great. But what’s next, what else? And it’s just not really directive, it’s still super generic. Exactly.

Christopher Penn 30:15
And again, when they talked about the content inventory here, this is actually fairly good right here from the GPT-4 Turbo model, saying, hey, create a comprehensive list, your website tools like Screaming Frog SEMrush, or website crawl will help you generate those two URLs, and include these pieces of information. So who is your URL page titled, The various metrics, collect and analyze your data, quantity, qualitative analysis, relevance to your current audit strategy, categorize your findings, prioritize your actions, that’s actually pretty good. Whereas Claude just goes very generic, and doesn’t doesn’t really dig into useful detail there.

Katie Robbert 30:52
And I agree, I think that the GPT, step by step, it’s like, okay, you can now put that into a project plan, and have someone start executing against it. So John, look out, this is going to be on your plate in about an hour.

John Wall 31:04
That’s right, I’ll get the punch list.

Christopher Penn 31:08
Now, we’ve been talking and using the pure language model version of these tools for the last 31 minutes. The one of the tools on the market ChatGPT has an advanced data analysis tool, what this tool does, which is better than pretty much anything else on the market for the beginning, the the entry level tool set is that it will process the data using actual math. So let’s do this, we’re going to take that exact same prompt, go back to Tatry retrieve the original prompt here.

Katie Robbert 31:49
This was the prompt that included the data and gave the background. Exactly.

Christopher Penn 31:53
So I’m going to include that prompt. And now we’re going to attach the actual data file. So I’m gonna load the spreadsheet itself uncleaned Same, same as we started with. And what it’s going to do is behind the scenes, I had to click on little drop down, it is going to do what language models do best, which is right there, you can right language. Uh huh. So it is discovered those stupid heteros. So let’s see, I’m gonna attempt to load the CSV with a Morse with a more flexible approach. And what it’s doing behind the scenes, it is writing Python is was writing Python code to process the data says, Hey, I found a bunch of crap here, I’m gonna trim that off. Never had an error.

Katie Robbert 32:45
For those of you watching, one of the things you may not be able to see on the screen, because it’s going by quickly is for every code snippet, it has a little box in the top right hand corner that says copy code. And so this is a great way for you to start building your own code base. So you know, let’s say you know that this is a question that’s going to come up over and over and over again. And you happen to know how to code in Python or whatever the system is, you can say, Okay, let me use generative AI to help me write the code so that I don’t have to start from scratch, just as a pro tip. Exactly.

Christopher Penn 33:20
So here you can see, it’s skipping a certain number of rows, it’s got the the main head that says, I’ve successfully loaded the data. Here’s what I see the title of the page sessions, event count and conversions, I’ll first clean the data by removing the summary row converting the conversions column to a numeric format, handling non numeric values appropriately. So it has it has gone through and done that. And now it says, here’s some key aspects, we’re gonna explore popular pages, engage in content, conversions, and paid efficiency, which pages are most efficient in terms of balancing traffic with engagement, event count, and conversion. So actually, in terms of things that you’d want to do with your data, this is pretty good. Because this is on guided, we have not re prompted the software yet. So it’s creating these three different datasets. And as Katie said, I can copy this code, let’s pull up VS code over here. Start a new piece of code here, we’ll call this J. Four is that’s not rubies, whatever. This is usable running code that you can run. So let’s see how it’s done. Here’s the analysis the results that says popular pages, my account is the most visited page of the top pages engaging content by event count. Conversions data is not recorded or not recorded properly, which is meant to take a look at that page efficiency. You can’t do that. Here’s some recommendations improve conversion tracking focus hypermart content enhanced data quality. So now we might say okay, great. Give me based on what you’ve seen, give some content to book recommendations for future content that are likely to do well. And because again, this is Python based data, it can then loads the Python data back into its main memory, the context window. And now, it’s the leveraging the language portion of language model to actually look at the words and phrases in column A of the spreadsheet, what are the topics and coming up with these answers? So one of the things that’s really nice about the OpenAI system is that it can flip back and forth, because it’s got under the hood, it’s basically four or five different models all working together, but can put all that stuff back together into one coherent stream. Whereas with other tools, you’d have to hop around from tool to tool.

Katie Robbert 35:49
What’s interesting, is, it’s giving recommendations that are a little different from what we got from the other head to head, because it could really do a more in depth analysis of looking at the data. And so the very first recommendation is go up in depth articles on marketing, analytics, and AI. So that’s what you do already, that’s the majority of the content that you create. The rest of these aren’t things that you necessarily focus on all that much. And so that, to me, is the interesting part of when you looking at the overall inventory of your content, and what’s performing, there’s a lot of things here that you don’t have a lot of, and so that I think is a really solid recommendation of like, okay, you probably like you know, email marketing inside out. Definitely write more about that. Things that are more interactive, things that are more educational, you do write educational content, but it tends to be at a 301 or a 501 level. So I think there would be definitely some exploration there. emerging trends in digital marketing? That’s a question that comes up all the time. And so what could you write about that? So I think that this is really interesting.

Christopher Penn 37:08
And you’re correct, because the the context window for this model now contains the quantitative analysis from the Python code that it ran, it can give better recommendations. Because again, the pure language models can’t do math, because it can’t do math, they have no understanding of what it is they’re looking at. They’re just trying to predict words based off the numbers. This is, as we’ve distilled down all the math into specifics. So what we get out here is pretty good. And then of course, you can continue to have conversations from here. So this is the not so secret part of gen of AI and analytics, which is that these things can’t do math. So use tools that can create the code that can run against your data. And then take the results from the data analysis. And put that in here, as opposed to try and put the raw data in itself. Because the raw data itself, it’s like ingredients, right? You can go to your, your your pantry, and just like take handfuls of raw flour and eat it and you won’t you won’t die it, it’s nutritious. It’s just Yeah, exactly. It’s gonna taste like eating raw flour. Or you can bake some bread with it, and then eat the bread, which is the distilled down product. There are a series of processes, that’s what’s going on here, under the hood. The advantage of this particular system is that it obviously can do that does that for you without having you having to write code. Now I will say, the code part, as you mentioned, Katie is really important. If it’s something you have to do over and over again, you can, if you have the technical skill yourself, or you have someone on your team who has technical skill, you can get that code out, run it first, and then skip right to the analysis part.

Katie Robbert 38:51
I feel like you’re going to have to keep saying, you know, maybe even just to me that gender of AI can’t do math, because for some reason, I feel like that is such a hard reality to wrap my head around because it seems like it should be able to because of the you know, qualifier artificial intelligence, will Dutch should be able to do math, and these are language models is and isn’t math, just another language? And the answer is in this instance, no. And so I think that that is such an important piece of when we talk about generative AI analysis, we’re not talking about the system’s doing the math to analyze the data. It is taking the information and writing a narrative around what it’s reading.

Christopher Penn 39:43
Exactly. They don’t have any actual reasoning capability. They don’t have any actual cognitive abilities. They they’re not thinking machines. These are prediction algorithms, and they’re very good at the things they’re very good. that they’re very bad at the things they’re very bad. And, to your point, a lot of people assume they’re kind of like magic boxes, there’s no magic here it is literally all just mathematics and algorithms under the hood. So part of using gendered AI with your analytics is knowing when you’re doing a task that is a language based task versus a non language based task, the smartest thing you can do is do the actual analysis in the tool of your choice, right, you don’t have to use ChatGPT. For that, as you as we started off the show today, you can do that here, and just good old fashioned Excel, there’s nothing wrong with this. In fact, you’re pretty much guaranteed to get correct math out of this 100% of the time, right? Addition, subtraction, multiplication, division, Excel, amazing. But then, you can take the analysis portion, have the language model, write the language around and say, Here’s my data. What conclusions could you draw? Where are the weak spots? I really liked in this content analysis, you did upload the spreadsheet of these different gaps and say, Okay, well, this is great. Tell me about a the gaps in my content, assume I have an audience of senior marketers, the VPs, CMOs, etc. What sorts of content based on the data we’ve analyzed, so far, would appeal most to this audience. So content gap analysis, again, that’s language. And now based on on the tools, knowledge of our audience, and you might want to provide an actual ideal customer profile as part of this, it would then be able to to give you more and more useful stuff like hey, there’s things that are not in your content, you know, advanced analytics, data driven decision making case studies of successful campaigns, ROI, performance management, leadership, and management, and marketing. These are all things that are Yeah, in my, because I think I use the last 28 days worth of page content from Google Analytics. This is not stuff I’ve written about last 20 days, because it’s all been generative AI.

Katie Robbert 42:16
Right. But this is information that we could say, Oh, guess who can write about that stuff? The person at Trust Insights, who knows all of those things inside and out?

Christopher Penn 42:27
Exactly. And so the final takeaway here really is for these generative AI tools. They are word prediction machines, if you’re predicting words, you’re going to be just fine with the tasks that are their best predicting words that use them for the tasks that are mathematics or calculation based, do that in a different tool, or use a tool that has a separate mathematics module so that you’re going to get a correct answers, and be the tool can then pass along that data to a language model to craft language around your data.

Katie Robbert 43:06
John, what data are you going to start to throw it your ChatGPT? System?

John Wall 43:09
Oh, that was the first mistake I made when I was playing around with prompts. like months ago, I was bothering Chris, I’m like, come on, I just asked it, you know, what are the five largest companies, you know, in this vertical? And whatever? And he’s like, yeah, no, they don’t work like that. That’s not anything like that. And so, you know, I tried like 20 More prompts. And yeah, I did not get any of that stuff. And so I think one area to be aware of, though, that’s really important, and we have this has been a use case for us, is a lot of times you just get to a technical answer. And you need something to rewrite, you know, it’s that rewrite function where you can say, okay, the data says these five things, write this in a way someone who has no marketing understanding, you know, can comprehend. And so you can simplify and make something more understandable to people who don’t know, you know, can’t do the calculations themselves or don’t know what’s going on marketing. So they’re, you know, there’s some opportunity there. But yeah, as far as doing like the heavy lifting and doing the regression analysis says for you, that’s not happening.

Christopher Penn 44:11
Yep. The other part that Katie touched on this as well, the Data Prep itself, you probably should spend some time doing Data Prep, because with any consistent, it’s always garbage in garbage out, when you look at what is inside that data file itself. You know, even just something as simple as what’s in the page titles like yeah, that’s from my WordPress blog. It’s carrying over that extra content, my name and you know, the marketing keynote speaker that’s unnecessary. We don’t need that. In here, this it’s pulling in some comments pages and things. We don’t want that we don’t need that. So spending some time doing old good old fashioned Data Prep will help get better results.

Katie Robbert 44:55
And I think well, so there’s that and then yeah, just making sure that The information is correct. You know, I’m seeing, you know, you write about things that aren’t related to marketing at all. So for example, while you’re probably wrong about lighter fluid, I know you use your personal blog for just sort of like, here’s everything that’s on my mind. And so, you know, if you’re looking to attract a certain audience, you would want to filter this data, once you pull it, clean it up, but filter it by things that are relevant to the relevant audience, or things that I guess I should rephrase that to say, things that you feel will bring in a paying audience that you know, appropriate to your business.

Christopher Penn 45:45
Exactly. For years, this the consulting billing rates thing was the one of the top pages on my website, and it brought in a whole bunch of folks that were not a good fit for, for my business. Obviously, that’s changed since then. But But yeah, having the ability to look at that data and clean it up and remove stuff that is not important is is vital. So that’s gendered AI and analytics. It is not as with with the beginning with the initial toolset that you start with, what’s out there, is capable in good in some areas, not capable and others. And your best bet is to not pretend it’s a magic box that can do everything, but recognize the strengths and weaknesses.

Katie Robbert 46:32
And if you want support with that, we do that Chris literally just showed that we do that. So you can go to trust insights.ai/ai services if you want to learn to do it yourself. You can take our generative AI for marketers course at trust insights.ai/ai cores. If you have other questions, you can go to trust insights.ai/contact. Or if you want to just join the conversation and see what people are talking about, you can join our free slack group at trust insights.ai/analytics for marketers, what did I miss?

Christopher Penn 47:01
That’s it for right now. I think that’s good for today, and we will see you all next time. Thanks for watching today. Be sure to subscribe to our show wherever you’re watching it. For more resources. And to learn more. Check out the Trust Insights podcast at trust insights.ai/t AI podcast and a weekly email newsletter at trust insights.ai/newsletter Got questions about what you saw on today’s episode. Join our free analytics for markers slack group at trust insights.ai/analytics for marketers See you next time.

Transcribed by https://otter.ai

Need help with your marketing AI and analytics?

So What? Marketing Analytics and Insights Live

airs every Thursday at 1 pm EST.

In this episode you’ll learn:

Upcoming Episodes:

Leave a Reply Cancel reply

Subscribe to our Weekly Newsletter

Pin It on Pinterest