Root Cause Analysis

This data was originally featured in the November 1, 2023 newsletter found here: https://www.trustinsights.ai/blog/2023/11/inbox-insights-november-1-2023-chatgpt-survey-analysis-root-cause-analysis/.

Today, let’s tackle a question we’re all faced with at one point or another in your careers for data analysis or as a data scientist:

Why are the numbers down? (or some variation thereof)

Answering this question can be one of the most challenging tasks you’ll face, partly because the question is an important one, but also because it’s usually accompanied by an outsized sense of urgency. The boss comes storming into the office/your desk/Zoom and demands to know why the numbers are down, or the client asks that very pointed question on the call.

How should you answer that question? To the topic of anticipation that Katie and I talked about this week on the podcast, how do you plan to answer that question, knowing it’s coming?

Unsurprisingly, the answer looks like every other answer in data analysis and data science: understand the problem, gather the data, analyze it, figure out what correlates to the outcome in question, and then develop a testing plan if you don’t have a clear answer. So let’s step through this process.

We’ll start with a barely fictional scenario: hey, our website traffic is down! Why?

On its face, this problem could have dozens, if not hundreds of answers. The first step is to understand the problem. The question we start with is, down compared to what? Last week? Last month? How are we down? If we start with the default view of our Google Analytics 4 data, it’s not terribly helpful:

It’s not even apparent that traffic IS down, but let’s say for argument’s sake that it is. Website traffic is an aggregated number made up of lots of subcomponents, like marketing channels. So our first step would be to break apart our aggregated number into pieces:

We had to switch to Looker Studio because Google Analytics 4 can’t do this kind of clustering and grouping. What we see is pretty apparent: direct traffic and organic search traffic are on the decline. Social and referral traffic are more or less steady. Email traffic has gone nuts the entire year.

By breaking up our big number into components, we now have a place to start investigating. We can’t do much about direct traffic, but we can look at things like organic search traffic. Now, search traffic mainly comes from Google, and even there, we have another big number we can break into components. Specifically, we can break search traffic into two pieces: impressions and clicks.

When we examine the data, we see that impressions and clicks are more or less correlated, but there are plenty of spots in the last year where clicks fall below impressions. This shearing means that our site is showing up in search results, but our audience doesn’t think what’s being shown is a good fit for their intent. As a result, we don’t earn the click.

What’s our next step? We’ve understood the problem. We’ve gathered the data. We’ve isolated what the problem is. Now we need to come up with a plan to fix the problem. Let’s look at our Search Console data in greater detail. What are the pages that are getting lots of impressions (and which terms) but not getting the clicks?

That’s quite a lot. That first page is one of our webinars, and let’s be honest… this page is kind of thin. Real thin.

We know that Google’s recent algorithm changes have focused heavily on helpful content and this… well, it isn’t. At the very least, it’s not very telling for what you’re going to get. So how would we test out whether or not this is the problem? We’d make some revisions to the page and then measure its performance over time to see if it gets ANY clicks at all once we’ve revised it. It’s so far gotten 0 clicks this year, so literally anything would be an improvement.

This is a webinar, so we have lots of potential content for the page to make it more useful. We can put a featured image on it so people know what they’re getting. We can fix up the meta description so it’s more clear. We can take a transcript of the session and use generative AI to write a detailed outline. Let’s take all these steps and implement them.

This is the last stage of the process, where you test out changes and track what happens. This is a case where it’s pretty clear anything would be an improvement over what was there – but had we not dug deep into why our overall website traffic was down, we might not have uncovered this and fixed it up.

When your stakeholders appear at your doorstep asking why a number is down, bring out this process. Understand, gather, analyze, and test. It’s the same process we use for our clients to help them fix their numbers when they’re going the wrong way.

Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.