INBOX INSIGHTS, March 12, 2025: Foundations in AI, AI Model Benchmarking - Trust Insights Marketing Analytics Consulting

INBOX INSIGHTS: Foundations in AI, AI Model Benchmarking (2025-03-12) :: View in browser

👉 Take our new course, Mastering Prompt Engineering for Marketers!

The Foundation Still Matters

I was sitting in a lecture hall at Worcester Polytechnic Institute (WPI) last week, speaking to a room of bright students through the Women in Data Science (WIDS) program. The energy in the room shifted noticeably when we got to the Q&A section, and a hand tentatively went up.

“What should we even be studying if AI is going to do all the work anyway?”

I could see the worry in her eyes, and looking around the room, it was clear many others shared the same concern. Having spent years building my career on technical skills that AI can now replicate in seconds, I understand that fear. Seriously, I do.

But here’s the thing – AI isn’t replacing you. It’s replacing the way you work.

The Human-Machine Partnership

Let’s talk about software development as an example. Yes, generative AI can spit out code faster than most humans can type. I’ve been amazed watching it generate in seconds what might have taken me hours.

But who’s telling the AI what to build? Who’s checking if that code actually solves the right problem? Who’s testing edge cases the AI might have missed?

That’s still you.

What’s happening isn’t job elimination – it’s job transformation. Instead of pairing with another developer, you’re now paired with a machine. The foundational knowledge you need hasn’t disappeared; it’s become even more critical.

Your Foundation Is More Important Than Ever

At Trust Insights, we talk constantly about two frameworks that remain essential regardless of how advanced AI becomes. Let’s dive deeper into each one:

The 5P Framework for Process

Purpose: Why are we doing this work? What business problem does it solve? This is where you identify the core objective and how it ties to business outcomes. For example, are you trying to improve customer retention, increase conversion rates, or streamline operations?
People: Who needs to be involved and what skills do they bring? This isn’t just about technical skills – it’s about domain expertise, stakeholder buy-in, and cross-functional collaboration. A successful AI implementation requires subject matter experts who understand the nuances of the business problem.
Process: What steps will we follow to complete the work? This includes defining workflows, governance, feedback loops, and how decisions get made. Even with AI doing much of the execution, you need a clear process for how humans review, validate, and incorporate AI outputs.
Platform: What tools will we use? This covers both the AI technologies and the supporting infrastructure. Are you using generative AI, predictive models, or computer vision? Do you have the right data storage, processing power, and integration capabilities?
Performance: How will we measure success? These are your KPIs and metrics that determine whether the work is delivering value. What quantifiable outcomes will tell you if the AI implementation is successful?

AI might help with the Platform component, but it can’t determine Purpose. It can’t understand the nuanced needs of People. It can follow a Process but can’t design one from scratch that accounts for organizational quirks. And it certainly can’t set meaningful Performance metrics tied to business value.

The 6C Framework for Data Quality

Just last month, I was working with a client who wanted to use AI to predict customer churn. They had invested in sophisticated AI tools but weren’t seeing the expected results. When we dug into it, the issue wasn’t the AI – it was the data. Their data failed several of our 6C quality checks:

Clean: Free from errors and inconsistencies
- Are there duplicate records?
- Have outliers been addressed?
- Is the data formatted consistently?
We found their customer behavior data had significant gaps due to system migrations.
Complete: No missing information
- Are all required fields populated?
- Do you have sufficient historical data?
- Are there any sampling biases?
Several key metrics had incomplete histories, making trend analysis unreliable.
Comprehensive: Covers all aspects needed to answer the question
- Does the data include all variables that influence the outcome?
- Are you capturing both digital and offline interactions?
- Do you have visibility across the entire customer journey?
They were missing important touchpoints from their mobile app, which skewed the analysis.
Chosen: Only relevant data, no noise
- Have you filtered out irrelevant variables?
- Is the data specifically selected to address your question?
- Have you removed redundant or highly correlated variables?
Their dataset included hundreds of variables, many of which weren’t relevant to churn prediction.
Credible: Collected in a valid, trustworthy way
- Is the data collection methodology sound?
- Do you have confidence in the data sources?
- Has the data been validated?
Some of their customer feedback data came from a biased sample that wasn’t representative.
Calculable: Usable by business users who need it
- Can the data be easily accessed and analyzed?
- Is it in a format that works with your tools?
- Can non-technical stakeholders understand the outputs?
The data was spread across multiple systems with inconsistent formatting, making it difficult to use.

After addressing these issues, their AI models started producing significantly more accurate predictions. The AI wasn’t the problem – the data was.

Applying These Frameworks in the AI Era

So how do these frameworks help you stay relevant in the age of AI?

One of the most successful approaches is applying these frameworks as a diagnostic tool. When an AI implementation isn’t delivering expected results, it’s rarely because the AI technology itself is flawed. Almost always, the issues lie in the foundational elements – unclear business objectives, misaligned teams, broken processes, or poor-quality data.

By systematically evaluating each of the 5Ps and 6Cs, you can pinpoint exactly where the gaps are and address them directly. This is where human expertise becomes invaluable – understanding the interconnections between business goals, human factors, processes, technology, and data quality.

Bridging the Gap Between Humans and AI

This is where the opportunity lies for all of us – not in competing with AI, but in managing the interface between AI and business value. The most successful professionals I see today are those who can:

Translate business needs into AI requirements: Understanding what problems need solving and how AI can help
Ensure data quality: Applying the 6C framework to give AI the inputs it needs
Design effective processes: Using the 5P framework to integrate AI into workflows
Interpret and contextualize AI outputs: Adding the human judgment and domain expertise
Bridge technical and business language: Helping non-technical stakeholders understand and trust AI results

Moving Forward: Questions to Ask Yourself

Whether you’re a student wondering about your future career or a professional adapting to AI, ask yourself these questions:

Do I understand the foundational principles behind the work I do? (Not just how to do it, but why it works)
Can I articulate how my work connects to business value?
Do I know how to evaluate the quality of data and processes?
Can I identify when AI is the right solution and when it isn’t?
Am I building skills in areas that require human judgment, creativity, and contextual understanding?

If you can answer yes to these questions, you’re building a foundation that will remain valuable regardless of how AI evolves.

The fear those students expressed at WPI is understandable, but their concerns are based on a misunderstanding of AI’s capabilities. AI isn’t making humans irrelevant – it’s making the human role even more important. The key is to shift from doing the work to orchestrating how the work gets done.

Quick plug – if you want help navigating this change, give me a shout!

What questions do you have about adapting your skills for an AI-enhanced future? How are you seeing the relationship between humans and AI evolve in your organization? I’d love to hear about it in the comments!

Reply to this email to tell me, or come join the conversation in our free Slack Group, Analytics for Marketers.

– Katie Robbert, CEO

Do you have a colleague or friend who needs this newsletter? Send them this link to help them get their own copy:

https://www.trustinsights.ai/newsletter

In this episode of In-Ear Insights, the Trust Insights podcast, Katie and Chris discuss data preparation for generative AI. You’ll learn why having high-quality data is the essential ingredient for getting valuable insights from AI tools. Discover how to ensure your data is clean, credible, and comprehensive, avoiding the pitfalls of ‘garbage in, garbage out’. Explore practical steps you can take to master data quality and make generative AI work effectively for you. Tune in to learn how to take control of your data and unlock the true potential of generative AI!

Watch/listen to this episode of In-Ear Insights here »

Last time on So What? The Marketing Analytics and Insights Livestream, we tackled part 1 on SEO for AI. Catch the episode replay here!

This week on So What? we’ll walk through practical data preparation tips for generative AI. Are you following our YouTube channel? If not, click/tap here to follow us!

Here’s some of our content from recent days that you might have missed. If you read something and enjoy it, please share it with a friend or colleague!

Take your skills to the next level with our premium courses.

Get skilled up with an assortment of our free, on-demand classes.

You could be reaching 32k+ marketers, analysts, data scientists, and executives directly with your ad. Want to learn more? Reach out and contact us.

In this week’s Data Diaries, let’s talk about AI benchmarks. In the AI world, and especially in the generative AI world, AI models are often rated by how they perform on a variety of standardized tests with obscure names like MMLU, GPQA, and other abbreviations that could be secret government agencies, AI tests, or Danzig song titles.

These tests do serve a purpose, and that purpose is to provide apples to apples comparisons – in theory. However, many tests have issues, namely that their test data was included in the training data of models. It’s easy to pass a test if you’ve read the answer sheet in advance.

More important, for you and I, these tests don’t simulate the way WE need to use AI. Your job probably doesn’t require linear algebra on a daily basis, except for a few professions. Your job may not require fluency in 15 different languages. Your job definitely requires reasoning, but not in the abstract, in very tangible, practical use cases that are relevant to your work.

So how do you know whether a new model is worth your time or not? Develop your own benchmark tests. Here’s a simple example of a prompt I might use to benchmark test a series of different models.

You’re a Google Analytics 4 expert. You know GA4, Google Tag Manager (GTM), Google Bigquery, Google Looker Studio (Google Data Studio). You know marketing analytics, metrics, attribution, multitouch attribution (MTA), uplift modeling. Today we’ll be looking at a snapshot of Google Analytics data for the company TrustInsights.ai, a management consulting firm specializing in analytics, data science, and AI. Trust Insights serves midmarket companies, B2B and B2C. Here is a snapshot of key events conversion data, showing the attribution paths from GA4. The conversion chosen is Contact Us form fills, indicating that someone has asked Trust Insights for assistance. (Don’t forget to include the actual screenshot!) Explain the following:

What do you see in the data?

What data should Trust Insights pay attention to?

What data is less important to Trust Insights?

What next steps should Trust Insights take to improve its marketing results?

Here’s the critical part: whatever you choose for a benchmark test, you should know the answer. You should know what the correct answer is, and then measure how well a given model’s response matches that answer.

Here’s a snapshot of this in action, using our screen shot, prompt, and the LM Arena Chatbot Arena battleground:

We can see easily above that the model on the left, GPT-4.5, did a far better job than the model on the right, Discovery.

Your benchmark test suite should encompass the different tasks you value most, from competitive analysis to content creation. Critically, once you settle on your test suite, don’t change the data! Use the exact same prompts, screen shots, and supplementary data so you get apples to apples comparisons across models.

When you implement benchmark testing this way, you’ll figure out which AI products, tools, services, and models best fit YOUR specific needs.

Here’s a roundup of who’s hiring, based on positions shared in the Analytics for Marketers Slack group and other communities.

Are you a member of our free Slack group, Analytics for Marketers? Join 3000+ like-minded marketers who care about data and measuring their success. Membership is free – join today. Members also receive sneak peeks of upcoming data, credible third-party studies we find and like, and much more. Join today!

Imagine a world where your marketing strategies are supercharged by the most cutting-edge technology available – Generative AI. Generative AI has the potential to save you incredible amounts of time and money, and you have the opportunity to be at the forefront. Get up to speed on using generative AI in your business in a thoughtful way with our workshop offering, Generative AI for Marketers.

Workshops: Offer the Generative AI for Marketers half and full day workshops at your company. These hands-on sessions are packed with exercises, resources and practical tips that you can implement immediately.

👉 Click/tap here to book a workshop

Where can you find Trust Insights face-to-face?

Social Media Marketing World, San Diego, March 2025
TraceOne, Miami, April 2025
Content Jam, Chicago, April 2025
SMPS, Washington DC, May 2025
SMPS, Toronto, Ontario, May 2025
SMPS, West Coast, Fall 2025

Going to a conference we should know about? Reach out!

Want some private training at your company? Ask us!

First and most obvious – if you want to talk to us about something specific, especially something we can help with, hit up our contact form.

Where do you spend your time online? Chances are, we’re there too, and would enjoy sharing with you. Here’s where we are – see you there?

Read our disclosures statement for more details, but we’re also compensated by our partners if you buy something through us.

Your ad for your product or service could have been here. It could have reached over 30,000 analytically-minded marketers and business professionals.

Well, there’s always next week’s issue. Want to be in it? Contact us for INBOX INSIGHTS sponsorship options.

Some events and partners have purchased sponsorships in this newsletter and as a result, Trust Insights receives financial compensation for promoting them. Read our full disclosures statement on our website .

Thanks for subscribing and supporting us. Let us know if you want to see something different or have any feedback for us!

Need help with your marketing AI and analytics?

Leave a Reply Cancel reply

Subscribe to our Weekly Newsletter

Pin It on Pinterest