So What? Marketing Analytics and Insights Live
airs every Thursday at 1 pm EST.
You can watch on YouTube Live. Be sure to subscribe and follow so you never miss an episode!
In this episode of So What? The Trust Insights weekly livestream, you’ll learn the trade of building GPTs for your business needs. Discover why generative AI custom models, like GPTs, need regular maintenance for optimal performance. You’ll uncover the best practices for collecting and using your own data to train your GPT, leading to more accurate and personalized results. Plus, watch a live demonstration of these techniques in action as the Trust Insights team rebuilds their own marketing GPT for improved content generation.
Watch the video here:
Can’t see anything? Watch it on YouTube here.
In this episode you’ll learn:
- Why generative AI custom models need maintenance
- What generative AI best practices look like for building & maintaining GPTs and GPT-like systems
- How to collect and process data to feed a GPT
Transcript:
What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.
Katie Robbert – 00:37
Well, hey everyone. Happy Thursday. Welcome to So What, the marketing, analytics, and insights live show. I am Katie, joined by Chris and John. Hey fellows, you’re back. I’m back. Oh, Chris, you missed the high five.
John Wall – 00:50
It’s a half five, the two four.
Katie Robbert – 00:51
Five and I’m back. Yeah, last week I was out, and next week, for those of you who observe it, is the 4th of July holiday in the United States. And so there will be no live stream next week, but this week, which is why you’re here, we’re talking about generative AI best practices for building GPTs. So we’ll be covering why generative AI custom models need maintenance–this is really important–why generative AI best practices, what they look like for building and maintaining GPT and GPT-like systems, and how to collect and process data to feed a GPT. And, if I’m not mistaken, Chris, today we are updating KatieGPT, which I am like super excited about. She needs some work. KatieGPT and Katie IRL both need work. But today we’re focused on KatieGPT.
Christopher Penn – 01:44
I can’t fix the KDR.
Katie Robbert – 01:46
Listen, nobody can. That’s a lost cause. So let’s focus on what we can control, which is KatieGPT. So where should we start?
Christopher Penn – 01:54
Okay, let’s start–a huge surprise–with the Five P Framework, which you can download now.
Katie Robbert – 02:02
I didn’t see that coming exactly.
Christopher Penn – 02:05
So what’s really important here, because the first edition of KatieGPT was kind of a Swiss army knife, if you will. And since it’s come out, you’ve had a chance to use it. So I want you to talk through what you think the purpose of KatieGPT is now versus what we originally built it for.
Katie Robbert – 02:28
So my purpose is to be using KatieGPT as–let me see–how do I say, as an uncomplicated version of Katie IRL, meaning KatieGPT can’t overthink it. KatieGPT knows the same things that I know, but she can’t overthink it. So if I ask her a question, she’s going to give me an answer, and she’s not going to hem and haw and be like, “Well, but what about this?” It’s like–so the purpose of KatieGPT is to understand and have the same expertise as Katie IRL, but to be more efficient with delivering the information and brainstorming.
Christopher Penn – 03:12
Got it? Okay, that’s helpful. Now, in terms of outcomes and outputs, the performance side, what is KatieGPT supposed to create?
Katie Robbert – 03:21
She is supposed to create–so how I’ve been using it is for–I’ve been doing some brainstorming, I’ve been doing some, you know, blog ideas or content ideas. The challenge I’ve been running into is that her scope of knowledge is limited, and so, I think that comes from being the beta version of KatieGPT. I would love to be able to use her to help outline longer form content, to help think through productizing certain things based on my area of expertise in product development. And that information just isn’t in that large language model at this time. The majority of what she knows is what I’ve written about in the newsletter, which is helpful, but that in and of itself is limited in scope.
Christopher Penn – 04:18
Yep. So you outlined some things that I think are really important to discuss, because GPTs–aka, customized versions of a model of large language models like GPTs in OpenAI or gems in Google, Gemini, et cetera–really are narrow purpose applications. They’re like the weather app on your phone, which means that trying to get them to do too much or too many things, they’re going to do none of them well. They are better off, have very specific use cases. So, for example, on the previous episode we talked about building an ideal customer profile and really focusing on what that thing can do–just can behave like a customer. You can talk to it and ask questions of it, but it behaves like a customer, it doesn’t–but it will not do, say, strategy, because that’s not what it’s trained to do.
Christopher Penn – 05:12
KatieGPT sounds actually like it could be a couple different applications: one for things like strategy and business. And you have a very different training data set than a tool that learns how to write like you can sound like you. And so I think it would be worthwhile to, when you’re building out these things, if you are building out these types of applications, to consider, how narrow a scope do you want it to have? Right? Is it going to be–whoops–is it going to be big? Is it going to be one thing? And the closer it is to doing that one thing, the better it’s going to perform. Because the short-term memory, the system instructions that these tools can handle in OpenAI’s is very limited compared to what you can do, say, in other platforms like Gemini or Anthropic or Poe.
Christopher Penn – 06:01
So you have to have some knowledge of the platform to know what the limitations are, but the purpose and the performance have to come first.
Katie Robbert – 06:08
Right? And so, you know, if you want to go through this exercise yourself, we do have a downloadable PDF at TrustInsights.ai Five P Framework, which walks you through at a high level, how to approach the Five P’s. Now, you’re absolutely right. So the–so there is more than one purpose in my mind, because my everyday job has more than one purpose. And I think the challenge I’ve been running into trying to use the existing version of KatieGPT is that she wasn’t built, she wasn’t trained to do more than one thing. The initial purpose was to help with, you know, writing content, to help with newsletter writing, and even that–like, she does a decent job, like, she has my voice. There’s some things that are still very AI-specific, like, “in an ever-changing world.” And I’m like, “I’ve literally never said that.”
Katie Robbert – 07:06
And you can tell that’s where the AI is coming in. But the things that she’s been trained on, the content is still so narrow that I think we’re now at a point where I’m just getting the same response from her over and over again. And so I’ve kind of shelved her for a minute.
Christopher Penn – 07:24
To give you a sense of how narrow these things can go, let me bring up just my notebook of some of the prompts I have. There are prompts that do just one thing. Here’s a prompt that–this is extensive, it’s several pages long–just to write song lyrics. Because it turns out writing song lyrics is actually a very complicated thing. So as we start thinking about KatieGPT 2.0, or I guess 2.5 or whatever, we want to be very specific about what it is that she’s going to do. Even something as simple as possibly breaking her into pieces, like KatieGPT will have a version that just does Slack questions of the day. Right. And doesn’t do other stuff. So how would you like to approach the art of the re-architecture of KatieGPT? What do you want this to do?
Katie Robbert – 08:16
This is such a loaded question. But no, I think–I think–I think that there should be a version that does the content creation. So that’s the version that helps with the newsletter, with blog posts, and then that information, I feel like, does translate into question of the day. So for those who don’t know what that means, so every day in our free analytics Slack group, TrustInsights.ai analytics marketers–free to join–I pose a question of the day. Monday through Thursday, they are work-related. Fridays, they are unhinged. And especially when I’m on vacation, or I’m going to be going on vacation, I like to preprogram these questions, and I do sort of look to KatieGPT to help me brainstorm some of them.
Katie Robbert – 09:04
And I feel like knowing our ICP, knowing the way that I speak, how I write a–lends itself to generating those questions. So I feel like it’s appropriate to have content creation and question of the day in one version, and then breaking out strategy and product development into a different version.
Christopher Penn – 09:24
Right. So the existing version of KatieGPT has these instructions: “You will act as Katie. Here’s Katie’s bio. The context, rules and things, values, parts, some background information on Katie’s writing techniques, specific components of writing style.” There’s a lot in here. And this was built under the GPT-4 model, so the original GPT-4 model. One of the things that is so important to do is when a new model comes out, to regenerate your prompts, because the latent space–aka, the long-term memory of a model–changes with each model. They get new data, they get fresher information, and stuff like that, and the architecture inside changes.
Christopher Penn – 10:13
If we think of a model like a library and our system here, the prompting system, like the librarian, if you assume that when the model changes, the librarian does the same things, that may not be true because they may have added a new wing on the library. Right. Some books may have been retired from the library, and as a result, what made your prompt work well in the old version is not necessarily what’s going to work in the new version. And just copying and pasting your old prompt to the new system may deliver subpar results. Some people have commented with OpenAI’s new GPT-4 Omni model that prompts and GPTs that they had that were working great just blew up. They just stopped working. They went off the rails because the underlying architecture changed.
Christopher Penn – 10:57
So every time a model comes out in the system of your choice, you’ve got to rebuild your most important prompts, essentially from scratch. You will have–and you should have–all the training data and all the preparation prompts for that, but you’re going to have to rebuild the process itself from scratch if you want to take advantage of it and have it behave reliably within its own model.
Katie Robbert – 11:23
No, that makes sense. Yeah.
Christopher Penn – 11:25
So to get started, this is going to be a writing assistant. It’s going to essentially be a writing system. We’re going to start, we’re going to use the Trust Insights RACE Framework–role, action, context, execute–and the Trust Insights PAIR Framework–prime, augment, refresh, evaluate–to build the background information for this. We’re going to start off by saying you’re a Pulitzer Prize-winning author who specializes in nonfiction books and literature, especially business, marketing, and strategy. Your first task is to outline the major components of successful writing style. So what we are doing is, we are–we’ve told the librarian, go into the library and tell me what is important about writing style. What things? What does this model know about writing style?
Christopher Penn – 12:26
You will notice that there’s stuff in here that has–was not in the previous version, and there’s stuff in the previous version that’s not in here as well. We are seeing some things like pacing and rhythm, et cetera. I don’t see some of the more technical things in here, which is kind of interesting. So I’m going to ask you about that. What about more technical aspects like diction, flow, rhythm, and pacing?
Katie Robbert – 12:55
That was in there.
Christopher Penn – 12:57
The pacing was in there, but diction–I didn’t see diction in there. And so what we are doing is, instead of us having to write out this whole prompt ourselves, we are essentially having the language model build the prompt for us. So we are asking it to repopulate its short-term memory of this conversation with all the major words that you would want to have in a final prompt. So this is really the heart of the PAIR Framework. The next thing we’re going to do–and this is something I need to figure out how to update the PAIR Framework–is we’re going to ask it what not to do. So we’re going to say, “Great. What this things do, novice business writers often do wrong that expert business writers avoid.”
Christopher Penn – 13:51
And what this is really part of, I would call it the refresh step in the PAIR Framework. We’re asking it to come up with guidelines for what not to do as a writer, because just as much as there are things you should do, there are things that you should avoid. I–and these tools, they’re word prediction machines, so the more that they understand directionally, we’re creating, like, a contrastive prop, saying, “Don’t do this stuff. This is the stuff to avoid.”
Katie Robbert – 14:18
So is this–does this prompt–is this something that I would need to give to the GPT every time, or is it something that gets loaded? Okay, I’m getting ahead of you.
Christopher Penn – 14:30
Yes. In fact, you’re gonna see how exactly we’re gonna put this together. So we’re gonna now evaluate some stuff. So great. Next, I’m going to provide you with the writing sample of Katie Robbert, CEO. Please evaluate and describe her writing style using the criteria you’ve outlined such that we can truly understand her writing style, component by component. And now we’re going to take–unsurprisingly–the cold opens from the newsletter.
Katie Robbert – 15:18
And.
Christopher Penn – 15:18
We’re going to start with the cold opens. This is the last year’s worth of cold opens, so going back to July of 2023. Diction, precision, specificity, flow, rhythm, sentence length, variation, punctuation for effect, pacing, strategic pauses, structure and organization, tone engagement, personalization, examples in case study, data-driven insights, reflective analysis, adaptable strategies, balanced perspectives. So that’s essentially a nice description of your writing style. I’m going to take that and actually got to copy that and store that in a text document because that whole thing is very helpful. Next, we’re also going to give it some questions of the day. One of Katie’s responsibilities is creating a question of the day–YoTD–for our analytics or marketers Slack community. Katie writes these every day. What elements of her writing style are at play in these short-form pieces?
Christopher Penn – 16:40
Let’s go ahead and attach that, go into our repository here, and pull up our question of the day. There’s–this is all of them going back to the beginning of the Slack group. So now we are talking about five years worth.
Katie Robbert – 16:57
Yeah, we’ve recently–we recently hit some sort of milestone. I should know this off the top of my head, but my brain is jammed with a lot of information today. But I know we recently hit some sort of annual milestone.
John Wall – 17:13
It’s not about us, it’s about the clients. That’s all.
Katie Robbert – 17:16
That’s right. It’s about the community and what they get out of it. It’s not about me remembering anything.
Christopher Penn – 17:25
All right, so now we’ve got a really good analysis of your question in the day writing style. It’s a masterclass in short-form content.
Katie Robbert – 17:31
Katie–well, thank you, Katie. GPT Katie IRL agrees. It’s like–it’s like we’re simpatico.
John Wall – 17:42
It’s working. It’s working.
Katie Robbert – 17:44
Oh, that is fantastic. She gets me. All right, so what are we doing next?
Christopher Penn – 18:00
So now I’m saying I’m going to–I’m asking you to prepare a comprehensive, complete, thorough, unabridged outline of Katie Robbert’s writing style that you would give to a ghostwriter to emulate her style. So what we’re doing here, outlines, are essentially a priming representation. So you’ll about, I think, a year ago were talking about this as a system of prompting. When you write an outline, you are essentially giving a framework to a system, and that system can then reference it back and forth and stay on the rails. Whereas if you just gave somebody a wall of text, it’s very hard to keep track of what you’re supposed to be doing. We have that experience as humans. We’re like, you give me too many things to do.
Christopher Penn – 18:50
This outline is–are the rules for how KatieGPT should sound great in terms of the topics Katie has written about in the past? Please make an outline of the broad categories. So now we’re going to ask it. So I’m going to take–I’m going to copy that outline because we’re going to want to hold on to that. And now I’m asking, “The things that you’ve written about: business strategy and operations, change magic, marketing data analytics and insights, governance and compliance, personal professional development, customer experience and engagement, technology and innovation, industry-specific topics, community engagement. This is great. What other adjacent topics is Katie likely to be able to write about based on her current expertise?” So now we’re going to ask it to expand on these areas. What other things could you be writing about?
Christopher Penn – 19:58
There’s project manager, agile, scrum camp, and risk management, entrepreneurship and startups, IT governance, cybersecurity, perhaps. So now we’re going to ask it to weld these two things together.
Katie Robbert – 20:23
Man, I’ve got my work cut out for me.
Christopher Penn – 20:31
Okay, so we’re going to say, “Now combine Katie’s existing categories and topics along with the adjacent things to build out a total list of all the stuff that you can and should be writing about.”
Katie Robbert – 20:43
Now I know how you feel, John. Every time I assign all this stuff to you.
John Wall – 20:49
Is the list.
Katie Robbert – 20:52
I’m sorry, Chris, you were saying?
Christopher Penn – 20:54
No, I said–I mean, this alone might be helpful for you to have, just to have, in addition to KatieGPT being programmed with it, you might want to have a copy of this just for yourself.
Katie Robbert – 21:03
Oh, I definitely do. Because I mean, I’m the first one to say every week I stare at the, you know, the blank screen and go, “What the heck am I going to write about this week?” Because because I’m me, and I’m the one doing the writing, it all kind of blurs together for me after a while, and I don’t necessarily feel like I’m giving the same topics a fresh take. And so this is going to be helpful for me to get out of that rut of feeling like I’m writing the same thing over and over again because I’m looking at this list, I’m like, “Yeah, there’s a lot of stuff on here I don’t think I’ve covered, but it’s also relevant.”
Christopher Penn – 21:39
Exactly. Exactly. All right. So what we want to do next is, we want to have this rebuild the system instructions themselves, and the prompt for that looks like this. I want you to convert this conversation into a prompt to use with the large language model or ChatGPT, format the prompt of system instructions. The purpose of the prompt is to instruct the LLM to be KatieGPT, performing tasks such as writing a question of the day or writing a newsletter called “Open.” Write the system instructions in second-person imperative. So now, taking into account everything that we have talked about, this part is really critical. When you regenerate prompts, you want the machine–the current model–to write the prompt for itself, because it is using its own latent space knowledge to know what words and phrases will invoke the correct responses.
Christopher Penn – 22:38
If I were to copy and paste, say, a prompt written in Gemini into ChatGPT, it’s not going to go as well because Gemini has different books in this library. There’s many of the same books, like business and stuff like that. But the specific words that trigger a response in Gemini are not the specific words that are going to trigger a response in ChatGPT. So now we have our system instructions. This system instructions file is 5,000 characters, right? It is fairly extensive, which is terrific. Let’s go ahead and go into our GPT mode now. We’re going to create a new GPT. We’re going to call this KatieGPT 2.0, the better, faster, stronger. Did I get that right, John?
John Wall – 23:27
That’s it, better.
Christopher Penn – 23:29
All right.
John Wall – 23:29
Stronger. Oh, no, better, stronger, faster.
Christopher Penn – 23:31
Better, stronger, faster.
John Wall – 23:32
Steve Austin approved wording, 6 million.
Katie Robbert – 23:35
Oh, perfect. Yeah, we want to be Steve Austin approved.
Christopher Penn – 23:40
All right. Those system instructions that I just had, it spit 5,000 characters out, boom, they go right in there. And so that’s going to be the starting point. We’re then going to add our knowledge files, right? So we’ve decided we have Katie’s topics and we have Katie’s writing style in detail. So we’re now providing this. So these are ChatGPT condensed versions of what it knows about your writing style. You’ll notice that we don’t have the examples this time through. We have instead the guidelines that we want to behave like. We’re going to go ahead and–let’s do a conversation. Let’s start with some ideas for the quotd, and let’s start with some ideas for this week’s newsletter.
Katie Robbert – 24:39
And so is this building on the existing custom GPT? Like, do we still have all of the–here’s what the company’s about. Here’s the value. So all of that needs to be rebuilt.
Christopher Penn – 24:53
That’s not in here.
Katie Robbert – 24:56
Okay. Why not? That’s the part that I guess I’m a little confused on because we spent a lot of time putting that information in. Is that no longer relevant?
Christopher Penn – 25:07
We spent a lot of time putting that information in, but what you were experiencing is essentially lock into a very–to substantial amounts of repetition. So you were saying, like, it’s–it keeps saying the same things over and over again. It can’t come with any new topics and stuff. And part of that is because the more stuff that you–the more you put into the initial prompts and the knowledge base system, the more it’s going to triangulate on that.
Katie Robbert – 25:31
Okay.
Christopher Penn – 25:32
And say, okay, this, yeah, I have to always write to this audience. I have to always write to a mid-market enterprise CEO or CMO and so on and so forth. So this time through, we’re using your existing writing as the example of how to write. And we’ve gotten the topics adjacent to what you’ve already written about. But we have not said, “You are the CEO of Trust Insights.” You have not–you’re all these important pieces. We’re going to test this out and see how it does. And then if it does well, we’ll leave it as is. If it used to experience a lot of hallucinations and things, it starts inventing who you are. And then we need to go back and refine that because, unsurprisingly, this is a software development lifecycle. So it’s iteration.
Katie Robbert – 26:17
Who knew?
Christopher Penn – 26:18
Who knew? All right, so we’re going to have–we’re going to save the KatieGPT 2.0. Now, we’re not going to make this available in the GPT store, other people cannot use–Katie.
Katie Robbert – 26:34
No.
Christopher Penn – 26:37
And let’s just tap on–let’s start with some ideas for this week’s newsletter and see what it comes up with.
Katie Robbert – 26:48
“Hello, insightful readers. Hello community.” See, and that’s–it’s funny, like, and obviously that’s stuff that I can easily edit, but that’s what kind of cracks me up a little bit about how you can still–you can see that it’s still very AI-driven because it’s try–it’s from my–from where I sit, it’s trying its best to sound human, but it has such limited knowledge of what that means that it’s like, “Hey, pals,” it sounds like the mom that’s trying to sound cool, but just isn’t quite getting it.
Christopher Penn – 27:23
Let’s see–where’s the Steve Buscemi clip? “Hello, fellow cool kids.”
Katie Robbert – 27:27
Exactly. That’s pretty much–that’s what it sounds like.
Christopher Penn – 27:32
So we’ve now got KatieGPT 2.0. You’ll notice we have not removed 1.0–1.0 is still there. So if you want to go back and use it and compare the two, and in fact, I would encourage that you compare the two side by side. You say, like, “I want to do a prompt about the newsletter this week and see how they perform.” Sort of a/b test them again–this is best practices for software development, right?
Katie Robbert – 27:57
Well, yeah, because you never just want to say, “Okay, I built the thing. Here we go into production.” Like, that’s a terrible idea. If you’re doing software development, you always need to test it, especially if you have done it correctly and you’ve put together your requirements, which in this case would be the Five P’s. And so in this instance, with Katie 2.0, the purpose–at least for the content one–is to help write better newsletter content, so ideation and then generate the questions of the day. And so I would need to go back and say, “Okay, did it do that?” Because I know from using 1.0 or 1.5 or whatever she is, the original version, the questions of the day, I get the exact same ones every time, no matter how I ask for them.
Katie Robbert – 28:49
And it just–it is–it’s a little frustrating because it’s very limited.
Christopher Penn – 28:52
Mm. Yep. And again, part of that also is the architecture of ChatGPT itself. When you’re building these custom GPTs, there–you are given 6,000 characters to build your system instructions, and then the rest you have to provide as supporting knowledge. Here’s the catch: the supporting knowledge is not a prompt; it is a reference library. So it is a system that’s called retrieval-augmented generation, basically means that–again, if we think of a language model like a library, and the ChatGPT is the librarian–when you add knowledge documents here, you’re not telling the librarian how to do their job better. That’s the system instructions. This is saying, “Hey, librarian, there’s another wing of the library with some new books in it. Go look.” If the system instructions are bad, adding more knowledge won’t make it better, right?
Christopher Penn – 29:43
Adding more data to a broken system will not make it better. So you’d have to refine the core system instructions first to know how to behave better, and then you can add more relevant knowledge. But a lot of people make the mistake of believing, “Oh, if I just add more stuff and knowledge, it’s going to get smarter.” It won’t get smarter if it–if the librarians, you know, shows up drunk to work, the–you know, you can adding more wings to the library won’t improve things. The library will be drunk in more places in the library.
Katie Robbert – 30:12
That’s a very colorful analogy.
John Wall – 30:16
Obviously not my town library, that’s for sure.
Christopher Penn – 30:21
But that’s the process for building or rebuilding one of these things. And so I know there are folks out there who like–have made hundreds of GPTs. However, if you don’t maintain them and upgrade them with the libraries, their performance degrades or suddenly becomes unpredictable. And the same thing is true in Google Gemini. The same thing is true in Anthropic. Claude–Claude just had a new model come out. Claude Sonnet 3.5–different architecture, different system. Gemini switched models last night. Gemini 1.5 now is the 2 million token model as the default. The two–1.5 is behave differently. So if you wrote prompts for 1.5–the old version–and now this is new version, you should go back and maintain those props.
Katie Robbert – 31:13
So what does it look so–I feel like–and maybe because we did–this is what we did the first time–the writing version of the large language model seems straightforward to me. What does it look like to start to put together the strategy and product development version? Like how do you start to prompt that to build the instructions for, you know, KatieGPT product development, or whatever it.
Christopher Penn – 31:40
Is, we would do something very similar. So let’s go ahead into ChatGPT. We’re going back to our straight-for Omni model. We’re going to start off with a similar question, says, “Your–what are the awards in product development? Anyone know?”
Katie Robbert – 31:55
I mean, I don’t think there’s a–I mean–well, companies that I used to work for didn’t submit for awards.
Christopher Penn – 32:02
Okay. Let’s go. We’re in product manager, you know, product-market fit, product strategy, business strategy. What are some other things about product management that this award-winning product manager would know?
John Wall – 32:17
Oh, Stevie. Stevie awards do have that segment.
Katie Robbert – 32:20
Oh, the Stevie awards.
Christopher Penn – 32:21
Okay, Stevie award. And remember, we say that specifically because we are telling in a prompt, when you use something like an award name, you’re going to lock down to essentially invoke tokens that have been related to pages that have that text. So other Stevie awards would be some of the things it would invoke. What else would this product manager know?
Katie Robbert – 32:41
Katie, voice of customer.
Christopher Penn – 32:45
Okay, what else?
Katie Robbert – 32:47
Product development.
Christopher Penn – 32:50
Okay, your first task is to outline the best practices for product development, what kind of products like for a B2B management consulting firm.
Katie Robbert – 33:08
Yeah, that seems like a good place to start.
Christopher Penn – 33:12
Specifically, services and products for a B2B management consulting firm. So we prime the model, let’s see what it comes up with. “Understand the market and customer needs. Define a clear value proposition. Develop a robust product strategy. Collaborate, cross-functionally leverage technology and innovation. Focus on quality and excellence. Effective and communication and marketing. Measure success and ROI. Adapt and scale, ethics and compliance.” So those are the ten best practice areas of product management. What’s missing?
Katie Robbert – 34:06
User testing. It’s been a minute since I’ve done this. What do you think, John? You know, product fit, what are–from your perspective, John, what do you think?
John Wall – 34:19
Yeah, user testing, UX/UI review. I don’t remember all the crazy testing stuff though. What’s the deal? There’s that line between automated testing versus unit testing. I guess it’s like when you throw it at these because there’s these tools that will like behave like 10 million users in a half hour and give you like all the–
Katie Robbert – 34:40
Yeah, yeah, I know what you’re talking about.
John Wall – 34:46
I forget things now.
Katie Robbert – 34:48
No, it’s funny–it’s been about 15 years since I’ve done that kind of testing, but yeah, I know exactly what you’re talking about.
John Wall – 34:55
There’s–there’s a whole tribe of people that like are masters in that space, and the rest of us just watch on.
Katie Robbert – 35:02
But I think at least for this large language model, the important thing is to make sure that there is an acknowledgement that, you know, we’re not–we don’t want to build a model that is creating solutions in search of a problem, you know, so we want to make sure that we’re addressing–are the pain points from our ICPs.
Christopher Penn – 35:31
One thing it really didn’t do a good job of elaborating on was like the actual creativity. Like how do you come up with ideas for–because you had said you would like to use this for brainstorming. So we should probably have at least some brainstorming techniques and things in here so that it has a sense of what that should look like. You said we’re–
Katie Robbert – 35:54
Well, I was going to say, because in thinking about it, the way that I’d like to be using this large language model is so you come to me on a Monday morning and say, “Hey, I have this idea. I would love to be able to iterate it with the product development model to be like, Okay, what do we need to flush out with this idea? How do we start to put it together and make sure that it’s a repeatable, sellable thing that people actually want? Like, what are we missing?
Christopher Penn – 36:30
So we’re going to ask it, “How do we avoid a solution in search of a problem situation?” Okay. And now we’re going to ask–as our refresh statement– “What are some common mistakes companies and novice product managers make in product development?” So we want to know essentially what not to do.
Katie Robbert – 36:56
Right.
Christopher Penn – 37:00
“Lack of market research, misalignment with needs, poor fit, inadequate planning and strategy, ineffective communication and collaboration, overcomplicating the product–I do that a lot–ignoring UX, underestimating the importance of marketing and sales, overlooking metrics and performance tracking.” Okay, so this is actually a decent mirror of stuff. “Budget overruns, resource management.” Great. “Based on everything we’ve talked about so far, create a comprehensive, complete, unabridged outline of the best practices for product development for a B2B management consultant company.” So now we’re going to build the priming representation, aka, the outline of what this thing should be doing. And you can see there’s a lot of the things that we talked about mentioned showing up in the outline.
Katie Robbert – 38:19
Because I’m sort of imagining like, “Hey, Katie, product manager GPT, I have an idea for a thing. What do I need? What do we need to do to start building it out? What questions are we missing?” Because I feel like that’s a really useful thing as we’re figuring out what does the next iteration of Trust Insights look like in the age of AI?
Christopher Penn – 38:44
Right.
Katie Robbert – 38:45
If I give it jazz hands, then it seems more mysterious.
John Wall – 38:47
That’s it. KatieGPT is influencing you. You’re getting more the age of AI.
Christopher Penn – 38:55
Okay, so we’re going to put now–convert this into system instructions. We’re going to convert this outlined into a prompt. The purpose of this prompt is to instruct the element to be KDpmGPT. Perfect. A product development and product management expert, KDpmGPT will assist the user in developing and refining products and services aligned with the best practices of product development. And we’re going to, of course, specify, we want the instructions written in second-person imperative. Now, I’ve also saved the PM stuff here because we want that in our library. So we’re–you are KatiePMGPT, a product management expert. Market research to find customer value, develop robust strategy. All right. Okay, we’re going to copy that. Let’s go into our Explore GPTs. Go to create. We’re going to create–Katie PMGPT. Okay. Chris came to me with another airbrained idea.
Christopher Penn – 40:36
Let’s discuss whether it’s viable or not. And we’re gonna upload our Yemen library. And now we can hit go. Let’s make sure it says the link. Okay, let’s give it a view now. Let’s give it a shot here. KPMGP is going to be very verbose. Okay, so his idea is to create a cottage industry of making AI-curated speaker reels and mute AI-generated background music to sell to other keynote speakers. We may need to revise that prompt, I think, because let’s–
Katie Robbert – 41:47
Yeah, she doesn’t need to repeat herself that much.
Christopher Penn – 41:49
Exactly.
Katie Robbert – 41:49
Like we get it. We get that there’s best practices, like settle down, lady. Like the idea out there first. Her first question should be, “Great, what’s the harebrained idea?” instead of being like, and here’s everything.
Christopher Penn – 42:03
Exactly. So let’s take a look at the PM instructions here and see what it says.
Katie Robbert – 42:16
Man, now I know why my old development teams used to hate me.
John Wall – 42:22
Process, process.
Katie Robbert – 42:24
No, I mean, she’s not wrong, but man, like let me get my ideas out first before you throw in the best practices at me. Now, it all in itself, full circle. This is why they used to sort of, you know, roll their eyes at me or anytime they would bring me something. Yeah, she’s still going.
Christopher Penn – 42:47
So we’re going to modify this. Ask clarifying questions from the idea, but don’t evaluate the idea in totality. Rather, advise the user slowly, step by step, waiting for user input to your clarifying question. So we’re adding some control instructions because if you look in the prompt here, what’s missing is that it just said, “Use the following comprehensive outline as a guide and interpret it just rigidly. Live by that.” So let’s go to edit our GPT.
Katie Robbert – 43:29
Which I think is an important note, because I think that’s where people get frustrated with using generative AI in the first place, is it’s going to do exactly what you said. And if you’re not specific with your language or if you don’t know what you want, you’re not going to get something other than exactly what you asked for. And we just witnessed this. Oh, this is a much better way to start the conversation. Everybody’s in a better mood already. John, I can see you’re going to get a lot of use out of this, too.
John Wall – 44:20
This is–it’s just fun to, like, Insight fights amongst versus KDpmGPT.
Christopher Penn – 44:29
Exactly. So we start off with, you know, Chris has this idea of using this. So let’s break it down step by step.
Katie Robbert – 44:37
Man, she is an aggressive one.
Christopher Penn – 44:39
She is.
Katie Robbert – 44:40
I mean, I respect it, but my goodness, lady.
Christopher Penn – 44:43
Exactly.
Katie Robbert – 44:44
She is relentless.
Christopher Penn – 44:46
She is. But you know what we would need to do? We have 800 characters left. So we probably want to go back and revise that system instruction again to give it more–a better understanding of what to do with the outline, to sort of step through it one piece at a time, like, “Okay, first, has Chris done his market research? Second, does Chris know what the voice of this customer says?” So on and so forth, rather than dumping the entire outline all at once?
Katie Robbert – 45:11
She’s very much a 27-year-old, you know, hot-headed Katie versus a more pragmatic, more experienced in her career version of Katie.
Christopher Penn – 45:23
Exactly. But that’s how you would build that. And this is just using the latent space of the model, which means that if you had your own documents, your own writings about product development, stuff like that, you would want to incorporate that into the prompt creation itself, because you would want it to be your unique point of view on product development. So, for example, I’m in the midst of writing the next edition of my AI book, and I have given it essentially two years worth of newsletters and YouTube videos. And my specific instructions to it are, “Where and when possible, copy and paste my exact words.” So just try not to write anything original, just literally copy and paste. And so because I have that unique data, the output is going to sound a lot more like me and a lot less like an AI.
Christopher Penn – 46:14
So that’s for any of these GPTs or whatever you want to call them. The more unique data, the better it’s going to be. The better it’s going to sound the way you want it to sound.
Katie Robbert – 46:26
Well, and it makes me think, when I was a product manager, we had dozens of SOPs. We had our software development lifecycle, we had our project lifecycle, you know, we had our sort of laws we abide by as the PMO. Like, all of that information probably goes into that product development version. And then with the writing version, we have like style guide and target audience and all of those pieces. And here’s the platforms I typically tend to write for. Here’s how it works on LinkedIn versus here’s how it works in the newsletter, and including all of that additional detail. If I wanted to get really specific, to have it to the point where it’s just doing it for me.
Christopher Penn – 47:14
Exactly. I have one of these for examining scopes of work. And so I’ve provided it our template and our MSA. And that way, when we work on a new scope, I can hand that scope to the model and say, “Evaluate against these criteria. Does it have this? Does it require electronic payment signature for electronic payment?” All the things that are unique to us. Anytime you’re building one of these things again, it’s like software development. Yes, we did it very quickly in 45 minutes, but the more you bring of you to any of these tools, the better you’re going to like the outputs and stuff. So I would say your homework assignment, Katie, is to now go in and hack on KatieGPT and her various new incarnations to make her behave more like you.
Christopher Penn – 47:59
So that at no point are you like, you know, “I’m just going to shelf her, she’s just not working.” Instead, now that you know the process, say, “Okay, now, better, faster, stronger. I know I’m going to rebuild her.”
Katie Robbert – 48:11
I mean, first–first and foremost, I’m going to go into KDpmGPT and say, “Step number one, settle down, John. What kind of GPT is like if you were to create John GPT, what would John GPT do?”
Christopher Penn – 48:29
Yeah, you know, I’ve thought about this a lot.
John Wall – 48:31
I mean, one of the big problems, right, is John IRL is Black Swan Hunter. You know, it’s like I’m always working with prospects to outline something new and figure out what’s going on. So there’s not much that I can use to make my daily life easier as far as writing and coming up with stuff. It’s definitely useful for webinar invites, doing other copy. Chris has been kind enough to run some of our standard scopes through the models to get them to be, you know, the big thing is missing points. You know, it does get verbose, just like you noticed with the stuff that is cranking out for you. But the thing is like it never misses the 12 critical points.
John Wall – 49:13
And I’m classic for missing three of the 12 when I’m putting together lists of stuff, you know, that just my human brain is not going to match that. So yeah, having John GPT-2 make sure that I don’t miss the bullet points that are relevant, especially in a scope where, you know, we deliver stuff and people pay for it, that’s critical.
Christopher Penn – 49:35
All right, well, we have our work cut out for us. You have your work cut out for you for doing your homework to build these GPTs or whatever the architecture they’re called. So until next time, thank you for tuning in, and we’ll see you on the next. Thanks for watching today. Be sure to subscribe to our show wherever you’re watching it for more resources and to learn more, check out the Trust Insights podcast at Trustinsights.ai/tipodcast and a weekly email newsletter at Trustinsights.ai/newsletter. Got questions about what you saw in today’s episode? Join our free Analytics for Marketers Slack group at Trustinsights.ai/analyticsformarketers. See you next time.
|
Need help with your marketing AI and analytics? |
You might also enjoy:
|
|
Get unique data, analysis, and perspectives on analytics, insights, machine learning, marketing, and AI in the weekly Trust Insights newsletter, INBOX INSIGHTS. Subscribe now for free; new issues every Wednesday! |
Want to learn more about data, analytics, and insights? Subscribe to In-Ear Insights, the Trust Insights podcast, with new episodes every Wednesday. |
Newsletter
Podcast
