So What? Generative AI Advanced Prompting Techniques - Trust Insights Marketing Analytics Consulting

So What? Marketing Analytics and Insights Live

airs every Thursday at 1 pm EST.

You can watch on YouTube Live. Be sure to subscribe and follow so you never miss an episode!

In this episode of So What? The Trust Insights weekly livestream, you’ll learn the difference between basic and advanced prompting techniques for generative AI. You’ll discover how to use the Trust Insights PARE framework to improve your prompts for better results. You’ll see advanced prompting in action with a live demo. Finally, you’ll learn how to create system instructions to turn an entire advanced prompting workflow into a reusable system prompt.

Watch the video here:

So What? Generative AI Advanced Prompting Techniques

Watch this video on YouTube

Can’t see anything? Watch it on YouTube here.

In this episode you’ll learn:

The difference between basic and advanced prompting
The Trust Insights PARE Framework
Advanced Prompting control structures

Transcript:

What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for listening to the episode.

Katie Robbert – 00:32
Well, hey, everyone. Happy Thursday. Welcome to so what, the marketing, analytics, and insights live show. I’m Katie, joined by Chris and John.

Christopher Penn – 00:39
Hello.

Katie Robbert – 00:45
Hi. This week we are talking about generative AI advanced prompting techniques: the difference between basic and advanced prompting, the Trust Insights PARE Framework and advanced prompting control structures. And I’m in the hot seat. And so I hope everybody has brought some knitting or some crosswords or something else to catch up on because I am a notoriously slow typer. Sorry, just had to plug something in. But I, I am in the hot seat this week because I, on the team, am the closest representation to the everyday marketer who is, who understands technology, maybe doesn’t use it day in and day out. I’m more of the casual user. And so Chris is going to put me through my paces and then buy me an ice cream cone as my reward.

Christopher Penn – 01:48
Sure.

Katie Robbert – 01:49
All right.

Christopher Penn – 01:49
I’m never signing up for that, but okay.

Katie Robbert – 01:52
No, I think it’s important because every week John and I, we sort of play the foil to your technical expertise, Chris. So you’re here, you’re typing, you’re doing these things, and John, I don’t about you, but I sit there and I’m like, how does he come up with that stuff so fast? And I know that a lot of it is preparation, a lot of it is practice. And so that’s sort of like the behind the scenes stuff so that the show moves along at a good pace. But there is a lot to be said about how quickly you can put things together, Chris. John, I don’t know. Is that sort of your take on it?

John Wall – 02:32
Well, no. It’s, in fact, I think you’re downplaying your role in this and that. It’s not that you’re a hobbyist at this stuff, you are using these tools as a professional. It’s just that Chris really, in spite of the marketing spin, he’s an IT professional. That is his first.

Katie Robbert – 02:47
Yeah.

John Wall – 02:47
He has his computer science major and, he has bang stuff out in Linux that we will never see or touch because there’s just no reason for us to ever go that deep down into what happens. So, yeah, it is. It’s always, as someone who’s not in it, to watch somebody in it flipping through, all their recorded macros and bopping around and keyboards to multiple machines and all these kinds of things, and we’re just like, whoa, wait a minute. Like, I’m in Adobe now. I’m doing this.

Christopher Penn – 03:17
I don’t know what you’re talking about.

John Wall – 03:20
Yeah, exactly. That’s, that’s how that goes. So, yeah, I, it’s all part of the mix, but yeah, pretty much most of our audience is on the same side that we are. And we’re all here to see Chris juggle the flaming stuff and make things happen, so.

Katie Robbert – 03:35
All right. Well buckle up and set the bar low. So, Chris, where are we starting?

Christopher Penn – 03:43
So let’s start with a prompt that you want to work on, a task that you want to work on. And for this, the first thing you want to do, huge surprise, is a user story. So, Katie, if you want to talk through a prompt and then a user story as to why we’re working on that prompt.

Katie Robbert – 04:01
Oh, well, I didn’t know that I had homework, so I did not come prepared with a prompt or a user story. So congratulations, I really am representing every marketer out there. I am making it up on the fly.

Christopher Penn – 04:14
All right, I will give you one then. Let’s come up with a, let’s come up with a topic for next week’s Trust Insights newsletter. So what’s your user story?

Katie Robbert – 04:26
All right, so as the CEO, I guess in this case it would be as the content creator, I want to write about a topic that will resonate with my ideal customer profile so that. So that this is what happened, so that the end-user can take an action, period.

Christopher Penn – 05:00
Okay, great story because it says, here’s who you are, here’s what you want to do. And then so that. So one of the things that I think is actually a good idea for folks to do is get in the habit of writing out these user stories because you can incorporate that into a prompt that you can tell the machine. Here’s what I’m trying to do. Help me do this thing. Second, you mentioned the ICP. For folks who are just tuning in, what is an ICP and what role does it play in this prompt?

Katie Robbert – 05:32
So an ICP is an ideal customer profile. And this is sort of your, this is the audience that you want to reach. Whether or not you’re reaching them is a different episode, and we actually covered that a couple of weeks ago on the live stream. So you can find that at trustinsights.ai YouTube. Go to the so what playlist and you’ll be able to see us cover audience analysis. But the ideal customer profile is who you want to be reaching with your content, with your services, with your ideas, with your thought leadership. And so for us, we use our ideal, I use our ideal customer profile to bounce ideas off of, for topics of content, of services, to say, “Would our ideal customer profile like this? Would they not like this? What would we change? How would it be different?”.

And so that’s why when I think about writing the newsletter, the topics, I always go through the process. Once I’ve written at least a draft, I say, “Would our ideal customer profile like this?”. And when I say that, I’m actually speaking to a custom language model that we created that has all of the data about our ideal customer profile. So it’s interactive. I’m chatting with it, saying, “I wrote this post for the newsletter, would our ideal customer profile like it or not? And if not, what do I need to add? What do I need to change?”. So.

Christopher Penn – 07:00
Got it.

Katie Robbert – 07:01
That’s what I mean by an ideal customer profile.

Christopher Penn – 07:04
No, that’s good. Now question, do you want me to type just so it goes a little bit faster and you tell me what to type?

Katie Robbert – 07:10
Let’s, let’s start with me typing because it’s, I feel like, again, it’s one of those, I mean, okay, so to John’s point, I’m downplaying, and I’m not that bad. I’m actually a better typer than I’m probably giving myself credit for. But I do use my thumbs. I’m self-taught, so I’ll admit that. Okay, so let’s get started. Yeah, let me bring up my screen. John, you might have to take over as hosting duties so that I’m not trying to host and do this at the same time. So that is my screen? Yes. Okay, that is your screen.

Christopher Penn – 07:45
All right, so we are using Google’s Gemini 1.5 Pro model within AI Studio. This, but we’re going to talk about today, works with any major model. So we’re talking Anthropic, Claude, Chat GPT, you name it. As long as it’s a big system, when Llama 400 B comes out, you’ll be able to use that. So the first thing we need to do is we need to follow the steps of the Trust Insights RACE Framework: role, action, context, execute. So this is why the user story is so helpful, because it informs a lot of this. You’re going to start by giving it by saying “you are a whatever”. Your first task is to “whatever”, and then some context and execution. So let’s go ahead and see what Ku is going to type in here for the basic RACE Framework prompt.

Katie Robbert – 08:32
Okay, so that is my screen. So you are the CEO who writes the weekly newsletter. You want to find your next newsletter topic that will resonate with your ideal customer profile. You want the topic to resonate so that your ICP will be able to take action. The actions could be for them to do something themselves or to reach out to Trust Insights for help. How’s that?

Christopher Penn – 09:25
Okay, that’s a good start. Now we need to attach the ideal customer profile. So you’re going to, in that little plus button right next to the run button at the bottom there, you’re going to select My Drive and then search for the Trust Insights ICP inside Google Drive.

Katie Robbert – 09:41
Could I attach it just as a file?

Christopher Penn – 09:44
You can, you can say “upload to drive” and then drag and drop the file right into the little upload window there.

Katie Robbert – 09:51
Oh, so it only comes from Drive. You can’t just detach it like from.

Christopher Penn – 09:54
Your, well, you can drag and drop it and then it will put it in your Google Drive.

Katie Robbert – 09:57
Oh, okay.

Christopher Penn – 09:59
It used to call upload a file. Now that’s just a pain.

Katie Robbert – 10:02
Well, for sake of just keeping it easy, I downloaded it. So here we go.

Christopher Penn – 10:06
All right.

Katie Robbert – 10:08
All right.

Christopher Penn – 10:09
Okay, so we have, so this is what your average user is going to do. They’re going to say, “Here’s my prompt. Let’s come up with some newsletter topics”.

Katie Robbert – 10:18
Okay.

Christopher Penn – 10:19
Now, based on what you know about, actually, go ahead and hit run. Let’s see what happens. So what it’s going to do is it’s going to ingest this and come up with some candidates of things that it could do. Right. Now, take a look at that, Katie. What do you see and what’s your reaction to it?

Katie Robbert – 10:42
I’m underwhelmed.

Christopher Penn – 10:44
Okay, why?

Katie Robbert – 10:46
Well, I guess first and foremost, they don’t feel like something that I personally necessarily write about: “building a social-first customer experience”, “how to engage and delight your audience”. I am 100% confident those are not words I have ever strung together. Now granted, I know that this model knows nothing about me personally. I didn’t say, “You are Katie Robbert. Here’s information about her”. And I’m calling myself a generic CEO, and it’s giving me back what it thinks will be helpful. So I’m underwhelmed because these are not necessarily things that I personally would write about, but they’re things that our ICP cares about.

Christopher Penn – 11:31
Exactly. So our next step then would be to look at something along the lines of using the PARE Framework. So the PARE Framework stands for prime, augment, refresh, evaluate. And this is both a set of steps you can take and a conceptual framework for how to think. So what I want you to do, Katie, is I want you to talk to the model. I want you to ask it what it knows about choosing newsletter topics, what its best practices are.

Katie Robbert – 12:02
Okay.

Christopher Penn – 12:08
And the reason that you need to do priming is because one of the things that’s funny about the way language models work is that they construct their outputs based on everything they’ve seen first. You can go hit run on that.

Katie Robbert – 12:24
Yeah, well, I’m still in the sort of like, the pro tip for me is like, hitting the enter key doesn’t just do anything. Well, and I suppose you can probably change the settings to have the keyboard shortcuts, but I just, I hit the enter key and I just sat there and I was like, “Nothing’s happening”.

Christopher Penn – 12:41
Yep. Oh, in the upper right-hand corner, change the model from flash to pro.

Katie Robbert – 12:48
Why?

Christopher Penn – 12:50
Flash is a cheaper, faster, dumber model.

Katie Robbert – 12:53
Got it. We want the better, smarter, slower.

Christopher Penn – 12:58
So scroll back up with just a little bit. See the little diamond icon in the upper right there where your prompt is? Yeah.

Katie Robbert – 13:09
Hit rerun?

Christopher Penn – 13:10
Yeah, we’re going to rerun this now. So just tap that and it’s going to rerun it now with the Pro model. So this is, it’s giving some topic best practices. And this is not bad. We’re preloading the conversation with a lot of information that you would want to know generally about writing a newsletter topic. So next we’re going to move on to the augment phase and we’re going to ask, you’re going to ask the model, “What common mistakes do less experienced marketers make when choosing newsletter topics?”. And what’s going on here is a type of prompting called contrastive prompting. So contrastive prompting is essentially developing the model. And this works really well with all these big models of understanding what not to do, like, what are the things that are bad for a marketer to do. So the “me” trap, lack of audience segmentation.

While it’s doing that, Katie, I want you to navigate to Trust Insights’ website and go to the team page. And what we’re going to be doing here is you’re going to go to the team page and you’re going to copy and paste your team bio because we want to tell the model who is doing the writing. Because we started, one of the things Katie points out rightly is we started with a just a “you are a CEO”. It’s not, it’s very generic. So the next thing you’re going to do is you’re going to say, “Here’s more information about who the CEO is that’s doing the writing”. And then you’ll hit enter a couple of times to move the line down and paste in your bio.

Katie Robbert – 15:08
And then run?

Christopher Penn – 15:09
And hit run for that. So now we are in sort of the refresh phase of the PARE Framework. We’re trying to get it to, to remember this. And the model is going to try and be helpful. Like, “Oh, dude, slow down, we’re not…”.

Katie Robbert – 15:21
Well, and that’s, that’s the thing that’s always sort of frustrated me. Is it, I feel like generative AI, in an effort to be helpful is like, “I’m going to give you a bunch of stuff that you didn’t ask for, but, it’s helpful!”. And you’re like, “Oh my God, slow your role. I did not ask for this. I will let you know when I need something”. Like, are those the kinds of instructions that I could give up front in a conversation? Like, “Don’t just come at me with information. Let me be the one to ask for it”. You absolutely not in those words.

Christopher Penn – 15:54
No, absolutely can. In fact, one of the statements I will use in prompts is say, “Acknowledge that you’ve received the information by replying ‘okay’ and nothing else”.

Katie Robbert – 16:05
I think that. What about you, John? Are you someone who you’re like, “Oh, it’s great, they already gave me the information,” or do you want to be more in control of the conversation?

John Wall – 16:16
I tend to, like, the whole spray and pray thing of just have a bunch of stuff and weed through and try and pick stuff out, but, yeah, I don’t know. It depends on the day and what I’m looking for.

Katie Robbert – 16:29
Unsurprisingly, I lean more towards having control over the conversation and not being giving things that I did not ask for. But here we are. But this is a learning moment. That’s a pro tip. You can tell the machine what you don’t want and how you want the information. So that’s something that I just learned. It sort of, it feels like it should have been common sense, but I feel like as you’re learning these systems, common sense goes out the window.

Christopher Penn – 16:58
Well, one of the things we say in a lot of our webinars and talks and things is that these things are basically the world’s smartest interns, and they’re still interns. So the intern is so eager to make a good impression on the first day of the job. You’re like, “Dude, chill”.

Katie Robbert – 17:13
Yeah, I haven’t even had my coffee yet.

Christopher Penn – 17:16
Exactly. Okay. So now we’re going to have you prompted for one more thing. So we’ve done basic priming, we’ve done contrastive prompting, and now we’re going to do a little bit of self-consistency priming. And what we’re going to do is say, “What, what tricks of the trade do you know as an expert email marketer that we haven’t discussed yet?”.

Katie Robbert – 17:43
So I had been using the word “content creator”. So does it matter if we switch context?

Christopher Penn – 17:50
It does matter in this case. If you want to use content creator, that’s fine. But since we’re talking about newsletter topics, I was assuming we’re working with email marketing.

John Wall – 18:10
This is where I need to, like, tap dance.

Christopher Penn – 18:16
With this type of prompting, make sure you specify “and that we haven’t talked about yet” because what we want to do is we are trying to force the model to examine all the tokens, all the words we’ve discussed so far and say, “What else you got?”. There’s stuff that has not come up in the conversation because we don’t want to repeat itself. We want to dig a little bit deeper.

Katie Robbert – 18:38
And I think, again, that’s another really good pro tip because I find a lot of times the conversations with the models get really repetitive to the point where I just kind of give up and walk away because I get frustrated.

Christopher Penn – 18:53
Okay, so this did not do what we wanted to do. So scroll back up to your prompt and it should be marked with a little user chicklet.

Katie Robbert – 19:01
This is a chicklet.

Christopher Penn – 19:03
Yeah, that little thing there. Click on the pencil button on the right-hand side.

Katie Robbert – 19:06
All right, so let me ask, so I wrote, “What are some of your, what are some of the best practices and pro tips you know about as an email marketer that are not commonly known by everyone and that we haven’t talked about yet?”. Where did I go wrong?

Christopher Penn – 19:22
It doesn’t know that we’re talking about newsletter topics. It forgot. So in that prompt, say, “What are some of the best practices and pro tips you know about choosing newsletter topics as an email marketer?”. Basically how to give, like, intern focus.

Katie Robbert – 19:40
Right? Okay.

John Wall – 19:43
Now I have to say, too, like, one of the great things of it is just the, the UI on this thing is always in flux. So knowing that the chickpea is there.

Christopher Penn – 19:51
That’s, and it changes like every week. Like, this, the UI’s moved, something’s different. And now hit that rerun, little spark on where on the model line. So if you scroll down a little bit. Yeah, there’s that blue rerun diamond.

Katie Robbert – 20:06
Yeah, these are, again, sort of, because I’m the casual user. I’m looking at this, I’m like, “I didn’t know that this was the functionality that was available”. So even, and this is why I wanted to be the one doing the typing, because I know for a lot of people, you don’t absorb the information as well if you’re just watching somebody else do it. But if you’re the one doing it, your brain is committing it to memory more often. And so now I’m like, “Oh, okay, this is how I rerun, this is how I edit, this is how I do this”. Like, that’s more helpful to me.

Christopher Penn – 20:42
Okay, so now we’ve got some things like, “Most marketers think: what topic do I want to write about?”. “Pro marketers think: what do I want my readers to do after reading this?”. And so on and so forth. So these are, this is actually, is pretty decent. I wouldn’t call it expert, but it’s definitely not your basic, one, one anymore.

Katie Robbert – 21:03
Okay.

Christopher Penn – 21:04
All right. Now we’re going to kick it up a notch and we’re going to say, and we have to, the specific term we want to use here is scoring rubric. You want to tell the model, “Build a scoring rubric to judge the effectiveness of newsletter topics based on everything we’ve talked about so far”. And the reason this matters. The word scoring rubric is one of those weird little trigger phrases that models have seen so much information about what that specific thing is, that it works incredibly well. It works better than pretty much any other term for this particular task of having it build a scorecard so that it can rate and review any kind of output. In the Trust Insights newsletter this week, we did this example of this with haiku and saying, “Here’s how to build a scoring rubric for judging haiku”.

And it comes up with this whole scoring methodology.

Katie Robbert – 22:11
So if I say, “I’d like you to build a scoring rubric to categorize all of the newsletter topics we’ve talked about so far”, is that what we’re trying to do?

Christopher Penn – 22:20
No, we’re saying, “Like you to build a scoring rubric to score and evaluate newsletter topics based on all the, on everything we’ve talked about so far”.

John Wall – 22:36
And does that really build an array about the data? I mean, is that kind of, well.

Christopher Penn – 22:42
Katie’s going to hit the run button. We’re going to find out.

John Wall – 22:43
We will find out.

Christopher Penn – 22:51
All right, “solid topic”, “audience resonance”, “pain point alignment”, “no clear connection to the ICP pain points”, “goal alignment, zero to 10”, “novelty and interest”. See this? So you can see, John, this is really coming up with an actual literal, “Here’s how I’m going to assign scores”.

Katie Robbert – 23:07
Which is really useful for any kind of content because we’re talking this morning offline about, “How can I use, if I have an ideal customer profile that I’ve just put together, what are some of the ways that I can use it?”. And I didn’t even think to use the ideal customer profile as the foundation for a scoring system for my new and existing content, for my emails, for my ads, for, like, whatever it is, for my services. Exactly. We’ll be saving this bad boy.

Christopher Penn – 23:43
Speaking of which, in the upper right-hand corner, there is a save button you’ll probably want. You should hit that now and name it something. If, for folks who use Google AI Studio, this will save this within your Google Drive. So we’ll have to save the session and any documents that you’ve uploaded will also get saved in Google Drive as well. Chat GPT obviously has its own storage mechanism. Anthropic Cloud has its own storage mechanism, but you always want to save valuable stuff. One thing that’s different about Google’s environment that is not true about Chat GPT is that if you close the window, Google just forgets the whole thing. It doesn’t store it as it goes, whereas Chat GPT stores it as it goes. All right, now we’ve got this great scoring rubric. Let’s put it to use. And this is going to be.

I’m going to talk through this first. We’re going to tell the model to run a program. We’re going to build this program. We’re going to say the first thing you’re going to do is you’re going to generate five newsletter topic candidates based on everything we talked about. The second thing you’re going to do is you are going to score all five candidates based on the scoring rubric. Yep.

Katie Robbert – 25:03
See, you guys got to learn how to fill in the blank space while I’m typing, like we do for press.

Christopher Penn – 25:11
One weird little thing: when you’re writing out sequential instructions, put each instruction on a new line within the prompt because, again, for whatever reason, it seems to recognize that you’re trying to do it. Basically build a piece of software.

Katie Robbert – 25:26
Got it.

Christopher Penn – 25:26
The third thing you’re going to do is select the top two scoring newsletter topics based on your evaluation. Then the fourth thing you’re going to is come up with two refinements for each of the winners. We are writing software now. We are programming. We’re just programming in English. The fifth thing you’re going to do is score the refinements against the originals.

Katie Robbert – 26:15
The amount of concentration it’s taking for me to type. This is, I’m going to need a nap after this.

Christopher Penn – 26:21
The sixth thing you’re going to do is select a winner and present it. So we have written a piece of software. It’s just that we’ve written it in English. And this is one of the things that people just don’t realize about these language models is that they are programs, they’re programming environments. It’s just you’re not programming in C or Java or Python, you’re programming in Danish or Ukrainian or English. So Katie, are you ready to run your software?

Katie Robbert – 26:56
I guess as ready as I’ll ever be. I will say, too, because I was thinking about, as we’re doing this, I have saved a file from a live stream we did a couple of weeks ago, where we generated, it’s called the “comprehensive list of content Katie Robbert can and should write about”. So I would be interested, because it’s about 150 lines, I would be interested to put that file through this scoring rubric after we’re done with this.

Christopher Penn – 27:29
Okay, let’s see. Let’s run the program. Here’s the output. Five potential newsletter topics. So far, so good. Scoring each topic using the rubric. Now it’s going to assign scores, which is good. Two topics, those are the winning from the first round. Now we have some refinements and it tells you a little bit about what each refinement is. Now it’s going to run the scores for the second round. “The winning topic: ‘Data Governance in the Age of AI: Your Guide to Avoiding Costly Mistakes’. Focusing on mistake avoidance creates urgency. Strong positioning implicitly positions Trust Insights as the guide to navigating these complex issues. Maintains your key strengths, still aligns well with Katie’s expertise in the ICP’s interests”.

So this software, instead of having to do this one at a time, you now wrote the program, and it ran the whole thing as we just watched it live. And now you’ve got your winner.

Katie Robbert – 28:23
And then I aggressively hit the save button.

Christopher Penn – 28:27
Exactly.

Katie Robbert – 28:28
Does it auto-save once you set it up to save?

Christopher Penn – 28:32
Nope.

John Wall – 28:33
How about the nav on the left? That will persist when you log in again as yourself, or no?

Christopher Penn – 28:40
Yes, that navigation is pretty fixed and stuff, and there’s a whole bunch. You can see all your existing projects and things.

Katie Robbert – 28:49
Yeah.

Christopher Penn – 28:49
So we’ve written software. You’ve written software. And this is the difference between basic prompting and advanced prompting, is you’re now starting to use things like control structures and sequential things. You could, for example, write a version of this prompt that would say, “Keep generating refinements and scoring them until a refinement scores above 85”, for example. And it would, it could take a long time, but you could have it right, a loop, like real software does.

Katie Robbert – 29:24
Well, I mean, this is it. I’m updating my resume. I’m out of here, guys. I’m going to go find a new job now. I got a whole brand new shiny skillset.

Christopher Penn – 29:35
You do.

Katie Robbert – 29:36
But it’s interesting because I didn’t, I know general, I guess, what you call basic prompt engineering. And it’s interesting to think that you can take it this far just because you’re trying to figure out a newsletter topic. And so it’s sort of that step beyond brainstorm because I would imagine this is now a custom model, or for lack of a better term, that is reusable. So I don’t have to, every week, go into one of the language models and go, “Oh, what am I going to write about this week?”.

Christopher Penn – 30:17
Sort of, yeah, jumping ahead.

Katie Robbert – 30:21
They always do that to you. That’s what happens when you put me in the hot seat.

Christopher Penn – 30:25
Exactly. So you raise a really good point. You want to have something canned that would follow this process really well. And so that’s what we can do next. We can say to it, “Okay, your next task is to convert this entire process that we’ve gone through of using the scoring rubric to select newsletter topics into system instructions for a large language model”. And what this is going to do, there’s a few steps to this one as well, but what this is going to do is it’s going to essentially have the model write its own prompt, which is one of the most powerful things you can do because it knows what triggers itself.

Katie Robbert – 31:16
All right, so your next task is: convert this entire process of creating a scoring rubric into system and selecting a.

Christopher Penn – 31:23
Newsletter topic into system instructions for a large language model.

Katie Robbert – 31:31
System instructions, or this is where my typing starts to break down.

Christopher Penn – 31:41
And now we go to. Oh, yeah, now we go to a new line. “The purpose of this system prompt is to generate ideas for the user for the newsletter topic based on the user providing an ICP”. And we’re going to do another line here. “Be sure to include the scoring rubric and all of the best practices of newsletter topics in the system instructions”. Because one of the things that every AI will do is try to be as least verbose as possible because every token costs money. On the next line, we’re going to say, “Create the system instructions to be complete and comprehensive”. And then on the following line, “Write the system instructions in second person imperative”. So this is going to change the language to have it right, to basically do the RACE Framework implicitly. So now this should do the creation process.

So go ahead and hit run and let’s see what it comes up with.

Katie Robbert – 33:25
So it’s interesting, you didn’t put me in the hot seat to write the software. You’re putting me in the hot seat to make sure I know how to spell these words, which, is still a hot seat. So it’s interesting because, again, this is not the approach that I would have taken to putting together a set of newsletter topics to write about. I probably would have started with our ICP. So that’s definitely where I would have started. And then, I probably would have said, “And these are the general things that I write about. Help me match them together”, and then called it a day. Like I probably, as a casual user, I would have stopped there and thought, “That’s good enough”.

Christopher Penn – 34:17
Yeah. And we, good enough sometimes really is just good enough. Sometimes, though, you want to kick it up a notch. So that’s what we’re doing today. So if you scroll back up a little bit, let’s see what it’s put together. So this is a really big old outline. You had system instructions. You were a helpful AI system. Here’s how to execute this task, et cetera. It will ask you for additional information. There’s the generate the original ideas in line three. In section three, it’s actually asking, “Hey, go and copy and paste the rubric from earlier in here” so that it’s available. So you would need to do that outside of Gemini so that it all gets sewn together, and then you can keep scrolling down. Now here’s what you would do with this.

You would, in the, see the three vertical dots on the bar there? You’re going to select “copy markdown” and you want it in markdown format because it turns out that AI models like the ones at Power Chat, GPT, and Gemini understand markdown, which is a formatting language. You would paste that into a text file, and then if you were to, you hit save, right?

Katie Robbert – 35:27
Yep, give me one second. All right.

Christopher Penn – 35:32
Okay, so just for, yeah, for what to do with this, you’re going to go to the upper left-hand corner, click “create new prompt”, and it’s going to start a new session. And see the dropdown there where it says “system instructions”? You would paste, once you’ve done all the editing, you’d paste that in there, that whole big old thing, the markdown that now is. And you can see the model now has 816 tokens already in it that has basically gotten this thing ready to use. So you could use this week after week, just start a new chat with the system instructions in there, and you’re off and running. And you could provide additional commentary and things like that as well.

Katie Robbert – 36:15
And so let’s say I want to do this, and where. Oh, here we go. That’s not what I wanted. I would then also need to find this scoring rubric. Like, where is the actual. So it said in the instructions, where was it? “Insert newsletter topic scoring rubric from previous…”.

Christopher Penn – 36:46
“Response”. Like, so squat a conversation where we first asked it for a scoring rubric, and made a long list of stuff. So you have to keep going up further.

Katie Robbert – 36:55
Okay.

Christopher Penn – 36:56
Keep going, keep going.

Katie Robbert – 36:58
Oh, I thought that was it.

Christopher Penn – 37:00
So there’s the end of the rubric right there. So scroll up a little further.

Katie Robbert – 37:06
Oh, I see this here.

Christopher Penn – 37:08
Yeah, so this you will also copy in markdown format into that section of the prompt. That’s why I said you have to edit that outside of Gemini.

Katie Robbert – 37:19
Got it. All right. So I’m just, I just want to make sure I don’t lose any of this stuff, so I’m adding this stuff into my notes.

Christopher Penn – 37:30
Yeah.

Katie Robbert – 37:31
“Insert newsletter response here”. There we go. So let’s see what happens.

Christopher Penn – 37:41
I lost it, the first tab.

Katie Robbert – 37:45
Thank you.

Christopher Penn – 37:48
So you’d probably just copy and paste the whole thing, the whole new thing in. Right? There you go.

Katie Robbert – 37:54
Okay.

John Wall – 37:55
Okay.

Christopher Penn – 37:55
Now you can close up the system instructions.

Katie Robbert – 37:57
All right.

Christopher Penn – 37:58
And attach your ICP, and say “let’s get going”.

Katie Robbert – 38:04
Great. Upload file. See, this is why it’s definitely different watching me do it than watching Chris, because I am not speedy. So what do I say? “Let’s go”, or?

Christopher Penn – 38:20
Yeah, “let’s go”.

Katie Robbert – 38:23
“Let’s go”, pal.

John Wall – 38:29
Oh, hopefully it doesn’t break it.

Christopher Penn – 38:33
Oh dear.

Katie Robbert – 38:37
This is why I’m not allowed to touch things. Like I’m not allowed to touch the sharp objects at home, and I’m not allowed to touch the tech at work.

John Wall – 38:43
It’s like Star Trek. You need your thing to say to make it run. Like, everybody has a different tagline.

Katie Robbert – 38:50
Well, yeah, it’s doing something, so that’s probably good.

Christopher Penn – 38:56
So now it’s going through the, clear, the prompt, the system instructions probably need a little bit of tuning because it came up with seven candidates. And it’s going to now go through and score these. So I think we need to do a bit of instruction tuning there.

Katie Robbert – 39:10
Okay.

Christopher Penn – 39:11
Yeah, it looks like it has stopped. So we will need to, after the show, tune those system instructions to be more clear. But the general concept is sound. Now if you were using Chat GPT, you would take those system instructions and paste them into a custom GPT, and then you would have that custom GPT be available to use where you wanted. And with Anthropic, you would build Anthropic artifact, and it would do the exact same thing. So all the major systems have this kind of functionality that is 100% what…

Katie Robbert – 39:45
I plan on doing.

Christopher Penn – 39:48
And this is really essentially coding. You built an app. You just happen to build an app in plain language instead of a computer programming language.

John Wall – 40:02
I didn’t get the breakpoints. There was that, where in the original prompt, it was asking for more specific data, or why does it pause before it runs the whole thing in the shot?

Christopher Penn – 40:11
It paused because the system instructions are unclear.

John Wall – 40:14
Okay, so it’s just saying there’s something I can’t…

Christopher Penn – 40:18
Yep.

Katie Robbert – 40:20
Yeah, I’m totally putting these in Slack for you to take a look at.

Christopher Penn – 40:27
So, today we’ve talked through essentially a few shot prompting. We’ve done chain of thought, which is making a list. We’ve done a self-consistency check. We have done, somewhat, automatic prompt engineering. We’ve done some automatic reasoning. We have done optimization by prompting. We’ve done rephrase and respond. And so if you were to look at some of the academic surveys out there, systematic studies of all the different prompting types, today, we’ve covered six of the major types of prompting out of about 30, that are known to exist. And those, these prompting techniques all have specific situations where it’s a good idea to use some and not others. So, chain of verification, for example, there’s a certain context in which you want to use that. Today wasn’t it?

Part of advanced use of generative AI is a, knowing all the structures, and b, knowing when to use which ones.

Katie Robbert – 41:29
And I definitely just did what I was told. So if I was given a pop quiz.

John Wall – 41:37
The pop quiz face comes out.

Katie Robbert – 41:39
Yeah. No, but I mean, it definitely gets me thinking about the way that I’m approaching and, priming the models ahead of just saying, “What should I write about this week? I write for B2B marketers”. Like, that’s where I think a lot of us run into the, “Oh, well, it’s not helpful, it just gives me generic information”, it’s because we’re not going through all of these steps. We don’t have to build a custom model every time, but we should be taking it a step farther than just the basic, “You are a digital marketer. You want to know how to increase your ROI. What should you write about?”. That’s very basic, but that’s where we all started.

And for some of us, that’s where we’re still at, is making sure we’re remembering to tell the model who the user is, what your intention is, and not just say, “What should I write about this week? Nothing. Okay, great. AI is not useful”.

Christopher Penn – 42:49
I mean, you’re absolutely right, that is the way most people use it. And honestly, that’s the way a lot of people who position themselves as AI gurus, like, “Hey, here’s my top 50 Chat GPT prompts”. They’re all the same canned bullshit that’s a paragraph long. They’re not pieces of software. They’re not detailed, thorough prompts that behave like software, that collect information from the user, that ask good questions back, and that deliver very high-quality results, including forcing these models to evaluate their own work. Part of the PARE prompt is that evaluation stage, to say, “Check your work. Did you do a good job? Did, you know, you were given the conditions for what a good job is based on the scoring rubric? Did you do a good job?”.

When I have these tools write things like poetry and song lyrics, that is probably the most important step, is to say, “You know what good songwriting is. Did you do a good job?”. And almost always goes, “Haha, no”.

Katie Robbert – 43:56
That’s tough, though, because that, to me, is subjective. And that’s just, the same is true of content, that good job is such a subjective thing that I think that’s where it really is helpful to start with, “Who is your audience? Who is your ICP?”. Because, and we, we ran into this earlier this week and, the joke is that I’m an n of one. You presented something that you had put together and it wasn’t to my liking. And so that makes it very subjective, but that doesn’t mean that it wasn’t good because there were other people who were like, “This is great, I love this, this is fantastic. Let’s see more of it”. It wasn’t my jam.

So, making sure you’re clear up front of who’s your ideal customer profile, who’s your audience, is always going to be the key when you go through the PARE Framework or any kind of, system instructions. If you’re saying, “Is this good?” based on what?

Christopher Penn – 45:01
Exactly. That’s why having scoring rubrics is so important. That’s why having an ICP is so important, because if you don’t have those pieces, then, yeah, the model is going to default to a generic “Here’s what I think is good” as opposed to saying “These are the rules”. So even something as song lyrics, you might say, “You are required to have nine syllables per line”. Like, that is a requirement. And then it can go, “Oh, hey, this line has 11 syllables, it didn’t, I didn’t do it, I didn’t do a good job”. Right. So the, the rhyme structure has to be AABb, not ABAB. “Oh, I didn’t do a good job”. It’s not, like, that. So that specificity matters so much.

Katie Robbert – 45:37
And little plug, if you want help defining your ideal customer profile or how to write good song lyrics, you can contact our Head Lyricist, John Wall, who is also our Chief Statistician. And he’s going to combine data with music, and it’s going to get weird.

John Wall – 45:56
This is it. I’m opening for Taylor Swift next week, so we got to have something done.

Katie Robbert – 46:02
I can’t wait.

Christopher Penn – 46:03
I can’t wait. Oh, goodness. So, Katie, now that you’ve done, now that you’ve been on the hot seat, how do you feel?

Katie Robbert – 46:14
I feel like I definitely have to go back and review what the steps were, what the system instructions that we created actually came out to be so that I can really wrap my head around, like, what did I just do for the past 45 minutes, other than write down what was dictated to me. Because I know there’s a part of my brain that actually absorbed the information. And this is now the second time that I’ve done this particular technique. I did this late last week with a different large language model, and now for this. And it’s starting to make more sense to me.

And so for me, it’s just a matter of practicing, it’s a matter of doing it a few more times so that I don’t feel like I have to keep going back to my house and be like, “What was the next thing I was supposed to do?”. Because it should start to come in logical order in my brain.

Christopher Penn – 47:09
Exactly, exactly. All right, so that’s going to do it for this week’s show and stuff. If you, we have a bit of an outline for the way we approach today’s show. If you would like a copy of that very brief outline, we’re going to post it in our Analytics for Marketers Slack group. So if you are not a member there, you should go and join. It is free. There’s no, there’s no cost, other than, just showing up and adhering to the rules. So with that, we will close out this episode and we will see you all next time. Thanks for watching today. Be sure to subscribe to our show wherever you’re watching it. For more resources and to learn more, check out the Trust Insights podcast at TrustInsights.ai/tipodcast and a weekly email newsletter at TrustInsights.ai/newsletter.

Christopher Penn – 47:59
Got questions about what you saw in today’s episode? Join our free Analytics for Marketers Slack Group at TrustInsights.ai/analyticsformarketers. See you next time!

Need help with your marketing AI and analytics?

Leave a Reply Cancel reply

Subscribe to our Weekly Newsletter

Pin It on Pinterest