Future of Machine Learning and AI : General Assembly Panel - Trust Insights Marketing Analytics Consulting

In this expert panel discussion hosted by General Assembly, cofounder and CEO Katie Robbert joins other AI experts Matt Ritter of AthenaHealth and Sara Saperstein for an evening discussion about the future of machine learning and AI. They discuss the uses of AI today in various industries, challenges of current methods and processes, data governance, compliance, what to look for in hiring AI professionals and data scientists, and much more. Enjoy this wide-ranging discussion and learn what’s happening now in artificial intelligence.

Future of Machine Learning and AI General Assembly Panel

Watch this video on YouTube

Can’t see anything? Watch it on YouTube here.

Machine-Generated Transcript

What follows is an AI-generated transcript. The transcript may contain errors and is not a substitute for watching the video.

Greg Ly
Tonight, you know,

how many people have been do ga before?

Welcome back as well.

For those of you who are new to ga we are a global educational organization that specializes in training professionals and today’s both in finance high tech skills. During the day, we have students who are in immersive programs and classes such as UX design, data science, and other topics as well. And we have part time courses in courses such as digital marketing, product management, and data analytics. So some of those are actually going on right now. In addition to that, we have three events such as this and other classes and workshops. We were founded in 2011 as a co working space and has since grown internationally and nationally to many major US cities, and internationally and places such as London, Singapore and Sydney. So we’re growing rapidly. So if anyone has any questions about any of the offerings, we have here in class in

I’m happy to stay after for a little bit chat, you can chat with our current mindset as

Unknown
well.

Greg Ly
So we’re with that rule, he

Unknown
has this, you know.

Greg Ly
So just start can with someone in the audience, be willing to share kind of why they’re here tonight. What are they expecting? take away from this panel discussion? And what you’re up on that?

Unknown
Yes,

Christopher Penn
I just to hear people’s perspectives on things like bias in machine learning algorithms.

Unknown
I’ve been in marketing.

Unknown
As a student, I can

Unknown
remember events for

Unknown
years for actually, we’re not going to do this basic stuff forever.

Greg Ly
On that note, can you raise your hands you work with data analytics machine learning in your jobs currently?

Unknown
Well, with

Greg Ly
that, let’s just get started. I’ll let our panelists

Unknown
using. So start with Sarah.

Sara Saperstein
And Sarah, Sarah seen an elite data scientists that mass mutual, you go ahead and say

Unknown
that are going to give a little bit about what you do and what Yeah,

Sara Saperstein
yeah. So I build model for underwriting for life insurance.

I’ve been in mass mutual for almost a year and a half now. And before that, I worked at a company story.

Katie Robbert
Hi, I’m here where I am the CEO and co founder of Trusted Sites. We are a data science consultancy for marketing.

I primarily overseeing the data son inside of myself, I’m not a data scientist, I do a lot of product sizing, and how to go to market with a lot of the things that we’re building. Prior to that I oversaw a marketing technology team. And I worked in health IT in clinical trials for me before that.

Matt Ritter
My name is Matt Ritter come from on the data science team event data health specifically, I need a scrum team that uses natural language processing to extract prescription information professors that we get.

Unknown
Well, let’s start off with

Greg Ly
the question. So

Unknown
I’m just going to go.

Unknown
in your own words, how would you define machine learning and AI?

Katie Robbert
Again, not being a data scientist, machine learning is the type of artificial intelligence and so really, the way that we try to describe, especially in marketing community is that it’s a lot of math and automation. And really, it’s helping you to do repetitive things faster and more efficiently. And just find insights wouldn’t serve otherwise find by trying to do all of your computations and analysis?

Sara Saperstein
Yeah.

I’m coming from a science background. So to me, I specifically means that you are replicating the intelligence of some

typically, we think of special abilities, but humans might have or other animals, maybe application like that.

Unknown
And so I think that gets us will take any sort of decision as being AI.

Sara Saperstein
Whereas, and it’s, you know, very high words like to do we use machine learning just to try to calm down to the high.

Greg Ly
Yeah, so on that, on that talking about everything that kind of goes with the terminology, AI has been controversial issue, and some are anxious and fearful of the technology, and they feel that robots me take over the world. Well, some are optimistic about the opportunity to create, what’s your take on the role of AI? And what are some common misconception, so machine learning and artificial intelligence?

Sara Saperstein
I think that one common misconception is that you guys want to take over our jobs. And throughout history we’ve seen when we have new technologies, yeah, it ends up taking away some jobs, once people don’t really want to do, and it opens up new jobs that are more interesting, more exciting, and take you no more pagan

Katie Robbert
or more, you know, in a personal

Unknown
interaction.

Sara Saperstein
So so I think that overall, we have a thing. And, and yeah, there’s all this fear. I think that as people get more experience with it, they’ll be less of your whole. So I’ve had two instances where I’ve worked on projects, where we’re basically taking the expertise of some, you know, like a team, and then using that to build our models. One was for sending out automated campaigns, not setting them up by deciding how much money to put into each of the different types of, of ad groups on social media. And, at first, the team Oh, I should say the other one is having filling this model for underwriting. So in most of these cases, the teams were worried about these is going to take away our jobs, they didn’t like us at first. And, and we work with them, you know, they grudgingly oldest their secret sauce, how to do what they did. So you get older models, but that has to be rolled out the model. And they work with that this started to see Oh, this is actually really helping me, because it’s automating the boring stuff, and freeing up my time to do the things that I really want to do, like thinking deeply about these problems.

And yeah, I mean, their lives a lot easier. And then they really loved our

Katie Robbert
No, I wholeheartedly agree with everything that Sarah just said, You know, I have a love for machine intelligence, because it does automate a lot of those less glamorous house and it’s really allowed are very small efficiencies of scale a lot more quickly. And we can offer a lot more valuable service by not spending our days like in the movie screening for such type of things. So it’s, you know, we talked about there’s, there’s going to be two jobs in the future, there’s you will either manage the machines, or you’ll be managed by the machines. And so really taking that step forward to embrace machine learning and artificial intelligence now learning what it does, how it can help you will really help you like get to that point you’re doing with all your insights versus doing all of the working in the trenches.

Matt Ritter
And so I’ll extend this into the doom and gloom scenario. I agree. So,

so everything that

is true, and it’s great, and and there’s gonna be a lot of positive outcomes, I’m in potentially the net positive outcome, probably.

Unknown
But

Unknown
he likes my competence.

Matt Ritter
Strong probably, like,

Unknown
between.

Matt Ritter
So the thing is that exactly like you’re saying it, machine learning can change the economics of a number of accidents. And it can make things that have always been possible. But we’re not economically feasible, economically feasible. Again, that’s often good. But often for things are sometimes for things like facial recognition, something that humans, of course, have done something that the please can do, they can review videos and everything, when there’s a serious crime, they can chat, someone throughout the city is resolved the Boston Marathon bombing. But he doing very, very cheaply, you know, 100,000 times more cheaper than like he’s used to doing that changes the game in a fairly qualitative way.

And so that’s how this very thing

Unknown
that we’re talking about can actually can be both.

Unknown
I’ll just add on to that.

Greg Ly
I personally think that humans are always learning and machine learning, and artificial intelligence are always adding upon what has been learned already. How do you think humans and artificial intelligence will kind of work hand in hand? And will there be a time where there will be a greater need artificial intelligence versus human interaction?

Matt Ritter
So one of my favorite examples of artificial intelligence and humans working hand in hand was it for you know, what Deep Blue be curious as far off in the 90s. And that was a huge deal on the first computers. And but actually, the the very best chess players were, what were called send cards, hybrid human computer flavors, where they work together efficiently. And that’s no longer the case, I have to throw computers just straight up with us at this point. And so, but I think that there are some places where we’re humans, and will continue to work together more effectively than reward Oh, well. You know,

Katie Robbert
it’s, it’s an interesting topic that we like to talk about a lot, because there’s a lot that humans can do that artificial intelligence won’t be able to do for very long time. And that’s more of the emotional things, you have to see the sympathy, and even just sort of understanding sarcasm is home, we were doing an analysis on sentiment on Instagram posts, for example. And, you know, people will post something sarcastic, like it’s a beautiful picture of age. And they have beautiful tropical James and I say, this sucks. And so the computer will start to understand that as Oh, that’s a negative thing. And so, you know, there’s a lot that artificial intelligence still maintaining from humans, because we’re very complex people, despite

us wanted to just be

Unknown
you know,

Katie Robbert
and so I think that that’s where you’re still going to be bold. And, you know, we’ll be learning from each other for very, very long time, I hope.

Sara Saperstein
Yeah, all the systems that I’ve worked on, had a human in the loop component, and I can get

Unknown
better that way,

Sara Saperstein
is an interesting example of the chest, sitting realize that now that he ever was totally getting, you know,

Unknown
maybe that does give us an idea of the direction that things

Sara Saperstein
are going just automate these things and not, you know,

Katie Robbert
eight

Unknown
years or something.

Unknown
We’ll just

Greg Ly
shift gears a little bit. I think we were coming upon this. But when it comes to AI, and machine learning a lot of questions when it comes to ethics. So can you tell us about an important ethics or other issues that you’ve run into and your work?

Sara Saperstein
Yeah, I’m getting underwriting we have a lot of these kinds of cases, as you can imagine, or even a lot of these considerations as you can imagine. Life Insurance has historically been a means for kind of welcome

among populations, and so we have to be really careful and life insurance

Unknown
has not so secret history on this too.

Unknown
So we take it very seriously. And as usual,

Unknown
we go to

Sara Saperstein
transparency conferences, and we have an outside team that reviews are portals and then tells us how these features are, you know, even though they’re not directly related to

protect the features, they’re leading through some of that information, I think that were ethics really come into play is kind of, for our work, there’s three parts, one is

kind of a lot like how are you comfortable? But how are you storing your data? How are you making sure to protect

Unknown
people’s privacy, because

Sara Saperstein
they’re giving you their medical data, they’re giving you their financial data, it’s just a lot of sensitive information, and you have to make sure that people can see it on the limited that you’re protecting it from, you know,

Unknown
types of

Sara Saperstein
features, then there’s also an algorithm side, you know, the weapons of mass destruction type of where you have to make sure that you’re not

your models aren’t bias, but also that you’re not perpetuating bias.

One example there is. So medically, if you have lower BMI, like a really low PMI, your risk of mortality risk is

Katie Robbert
higher, typically, but there are some people who just through their

Unknown
genetic makeup, they just have lower.

Sara Saperstein
And our model, we were looking at a case where our model is penalized for someone that we believe was otherwise healthy, and just have a naturally small bill is really concerned us. Because if if What if by having that in our model, it’s those people with smaller builds his naturally. So to go to other companies, we are missing that data, now our training set, and the our model will continue to believe that small built on healthy. So you know,

we reviewed it with the team and you know, found out that actually the model was doing a better job of preventing that and the human underwriters. So we were in the right direction. You know, and we talked about ways that we might be able to mitigate that going forward. And it’s interesting, too, because this is the kind of case where, you know, anything genetic data for underwriting is media sensitive topic. And this is a case where it would actually really benefit people say, like, Oh, we can see from your genetic makeup that customizable,

Unknown
or

Sara Saperstein
something. Yeah, so the protection of the data, and you’re

not getting any bias.

Katie Robbert
Right now.

Sara Saperstein
So

Unknown
yes, there’s some reason that Mike,

Katie Robbert
Mike, that’s

Unknown
me.

Unknown
Okay.

Unknown
Thank you make sure that that’s working somebody else. You know,

Katie Robbert
we were talking about this a little bit before the panel started. And I’m a little bit to where I’m not dealing with any sensitive data, the marketing data to see what comes out of Google Analytics or other systems like that, you know, there’s no identifying information. But one of the things that we do is we use machine learning algorithms to identify social media influencers. And one of the questions that we started asking our minds is, is, are the results diverse enough? isn’t what you’re looking for? Is there too much bias? You know, when you’re looking for influencers for the next fashion thing, or food thing? Whatever the thing is that you want those influencers for? Isn’t a diverse group? And that’s a question that we’re really challenging people to think about. When they’re going out there, well, I just want to Kardashian or I just want the person is going to get the most three, is Apple representative to your brand? Or do you just want, you know, the person is going to get the most likes? And so we’re really trying to challenge them when we’re building those algorithms out and trying to like, ask them to really think through Is that really what you need to start to prevent some of that bodies that we are seeing that just comes

out of the same software over and over and over again, we’re trying very

Unknown
slowly but surely, to combat a little bit of?

Greg Ly
Great. So our next question, what is the recent trend about AI that you find interesting, interesting.

Matt Ritter
One of my pet things that my coworkers are probably annoyed that is active learning, I really think that there’s so much data in this world, but we need even more of it. And active learning is what helps guide the active collection of data. So the data collection, and actually even though the and I just I found time again, that when we started project, we have almost exactly what we need. And the successful projects are the ones where we can make a little logical leap and make it work. And of course, we tested in production, and we really are sure that it works. But there’s so many words, it’s painful were that were within visual distance of exactly what we need, but we just don’t have because it didn’t happen to be what made operational sense when the application was originally developed. And so I think, again, that was learning is where you can deploy effortful information gathering, often with a human to build out that data set and build a really efficiently so it doesn’t need to be 10 million lines, it can be 1 million or something so expensive.

Sara Saperstein
So one

Katie Robbert
is a big push towards.

Sara Saperstein
Because there’s a lot of phones that we can go, I think that the army to take that step back and say But should we, you know, should we build a model to predict someone’s criminality based on their photo? And so,

like, I think that this gets back to what we’re talking about earlier with the dangers of the I, you know, I think that the dangers are someplace herself, you know, and what to do with it. Because I’m just so excited to see that people are really cared about this and wanting to make sure that the malls, or Africa and

Greg Ly
make a really good point. I think what what it comes down what people that everything, just integration, people that are personal information that we don’t want out there. And a lot of times, I don’t think he will really trust the machine where a human can do it. So I think another question I have is, do you think there will be a point I think we touched a little bit on this already, in which artificial intelligence here actually understand human emotion you think we’ll ever get close to that?

Unknown
And

Unknown
is that where we want to go?

Katie Robbert
I would like to believe and i and i say that i very busy say psychology, I have a working understanding of the human brain is and emotions and the new was it goes beyond, I think that artificial intelligence will be able to understand and replicate basic emotions, anger, sadness, fear. But when you really start to get into those nuances, you know, when someone immediately has an instant attraction, or if something sort of rubs them the wrong way, and they’re kind of in a bad mood today, those are the types of things that I think artificial intelligence will have a more difficult time getting, I can’t think that use cases for why you want that. But, you know, if you’re trying to find, you know,

companionship. So, you know, there’s been, I think, movies that sort of the idea of what artificial intelligence is really meant to do Ex Machina great movie, but it’s a little scary. But I think that that’s sort of the idea of what they want

Unknown
artificial whole team to do.

Katie Robbert
I just, I think we would struggle to get there to really get it right. And it looks different for everybody. And it’s not a critical thinking, you know, x plus y equals z, that’s a decision that machine to me, it’s more nuanced than that. So I would venture to say no.

Unknown
And I’m probably wrong, but I wouldn’t.

Sara Saperstein
Yeah, I find this one, if you think about the complexity in the brain, you know, all the chemical reactions that are going on of protection,

and the different ways that you can, like, take those relationships within those neurons. And then you think about how long it takes us to learn. person, you know, it takes us so many years to learn to like babbling these words, long, able to detect sarcasm, which I still can’t

Unknown
do. So,

Sara Saperstein
yeah, I mean, sure we get all of our resources on it. It could be are, you know,

Unknown
why? Why do we want to?

Matt Ritter
first want to show you that my friends consider me a very optimistic person.

I think that everything that you’re saying is absolutely right, the deeply understanding human emotions really, really difficult. I think we are certainly not good at that deeply understanding each other. I’m not even going to deeply understanding ourselves. We’ve all been surprised that the intensity of our reaction at one point,

but I guess I might we cast the question slightly, you can do it well enough to cause problems.

Unknown
That is certainly Yes.

Matt Ritter
machines that can people’s emotions, I under appreciated machine, on advisor appreciate in this context, of course, the pot of thunder fears is slot

Unknown
machines, machines aren’t verified, we suck money out of people.

Matt Ritter
And then they come back, the next day is going again. And it’s I don’t know exactly how they work psychologically. But clearly, for subset of people, it’s not a small subset. They really push all the buttons. And I would argue maybe they don’t understand the emotions, not trying to put the design has managed to manipulate emotions anyway. And I think we even Marshall intelligence, maybe a great portion of people.

Greg Ly
I had a little bit, but I, I hope to get to a point where you understand us enough. But I think not having a I not necessarily have the emotional capacity of the human being just make sure that we’re still

Unknown
the picture. Right.

Greg Ly
So I definitely think

so each do work in different industries that have worked with AI and machine learning in different capacities. Can you elaborate a little bit more about? How do you think AI will continue to change in your industry and how it will.

Katie Robbert
Specifically in the marketing space, there’s still a lot of, there’s a large segment of marketers who still don’t know what to do with the data that they have level of outside sources of data. So one of the ways that we’re seeing machine learning and artificial intelligence really help marketers do their jobs better is makes sense of data that comes from a lot of different sources. So you have your social, your web analytics, your CRM, your marketing, automation, you know, they try to measure everything, which is great, but you can’t do something with all of that data. So we’ve been able to do driver analysis, predictive forecasting, text mining, sentiment analysis, all of those things, using machine learning algorithms, to help them understand what’s really important that they really need to be paying attention to. So in terms of the marketing space, we really see the sort of the newer, shiny thing, but the kind of more, you know, okay, this isn’t just music in my day to day, and I can make better decisions about what I’m doing with my marketing where

Matt Ritter
I can be optimistic here.

So I worked in health care, and health care, of course, with

Unknown
giant space. So when you think about character,

Matt Ritter
and how you probably think of Robo doctors, right ones that are doing automated diagnosis, and that is very exciting. And I think that it’s easy to get fixed and lose track of the many, many, many other problems with healthcare besides an accurate diagnosis. And many of them are operational. And what’s nice about that is that money is losing right now. You know, the, we only get angry insurance companies, we all get integrated, how long you can pay for things to happen. No one wants it to take that long, but that’s more work on everyone’s side. And if we can, and we are actively building out systems that can do those exchanges much more fluidly. The same things will happen but faster with less cost, and everyone was great.

Sara Saperstein
Yeah, so in life insurance, I guess, in a sense, we’re lucky to have a long history of rigorous. So it’s been sort of doing, you know, data science type things for a long time. And as insurance industries are starting to elect Trump in the trees are starting to incorporate more people are trained and data science, these, you know, it’s better, in particular, so I work on life underwriting, so my goal is ideally like to live longer than they think they will. And I would like for us to be good at predicting how

Unknown
long people will live.

Sara Saperstein
But more on the side of like getting people to live longer. So as we get more data, as we see kind of what correlates to that we might be able to, you know, working with medical teams have a better sense of what will help people to live longer. Some companies are using, for example. So it used to be that, you know, they would you know where you are right now and get a bunch of blood work done, to see kind of like, how your mental picture lots, but what if you are trying to improve your health, your internet startups, and you have a better result than what it looks like, based on your high cholesterol levels. You know,

we want to reward that kind of help here, because we want to help people live longer. So I think that there could be some interactions with our customers to help help them kind of what’s working and what’s

Unknown
important.

Sara Saperstein
One happy life. And there’s also this idea that, you know, you have to do people together into these responses. And as a, you know, as a policyholder, you would want to be

Unknown
grouped with people

Sara Saperstein
in a fair way, you know, you want to make sure that you are with people who are taking care of themselves as long as you are going to live on. And so you know,

getting better at discerning that recipe that we’re going into the key features, the actual like causative factors of your health, rather than, you know, stop that kind of just kind of historically been associated with

Unknown
it would be really great for

Greg Ly
this happening, touch base a little bit. So I’m on marketing, marketing, marketing, gratitude, currently, and I’m reading a book about predictive analytics. And I think we’ve definitely talked a lot about that on this panel so far about how machine learning and artificial intelligence kind of just helps predict basically things that we as human can’t see yet. But am I kind of noticed different trends? So in addition to like, kind of predictive predictive analytics, and that has that what you think is the new use case for AI?

Sara Saperstein
Not sure if that’s exactly, that’s a good question. But I’m really excited about self driving cars, I’m sort of anti part of myself

that we have more self driving cars, maybe is no longer is a thing for people in cities to own a car, and then maybe get a little bit away from the dependency on cars. I think there’s a lot of those kinds of unforeseen consequences of these, you know, ai that that might come about,

Christopher Penn
just give

Sara Saperstein
us like, you know, three us up from that kind of stuff, you know, the orange, or get us

on our own cars.

Unknown
Other

Matt Ritter
I think that there’s a lot of potential for systems that enable you to

navigate and negotiate marketplaces more efficiently. So this is something that I thought about off and on for a long time. I’ve never actually put into words before, so bear with me. But I think that, you know, a lot of the reasons that we do things that we do in this capitalist system, is because we are so cognitively limited, right? So why does 999 work, everyone knows what that means.

And so if we actually had systems that were waiting prices, but also thinking about the benefits that you’re looking for, and things like that, that really understood you, you know, I just bought a plane ticket. And it was better than terrible. But, but, um, but like, I’m not a terribly unusual person, in terms of what I’m looking for. Put it is for playback, you’re right, it’s not just a little pricey. So the, but it’s still found myself sort of cross cross referencing, you know, like, fa reports that I will often different airlines are late and stuff like that. So the nothing about that it’s conceptually complex, it’s just the data is spread all around that something that can be very easily, I wouldn’t even led to artificial intelligence, instead of bringing data together. So I think that, that that’s fairly small ball, it could go all the way up to buying cars for thousands of dollars buying homes for godly numbers of dollars, and, and all the way down sale. And I think that not only would it say to you a few minutes here and there, but it would eventually allow pricing, which is kind of the market signal that we’re counting on to move our economy to be much more rational than a really obvious kind of this kind of duties that we see 9999

Katie Robbert
it’s not a new use case. But one of the, in the past few years, the more reason is this idea of the smartest systems, I mean, everybody has them on their phone, everybody hasn’t been their house now. And I really think that has changed the way that we schedule ourselves. So we do our daily tasks. You know,

Unknown
we get so used to now just yelling out of machine. Just actually there’s a funny story with my business partner

Unknown
who was

Katie Robbert
assigned to us. We were sitting waiting for a client meeting. And I asked him a question. And he started responding to me as if I was smart assistant. He’s like, Okay, so here’s what you want to say, Katie?

Unknown
space,

Christopher Penn
can you buy?

Unknown
Right here.

Katie Robbert
But I so i think so while it’s not an upcoming thing, I think just more of it the normalization of artificial intelligence in our everyday lives, making it more accessible to people taking the mystery out of it, just making it something you can literally just grabbed off the shelf. And Ok, now I’m using artificial intelligence and it’s not scary. I think that’s really what I’m excited about.

Greg Ly
I’m going to go back over little bit cheaper, your

Katie Robbert
visit just something that

Greg Ly
I’ve noticed, and when I’m out with friends, or if I’m talking to someone in the room, and either Siri or Alexa and listening, that sometimes I’ll see an ad for I’m I’ve managed up to Denver, in July. And I recently saw an ad that came up on my phone right after I talked about it with someone. So in terms of that, do you think there’s a point in which we’re AI? There’s a line that needs to be drawn with how much data know? And if so what is outline? And how much should be artificial intelligence versus humans actually kind of controlling what what data that they have?

Sara Saperstein
I will say first that I think statistics can help us with this kind of example. So maybe they’re the airlines are pushing them for for some reason.

flights, they want to encourage more there. So you might have seen something previously that suggested Denver, and then and I’m not saying this new indefinitely their case. But I think this happens a lot. So it kind of praise you and you don’t know about it.

Unknown
And then you’re like, Oh, I should go to Denver. And then

Sara Saperstein
when there’s an app for it, then you notice it, because you’ve been thinking about it. Whereas if you hadn’t been talking about going to Denver, and he’s on have, for example, say you wouldn’t even register it.

Unknown
So there’s this

Sara Saperstein
cognitive bias, there is confirmation bias to pick up on those things. So I think that we see that,

and sometimes that’s not really what’s happening.

Although, I don’t know, you know, like some of these they might be be listening. I feel like my Alexa is not listened to me.

Unknown
A little better.

Unknown
Um,

Sara Saperstein
but yeah, so I think that is important for us to think about what are the cognitive biases and one of the ways that you can be useful in this like assessment of how the algorithms working. And I think that a lot of this comes with the fear, educating ourselves like you’re doing becoming to ga events, too, because

I think that educating yourself on how these things work will help you be better discerning

Unknown
what’s going on in the algorithm? or

Sara Saperstein
online? And then there was this the latter part of your question?

Greg Ly
How, or where do you think the line is drawn? Oh, really?

Sara Saperstein
Yeah, I think that a lot of that comes down to being very transparently aggressively transparent about what you’re doing, what data is being used, and how I, in my own work? I’ve been, it’s been recommended to me that I don’t tell, you know, that I don’t share how the sausage is made with our stakeholders.

Because Oh, it’ll scare them all the one understand it. And I have found that to not be the case, a lot. Like I love explaining in detail about our models over again, you know, I think everyone’s very intelligent, you can understand it just fine. And they appreciate understanding how it works so that when something goes wrong, they can help you they can debug it themselves. Or they can tell us Oh, this is what’s going on,

Unknown
that empowers them to use it.

Katie Robbert
And I think we need to do that for general

Sara Saperstein
public as well for algorithms that we interact with, on a daily basis.

Katie Robbert
Yeah, no, you absolutely make a great point is that education piece of it, and the transparency. And so I think it’s a shared responsibility between the companies who are building these algorithms, and the people who are purchasing them. And so, you know, there is some level of transparency, I mean, who really takes the time to read a privacy policy or Terms of Use, a lot of the information is in there. So the onus is on the people who are using the systems to actually read these things, and understand and challenge the company said, I don’t have enough information, I can’t buy your software, or I can’t buy your product up. And then understanding how to change the settings on your Alexa order series so that it’s not listening all the time. You know, so I think that there’s sort of that shared responsibility. That’s just not happening. A lot of people just want to do so like, cool. You told me a bunch of dad jokes all day.

But I need to also be aware how is listening? You know, I think some way recently came out with the campaign commercials, and they’re owning it. They’re really sort of like, okay, the AI is always listening. How did it know I wanted something spicy. And Sophie, what’s it like, just kind of blanks like, they’re only there. Like, we know that this is how it works. I thought it was just so genius. Because it’s again, it normalizes it. But you have to educate yourself on how these work and how you as consumers to protect yourself and your own privacy.

Matt Ritter
Totally agree. And I just say, you know, each new technology, there’s sort of a panic when it first comes out. And it’s not a reasonable without the panic, people would be taking advantage of. But eventually, hopefully with constant work,

Unknown
all governments

Matt Ritter
we find it out.

Christopher Penn
Well,

Unknown
what?

Greg Ly
We can find anything, but thank you for that. I think I less afraid now.

Unknown
I’m so looking forward.

Greg Ly
How do we individually and collectively prepare for the future of work with AI pain?

Katie Robbert
Be curious, ask questions. I think that that sort of, you know, I know I’ve said I’m an artist, scientist, but I’m perpetually asking questions. I’m not shy about asking questions. How does that work? What does that mean? Why did you do that? What is that, and really just trying to understand what’s happening. But you have to have that motivation, you have to want to know more, you’re going to find you know, against look back to that idea of there will be two jobs, either you manage the machines, the machines will be managed by you. It’s going to come down to that you know, self motivation of you want to learn more, you want to at least be able to ask questions was that they use versus just being complacent? And okay, well, the machine will do that for me while you’re on the job. So here you are. That’s how

Matt Ritter
I think that there are absolutely things you can do. And one thing to keep in mind is it’s not. I mean, I think probably everyone in this room has tried to build something of one type or another, I have been many of you, coming programming, General Assembly, and homes takes longer than you think. So,

so this will happen. But it won’t be like you woke up tomorrow, and you should have been studying all night. I would say that, regardless of your role. And other thing that I think is important is that

the science is going to be important. But data scientists telling everyone we’re going to be able to find the test great. Many people are going to be marketers for companies that have nothing to do with data, but use it to make some of the processes efficient. And I think that the right way to prepare for that role is to play around a little bit you know, like dramatically over fit the model just to like deeply understand what that means. And use some garbage in garbage out good understand what that means. It just gets the point where when someone awesome, like Sarah comes to you and tells you how the model works, you have played around with it enough just a few hours, so that you really can understand about this apple business.

Sara Saperstein
stories about ways that models can go wrong. So one of my favorite stories, which I repeated over and I said before is University of Washington built a also presented say aspect as a husky.

Unknown
And

Sara Saperstein
so they decided to build a wolf detector like well, purchase has to be discriminated. And so they had all these pictures of huskies and all these pictures of walls, they

Unknown
became a model and it did really well. And they’re like, Oh, this is great. You know, we can have to smoke detector.

Sara Saperstein
But definitely like, Okay, well, we should understand how the models working. So they look at what pixels are most, you know, you can’t get into some of those higher order names that easily enough to say, okay, on the pixel level, which ones are the most predictive? And they found that it was great. It was it was kind of the pictures of Husky we’re not in snow and the last word,

the story so that you can have that critical I have, okay, this model is this. But what other things could pick up on

Unknown
the addition to

Katie Robbert
how we individually?

Greg Ly
How do you think how many startups in Boston with good hobbies, that you guys are part of our leveraging AI in their products

Unknown
and services?

Unknown
House other companies? know,

Katie Robbert
it’s interesting, I think that a lot of people are trying to understand it better how they can, they might be starting smaller, something like a robotic process automation to try to automate something. I think a lot of companies and startups so trying to start to dip your toes in it, I don’t have any hard data to say that that’s what’s happening when we go to other conferences. That’s what we’re seeing people talk about more, because they want to be able to scale, they can’t add head, but they need to be able to do more work. So how do you get there, okay, machine learning algorithms, artificial intelligence. So I guess can become more standard. I think, as companies like Watson, they don’t have like, they don’t have all this stuff straight on the box yet, but it’s becoming more it dropped more accessible for companies to use, I think that it will become easier for companies, especially

Unknown
startups to start to

Katie Robbert
say, okay, we can build on foundation of machine learning and not have to add a lot more people right away.

Matt Ritter
We might have found something we can disagree on it. And I say this because I disagree with myself

how companies pieces and I think in the big build versus by him. And this is a unique API, but it’s got its own flavor. So you mentioned Watson and of course, Amazon wishes some of its own kind of black box services. And there’s entire startup companies that will basically suck out your data model. And then at any point, you can send them a monthly will do and I’ll send you back a label, and you never

Unknown
get access to the Molly just pay them for pain.

Matt Ritter
Or you hire people you know, or maybe you hire contractors in between, I would be very curious. Either you started thinking about the best way to do that, or how you make the decision and hit what the builders but

Katie Robbert
so, interestingly, you actually have all that a lot of our clients, we are on the side of building ourselves, because we haven’t found a lot of trade off the shelf. products, you can do predictive modeling within marketing before Google Translate for examples for searching for social media. For me, we’re on the side of building it ourselves with

our classes you help people by because we get asked that question a lot of times they will or I spy something off the shelf or not, you can find us and we can build for you. But a custom because everybody wants something different, we haven’t yet cracked the code on that standard thing that works for everybody. So I’m, I’m on the side of Bill. But that’s because of

Sara Saperstein
your option of buying is that there’s another team of people who are using these tools and deeply about the data, then, you know, that’s just kind of like do your searches to see what makes the most sense for your company, grow. If your solution to buy is to have fun IT system were Feeding America. And there’s not anyone who say to people about how to use and how it’s

Unknown
going to be generated.

Sara Saperstein
garbage in garbage out some situation. I already got this at some of the

Unknown
like automation for data science.

Sara Saperstein
Because the the running running the models easy, but knowing which one is run, knowing how to end of the day before it was in the model,

Unknown
knowing what considerations that need to have that sort

Sara Saperstein
of science comes into play. And I do think that, you know, you need some professional team working that thinking about the data trying to understand, you know, where did everything right, so we’re not considering? What are the, you know,

Unknown
special considerations for this form

Sara Saperstein
of data that we choose Come on that and he will important. So I don’t know, it’s like, you know, I, I agree with me that one standard probably designs, and I’m biased from, you know, organizing the Boston data science, I see a lot of companies that are using data science.

business. So I don’t know too much about, you know, Tobias, I don’t know who’s not using data science. But it does appear that the industry is going

to have that. And then yeah, like, just making sure that companies are, are hiring the potency to to get the work done rather than kind of shortcuts and then

Unknown
models.

Unknown
Kind of I think

Greg Ly
it’s the models work well and has shown kind of success rates within a certain industry, and buying might be good. But there’s also the benefit of building your own model, since you get you are looking for specific date and you want to use artificial intelligence for something specific that only you have, that no one else has done, then.

Unknown
Probably fast. So, um,

Katie Robbert
so you’re bringing up a really good point about understanding the model itself. And so on the side of Bill, as you’re making things, so I would have my answer of Yes, I’m always on the side of Bill. But if you aren’t by you do have to have the right staff and people to even understand what the thing is you and where we see a lot of companies and clients go wrong is go buy something off the shelf and say, Well, this will work, I can just do this. And they’ll plug it in and turn it on. And then it doesn’t do the thing that they wanted to do because they don’t understand how to use it. So you’re absolutely right. You made a really good point. I

agree with her.

Unknown
I guess we’re

Greg Ly
all in agreement. Yeah. But definitely with investment, when it comes to models and building by the team behind it is with most important I think a lot of holidays, made by product and not necessary, have the infrastructure within or have not actually invested your time and training employees are getting the right employees actually help

Katie Robbert
build upon

Unknown
that data. So I think that is not one of the most important.

Unknown
So we have time for tomorrow.

Greg Ly
So what’s your recommendation for anyone interested in getting into the field of machine learning? And AI? And what’s something you wish you had to know?

Katie Robbert
Um, I wish I had known that might not be useful.

Unknown
Not

Katie Robbert
you know, I would say start looking at blogs articles, there’s a lot of really good open source information. Our studio is great place to start. If you want to learn how to code or understand, you know, sequels are really great, because the site as well, there’s a lot of free resources available for people who just generally interested in testing waters. So again, I would go back to be curious, that’s how you start. And that’s how you’re going to continue to be

Unknown
successful, or any career but data science. Yeah,

Katie Robbert
it’s really, it’s difficult. It’s, you know, overnight, as he said, You have to really be motivated, we have to really kind of wanted.

Sara Saperstein
Yeah, I would say learn the fundamentals. There’s a lot of math behind it, because classes, if you can see the three programs in data science, now they have those, I think that’s awesome. What I wish I noticed that you don’t have to waste your points in grad program communication program, is I know, we went straight into ambassadors and then finish the work. In the field, they did the right thing, just as long as the program that you are in, really does cover those fundamentals, because there’s the tools, and there’s understanding how the school reviews, but then there’s understanding the math and accomplish

Matt Ritter
something that I’d say, I’m 90% sure it’s too is that you should learn to domain. So we talked about data science and math actually a little bit ridiculous, right? There’s lots of sciences, and they all use data. And so, you know, it’s a certain sort of alluding to many people get into data science by studying a specific sites and physical science, getting really good at processing that data, and then realizing that data and whatever else is sufficiently similar, but you might as well jump right into healthcare, you might as well jump right into x, y, z. And because the data is somewhat different, and possibly different, but you can, you know, the natural language processing is the words, the doctors right are not natural language.

And so it is all the techniques are chronic disease is a different problem. And if your interest marketing, you’re probably going to have much more data, right? Because there’s so many events that are relevant marketing, etc, etc. So you can find us you’ll nutrition in, that will really help accelerate you to bring the right things

Unknown
for your direction.

Unknown
I don’t think they need to know this.

Unknown
Industry.

Sara Saperstein
I think that one so that you really understand the ins and outs

Katie Robbert
of something is good.

Sara Saperstein
What would I find? I really love that the design is that I can go from working, or even before that working in computational neuroscience and working in Atlanta, and then now the life insurance, I was interested in so many different things. And these tools can apply across a range of different fields. You know,

depending on the type of data problems.

So yeah, like, I think that you don’t necessarily need to be a domain expert, even building expertise. But I think those tools,

Unknown
and I’d have to like

Unknown
just a small gap to offer data science offering.

Unknown
So just around us all

Greg Ly
off. Is there anything left that you guys

Unknown
would like to say about AI machine learning?

Matt Ritter
models accuracy, 100%, you probably over

Unknown
example, I’ve seen 95 on my data today, and I’m like this.

Greg Ly
So with that, I’m going to open the floor to audience, the audience for questions. So does anyone have any questions?

Christopher Penn
with what’s happened in Europe with GDPR requiring model interpreted billion California is privacy legislation likely didn’t seem How are your companies starting to try and stay ahead of legislators who are legislating AI, even though they don’t even understand how the internet works?

Unknown
Yeah,

Sara Saperstein
we we already are, you know, that’s already been a huge thing on our mind in the type of data that we have. And, you know, we’re in a very heavily regulated industry, but we want to make

sure that we are

even even better than that. So life insurance, New York recently came out with really strict requirements on disability. And, you know, yeah, it’s just always top of mind for us. So I think that because we’re not

Unknown
commerce, it’s a little different.

Sara Saperstein
In that case, it’s more about explaining to people like winding up with this class that pays off, what particular health factors made a difference?

Yeah, and then, as far as the data is really, very much staff of people who are our

Unknown
nation.

Katie Robbert
So we do work a lot in our state our in a question of people’s behaviors on the web. And so we’re fortunate in the sense that a lot of these systems such as Google Analytics, and a lot of things like HubSpot, built in those settings as they need GDPR is coming. And so we make sure that our company is compliant with what we do with the GDPR regulations. And then when we work with other clients, we tried our best to do the same. That’s honestly that’s their decision what However, they want to monitor your own data, but we always try to lean towards, okay, here are the best practices, even if it’s not affecting you at the moment, it will at some point.

Sara Saperstein
Yeah, I think that’s a little bit of a difference, too, is that with

having your data collected, and you may not know about it all, they brought up such a great point earlier about taking that initiative ourselves, to go through our Facebook privacy settings, and notice he was there and all these devices. But having people that’s like kind of wer passive as a collective versus data that you are given a company? Yeah. So like in the life insurance case, you’re filling out an application, and you’re like, you know, what this is going towards

going towards, you know, underwriting your policy. So I guess we have a little bit of sense that it’s like a lot of times already

Katie Robbert
really clear.

Unknown
What, what makes us more than

Unknown
anyone else have a question?

Unknown
So one of the is not criticisms of a high, but observations to that most actually already has to be narrowly focused, and I was worried about your face, and how that kind of shapes all of your industry is that, for instance, there’s a healthcare where we’re still just treating sick people were obviously that could be probably used to push,

Unknown
share a lot more effectively, and then drive down costs.

Unknown
lectures, it overlaps.

Matt Ritter
Yeah, great, great point, I would say there’s there’s two interrelated issues. As you noted, healthcare has had a problem with

kind of is over focus for Well, before data science was a thing. And, you know, that has a lot to do with payment structures. So you probably don’t have a doctors get paid to treat sick people not to keep people out, which seems sort of like just being silly to draw that distinction. But it makes a huge difference. There is legislation and there are programs, active programs, I bet try to do the opposite to get paid a certain amount, her health per patient. And if it costs more you the doctor to treat them, you have to eat that cost. So you suddenly become very interested in keeping the cloud.

And of course, that, certainly speaking, he has nothing to do with data, but they start to get very interested in how they can you say that the people Welcome to the so that’s sort of changed the business model, change the regulation.

From a data perspective, one of the other competitors for technologies that

Unknown
I think are very excited in the forefront,

Matt Ritter
is called transfer learning. And that’s training, sort of these general systems that can then just a little bit of extra training to do some specific thing or another. And they’ve had some success with video games, obviously, very different. But it’s sort of this concept that you know, human baby, once once they’re old enough to throw a red ball, and you throw a blue ball pretty much instantly, or even a blue.

And so trying to get a little bit more like that.

Katie Robbert
So ironically, and that doesn’t make sense, everybody in this room, but marketers are really bad at collecting data, like horrible at it. And that’s one of the things that we’re trying to work on is getting better data questions so that we can get marketing marketers out of this reactive only space, we have a hard time getting people to understand what even happen before we can even introduce AI to what could happen. And so that’s, unfortunately, in our industry, that’s kind of where we’re stuff is getting better data collection. So you can even apply some of these words, technologies, which is

Unknown
that’s where we are.

Unknown
I think it’s a really fascinating question.

Unknown
There’s sort of a trade off

Unknown
in a sentence between

Unknown
privacy and having these sort of other images. So for example, in the advertising space, if you want your

Unknown
advertising can be hard to exactly

Sara Saperstein
the things that you want to know about. You have to get that information and where you have to allow have that information should know about where you spend your time where we go into interested in. And I think that we need to think about where that balances, or set up systems, I think this might be a better approach setting systems where it’s really clear what you’re sharing and how and you, you have that control over it. So you can kind of tweak those lovers how much you want to tailor we have that have somebody read by these positive he didn’t go in and change the privacy side.

Katie Robbert
Think culturally resorting to the side

Sara Saperstein
where we fall in that I think that the US typically we’re more okay with losing some privacy in order to get your, you know, recommendations or, you know, like healthcare.

That’s all very protective, you would have to open up more in order to build the size of holes.

Yeah, I think there’s just is such a rich category of you know, I don’t know thinking in

those kind of our thoughts about what do you want? How can we serve some some people have some people with insults.

Unknown
Any other question?

Unknown
Yes.

Unknown
wondering about any sort

Unknown
of these developments, make your work easier in terms of machine, I will

Unknown
try to

Unknown
point our company will be those models. And for a long time, you have to remember managers, we have this tool that was just training and the whole array of your settings.

Unknown
Even with the market example, eliminated a lot of our working lives easier. So

Unknown
sorry.

Unknown
Since the

Matt Ritter
start with the bad news, which is that it was an internal visual, but the fact that it can be an inspiration, that is a pretty doable, and we need that something called that model performance tracker, which is simply once you start training more than 567 models, you begin to lose track of which one was the good one. And so this system, you know, automatically plots of a woman falling off against each other, or what to kind of stuff performance indicators, we want to choose not the best fighter overlap busy or something like that. But something that does well in some subset of interesting. So that’s something that I looked around for a while couldn’t find a good open source free and paid option for. And so I’d like to be much better if

Katie Robbert
we were able to develop the model, the using driver analysis apology, we had a client who had all their data sort of spread out everywhere, they had no idea what’s happening. And they were charged with understanding what was working on social media. So we were able to bring in all of these different data sets about 1000 or so variables. And what they thought was working was actually not it was really sort of the size of their audience and their posting. And so really being able to use

Unknown
those models to develop

Katie Robbert
deeper insights into things that we sort of take for granted. But what works on Twitter,

Unknown
we were able to really

Katie Robbert
do a more sophisticated analysis for them. So that they can understand that it’s not the number hashtag, but it’s actually the length of the post that’s working or the time of day, but really they’re focusing on you know, emojis that they use, like it goes very basic things that were able to apply those models to make their life easier in the long run. So then once we developed it, you can keep running it as they collect more data. And as people’s behaviors change.

Unknown
A

Sara Saperstein
lot of little things like the tools that may or, you know,

Matt Ritter
open source tools

Sara Saperstein
that are available. And I

have a different perspective, I’m really excited about this recent explosion of data engineering. You know,

Unknown
so I think typically the data scientists

Sara Saperstein
are, in the last few years, they’ve kind of this know, ingestion and an adult models in the worksheet quiet.

Unknown
But for me, the fun part is

Sara Saperstein
thinking about the data and building the model. And I don’t really want to worry about customer data. And I don’t want to worry so much about women are some of the you know, like nitty gritty of that. And so having data engineering, like he needs to support that, I guess start starting to specialize in you can hold for data science,

Unknown
I

Sara Saperstein
really like that I think that then allows people to become more efficient, to

Unknown
specialize the Sophie I guess really what it comes down to there’s

Sara Saperstein
so much to know who to those in our hearts, staying on top of it saying our schools our life easier. And so this will have to do it for the whole you know, those fail. So having a specialization.

Christopher Penn
So there’s been a lot of development with a show is auto ml, Google auto ml Watson’s auto AI from IBM, these tools promise to do feature engineering automated one hot encoding stuff, and then spit out, you know, run the batch test like he was a 70 models and the 15 variations of energy boost that we’ve done. What’s your perspective on?

Are these good things? Are these bad things are? Or are they shortcuts that without a train data science team, we just got to kind of say, Well, I put it into IBM Watson Watson is always right. So we’ll just go with whatever wants to tell us to do even because you have no ability to look at like the R squared and go, that’s like 5 million. That’s, that’s clearly about right.

Sara Saperstein
Yeah, so that’s a little bit what we talked about earlier,

where you don’t have people who know what they’re doing that, yes, garbage in, garbage out. I think that having these kind of tools that you’re like, on some of my time around the place.

Like I can have a band that just does that

Unknown
really easily.

Unknown
For me, like it was just

Unknown
easy, like, yeah.

Katie Robbert
But you know, we work within Watson a lot. And one of the things that Watson concert to do is if the algorithm for you based on the output that you’re looking for, and I think that getting that assistance from the software is going to be really helpful. You can’t bypass you can’t shortcut and say, Okay, that’s it. That’s what it says that’s the answer, you really still have to challenge it and say, is that the answer? Like, is it really 100%

Unknown
or so I

Matt Ritter
think the rate at which you’re seeing

valuable data science projects is like the rate in which I could suck water have a pool with a strong. And so

Unknown
this sounds like a bigger like a

Unknown
little bigger, stronger. The,

Matt Ritter
and that’s great. I don’t know, like, I think that there’s, there’s still plenty to do. And any friction, you remove one part of the system will increase the flow until the friction another part of the system is painful. And then that’ll be my job and like

Unknown
any other questions?

Unknown
What you focus on, hey,

Unknown
I’m asked

Unknown
to

Unknown
do is basic things that are involved for me, probably not production.

Sara Saperstein
I would say think about data, I don’t know if that’s kind of answered, or the magnesium. There’s no substitute for spending time for data, you know, doing a pair plot, understanding how the different variables are interacting with each other, questioning the assumptions that are making about the data.

Unknown
And I know that I’m hiring data scientists having that.

Sara Saperstein
Having that you asked me that question earlier, those are the things that I look for you to learn the tools, you know, maybe you know, this data pool, with this, you know, copies of this other set of tools, more that, you know, so easy to run these kinds of tools now that the hard part, you know, the new friction point in our lives in me, as always, is

Unknown
knowing products. And parents.

Katie Robbert
Surprisingly, I agree.

Now, and you’re absolutely right, us, while these tools are taking away some of the, you know, things that will take you hours to do it into a minute. So you still have to understand the data that’s going in, and you have to understand what it is he wants to come out. I mean, really what assuming is to take away some of the processing time, you know, the putting the variables together, transforming the spitting them out into a different form. That’s really what it’s doing for you. But you still need to understand what which algorithm did it happened? Why did they pick that one? or What happened to that variable that suddenly disappeared? I thought it was important. So yeah, you’re absolutely right.

Unknown
You’re saying

Matt Ritter
so take the pessimistic.

Everything they said is true, I will add another educational perspective, which is what they’re talking about is sort of the highest conceptual level stuff that AI in almost any domain can really do. And that’s been happening for a long time. There’s also like this really low level, nitty gritty stuff, a lot of it that will be specific to the organization that you are working with, that we’ve got going to be an open source tool for dealing with organizations to

Unknown
the database.

Matt Ritter
And how to connect with the, you know, Watson for this, but then because some other department wants to use us when you definitely should have the Watson into the Ws my first together. And suddenly, he runs a standard everything. And I guess I just had this moment where I was sitting with our architect or designer, architect, and had very much from perspective of different friction points, we realized that we could use criminal records. In place of the JSON else, we’re moving our game, we have

Unknown
tons of tons

Matt Ritter
and tons of records. And we realize we can go much faster if we use some pre digested heard about things. And then all of a sudden, we’re worrying about like network latency between us and wherever exactly Amazon’s AWS thing is, or Google’s thing, or, you know, whatever reason. And it’s like really getting down to the level like, what’s the amount of time it takes a bit of information to move from our hard drive to our CPU to across the wire to their CPU and the GPU, compute, compute compute, down, down, down, down, right. And once I was thinking about the system on that incredibly nitty gritty level, I realized that I was going to have a job for long.

Unknown
Oh, so this is like this crazy question. What’s your prediction

Unknown
on singularity would have

Unknown
one or two reasons?

Matt Ritter
So in an effort to get any answer at all, I would say that, but I think Bill Gates said that people always overestimate what they can do it one year, and underestimate what they can do in 10 years. And actually, obviously, we

Unknown
have another question over here.

Unknown
To your

Unknown
point about how you really love the modeling part of the event data,

Unknown
but not so much the deployment in or ingestion portion.

Unknown
As you he about specialization, right? Like how do you ensure that, yes, you’re doing the person you love and somebody else’s moving parts of day love, but that you guys can still work and collaborate together, that you’re building something that can be deployed, especially in a scenario where maybe building something offline

Unknown
is great, you can run a very long time, but it has

Unknown
to

Unknown
be running a dynamic situation, just consuming data,

Unknown
that you’re asking scientists thinking about that as part of building a model?

Unknown
That’s an amazing question.

Sara Saperstein
That’s such a good question. The way that you you deal with that, I think is that you collaborate with those teams posted on the beginning, you, you know, like, you can’t be at all in our stand ups, right? Like you can’t use it enough to say, but we can have these checkpoints where you do talk to each other about what are the challenges you consult at the beginning of the project and all throughout the project. I think that maybe there is a certification before of maybe some companies will have this were inside work on a model. And then they want a lot of it was wall to the engineers engineers were like,

yeah, that’s not a consistent on like, the data, scientists might be oblivious, but like the rest of the company was not happy with that. And it makes them all sizes, too. Because you don’t know like, you might build a model that doesn’t work as fast. And then you realize that’s not going to work on the website, because no one wants to wait 10 seconds for your default, having having this communication involved. And

Katie Robbert
one of the steps that we like to skip over a wallet, but it’s one of those critical is that planning phase of a project. And it’s the most boring, it could be the most time get some good documentation and writing up the different scenarios. But then also making sure you get that input from the different stakeholders. I was a hunter manager for about a decade, work with epidemiologist, software engineers, US stakeholders. None of them agree. And myself sole job was to get them all on the same page about the output. And it’s tough, and it takes a long time to give up. But that’s the most critical piece to ensure it to your point that these different teams are working together towards the same goal with the same expectation and understanding is that planning art is the worst part. But it’s also more important.

Matt Ritter
obviously have a plus one the product managers are is everyone’s responsibility, but the product managers are the ones that the bus stops there.

Unknown
Thank you guys for being here.

Unknown
You’ve spoken a lot about machine learning as automation of the prediction. But I’d like to get your thoughts on

Unknown
what happened, return the process around,

Unknown
instead start generating content. And I’m under degree fearful of this. Because if you have a thing that can

Unknown
understand what it means deeply to be one class or another, you can use that same thing to generate things that are indistinguishable humans in. So for instance, you could generate a video of Donald

Unknown
Trump declaring war on bacteria. It’s true that in an in an era when less and less resources are dedicated to investigate journalism, that will propagate and and cause

Unknown
mass destruction

Unknown
on high scale that I don’t see many systems in place feel, how do you as pioneers of the AI machine learning from people about generated

Unknown
processes?

Unknown
And what can you do this week, and I

Unknown
think that’s a pessimist. pessimism

Unknown
shouldn’t be a great. You know,

Katie Robbert
it already exists. We know it says, you know that the AI is already creating content, is also working on a PR agency. And

it’s so easy now for AI to just spin up an article about something completely, you know, simple that we don’t even really think about, you don’t realize that it’s actually written by a machine. And so I think, my personal opinion, is that you still have to have those checks and balances. So yeah, you can have a machine churning out 100 articles a day, because you don’t have to stop, you still need that person who’s the gatekeeper to say, this one is on brand for us, this one, you know, is accurate, least you can fact check it, you know. So I think that’s where sort of we’re talking about, they’re still going to be a role for human beings, you have to apply that critical thinking and human judgment and say, Okay, this is actually something that we can push out responsibly, and ethically. And so, and I’m not saying that all companies do wish, a lot of companies don’t use this is just my opinion of how it should work. Perhaps it’s in that either thing, and I understand that, but it’s how we operate in our company, we use a lot of AI to create a lot of our content, but we need to make sure that it’s Sunday, possibly.

So you should be afraid, and I can switch up here for that, because it is already happening. You know, that’s how a lot of people like their families got torn apart to the last election, because of Facebook, because of what Facebook was showing, depending on your interest, depending on the type of people You follow what your interact with, and you started to only see the things that balance that possibility to get that broad spectrum of content that sort of showed both sides of it. So it is a scary thing. And it’s it goes back to that it’s your responsibility to be aware and educate yourself on it, you know, but it is also shared responsibility. These companies be doing it responsibly, ethically. And unfortunately, we have no control of that was only control ourselves.

Sara Saperstein
Yeah, with the easy things that are out there. I think our days of Oh, that’s a sharp I can tell by the principles are definitely over.

Unknown
I think that a lot of it is, you know,

Sara Saperstein
I think that those of us who have grown up on the internet, are more discerning, and we know to be very careful sources.

But people who haven’t are hysterical, and I’d say probably also, even those of us who have lesser, credible sources. So you know,

I think part of its on us as models to think that high, slow down, take that time to educate people, so that they can be more discerning. And then I think it’s also more along the lines of cyber security, where, you know,

Unknown
it comes with a circle of

Sara Saperstein
trust, you know, do you have, you know, news media that you can follow the contrast? When I get when I see an article, especially the question, is this a reliable source of this as proof of a reliable source? You know,

it’s unfortunate, and that’s kind of the way that we’re going. I’d like to have more

Unknown
like, to more optimistic,

Sara Saperstein
approach them to speed like paranoid or something.

Yeah, I think in each other on what, and asking, you know, having more critical thinking, this, this makes sense. You know, unfortunately, there sometimes seems a little chaotic. Yeah. And so if you all in the US, you know, we put her in the morning, like, Who knows?

Katie Robbert
A little bit careful.

Unknown
Um,

Sara Saperstein
but yeah, discerning eye and then those of us who are in this field should take responsibility to

Unknown
your

Unknown
All right, I’ll do my very best, I can do my very best to be optimistic.

Matt Ritter
i think i think the best thing okay the one thing that was worth remembering is that kind of early days with computers you have like these you can repurpose it to whatever hacking into the department defense in here is a little bit cold water and it’s by all accounts we as as he should not have survived that like lack of security lack you know over interesting

So trying to draw a really good analogy, maybe, although ridiculous things will happen, we will fund it to

the countering that the things move much faster. Yeah.

He said he he best practice if I can make is that, although it’s a tragedy, when a lot of people are misled for a matter of

Unknown
hours or even days, more in some cases we

Matt Ritter
deliver, including overlapping, critical decision making democratic process.

Unknown
To some extent,

Matt Ritter
people that tend to get really worked up about these things can do art

direction and other people, confirmation bias cetera.

Unknown
So

Matt Ritter
I guess my hope, and it is I said, it’s really

more of a private citizen.

Although it will be that they will be staying there will be lost, it

Unknown
will be

Matt Ritter
just one of the many things.

Unknown
So one of the common themes a

Unknown
lot of these questions

Unknown
around is.

Unknown
So I’m curious to have you guys were highly regulated industries, how presence in the US

Unknown
Council have said, in some way, shape or form line for the algorithm to generate?

Unknown
How you guys are operating?

Unknown
Except, except for you?

Unknown
What?

Unknown
What is your interaction with the guardrails

Unknown
and the enterprise?

Sara Saperstein
Yeah, we have many different departments,

Unknown
and we are

Unknown
regularly auditing all of our models. We have those data science models, and we also have models.

Unknown
And yeah, like, so we just we have these experts to move over regulations to

Sara Saperstein
and also provided by our own internal regulations for how we’re handling the data, fairness of our models,

Unknown
models,

Sara Saperstein
how the tracking changes, and how connecting,

coming in, and, you know, what, if someone were to come in and and change the model object or something, and, and have it, you know, we’ve had it out where

you have dedicated to this or thinking about all the ways is to go along and teacher just like giving as much information as we can and then they tell us Are you insane?

Unknown
Yeah.

Sara Saperstein
It’s kind of amazing, like, how much of

Unknown
the business and based on that, but it’s really important.

Unknown
Yeah.

Matt Ritter
Okay, go into all the details. But we and your organization about the dealings with me for a long time, what’s new is the exact models and we do have some processes to deal with the new challenges that that provides or creates, but we, like every organization in the states have reasons of regulation and auditing around just how we manage the data on and to a large extent, are specific workflows, as well established practices that everyone in the company is offering, and,

Katie Robbert
unfortunately, marketing, so the Wild West, and there are no standards, there should be no, but I think we’re getting to a point where it’s needed. You know, I can’t go into too many details. But we were recently working on a legal case with a company who had been Miss recording some of their data to their customers, because there’s no standardization for how the data is collected, how to report it. And so while not life or death, it does, you know, Plan to Eat a lot of money, you know, so for some people that is life or death. So, you know, there’s a need for data governance, in any industry, that really flat status, make sure that they’re doing it in a standardized responsible way. Unfortunately, our marketing we just don’t have it, but it’s definitely definitely

Unknown
also give a shout out culture. So having a company culture of openness, honesty, transparency, blameless culture, so that people aren’t incentivized

Sara Saperstein
to hide something.

Unknown
I think all of us

Katie Robbert
one more question.

Unknown
Well,

Christopher Penn
thank

Unknown
you so much.

Unknown
Hanging around word,

Unknown
chat.

Need help with your marketing AI and analytics?

Machine-Generated Transcript

Pin It on Pinterest