Processing Unstructured Social Data

This data was originally featured in the June 12th, 2024 newsletter found here: INBOX INSIGHTS, JUNE 12, 2024: UNIVERSAL ANALYTICS DATA, PROCESSING UNSTRUCTURED SOCIAL DATA

In this week’s Data Diaries, let’s walk through a recent example of how generative AI can help us understand real-world data, taken from our Generative AI for Sales webinar. For our example, we wanted to understand how employees themselves may affect sales, especially in a B2C retail environment.

The first question we have to ask is, where could we get this data? Certainly, there are official sites like Glassdoor where people can leave ratings and reviews of employers, but the challenge with Glassdoor is that it doesn’t have a public API. It used to, but they locked that down in 2021.

So where else could we go? Well, if you apply for a free developer API license from Reddit, you can use Reddit. I applied for one and got it within the hour, and it allows you to do small data extracts. By small, I mean being able to download the entire contents of a subreddit in an hour or less, as opposed to getting a raw data feed of all of Reddit (which would be the use case for a software company).

That’s my starting point – Walmart has an unofficial employee subreddit, so I extracted 90 days of posts and comments from it using Python (which Google Gemini wrote for me, because my Python skills are ). Now, this data is almost completely unstructured:

We get the post title, the Reddit username (which is often a throwaway account for good reason), the date, number of comments, a link to the original post, and the post content itself. The goods are in the title and post content, but parsing that data would take ages. 90 days of posts is about 90,000 words. 90 days of comments is more than 550,000 words – about the length of the Lord of the Rings trilogy.

Thankfully, generative AI tools like ChatGPT 4-omni, Claude 3 Opus, and Google Gemini can hold more than 700,000 words in their working memory, which makes them ideally suited for processing this kind of information. We can use them to summarize the information at a high level:

And this toy example using just the posts is still actionable; if you worked for Walmart, this data would give you an action plan and a starting point to understand what impact employee morale has on sales:

Again, these are toy examples of prompts and responses; if we actually worked for Walmart, we’d want to break each of these major categories into sub-categories, and perhaps even build an enterprise-level system that could process them on an ongoing basis.

The key takeaway here is that generative AI tools give us superpowers for processing and understanding the data we already have. They’re incredibly powerful for helping us turn analysis into action.

Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.