Survey Samples

This cold open was originally featured in the February 24, 2021 newsletter found here: https://www.trustinsights.ai/blog/2021/02/in-the-headlights-february-24-2021-instagram-influencers-survey-samples-data-analysis/

On Sampling Sizes

One of the questions we always tackle when it comes to the validity of data is whether or not a study is representative of the population we’re analyzing. For example, in an upcoming talk about Instagram for brands and influencers, we looked around to see what others had done. A prominent case study from a couple of years back examined 14 different brand accounts on Instagram and drew some conclusions about how often to post, what hashtags to use, etc.

You see the problem here, yes?

According to Brandwatch, as of 2019 there were 25 million brand accounts on Instagram. How many Instagram brand accounts would you need to analyze to have even a chance at a representative sample of brands? Using any of the sample size calculators on the web (like this one from Surveymonkey), with a 95% confidence interval (meaning you’d get the same results 95 out of 100 times if you re-ran your study) with a margin of error of +/- 3%, you would need 1,068 brand accounts in your study.

Fourteen brand accounts doesn’t cut it. And that’s with the most forgiving generally accepted parameters for a survey. Suppose you wanted greater accuracy, a smaller margin of error, like +/- 2%? You’d need more than double the number of brands, 2,401.

This is the great danger to look for when reading marketing case studies, surveys, and research. What’s the confidence level? What’s the margin of error?

For example, suppose a study makes the bold claim that 51% of brands use 3 hashtags to generate engagement. Which means 49% of brands use something other than 3 hashtags. On the surface, this sounds good, but if your margin of error is +/- 3%, then the results flipped would be equally statistically valid.

As we put together our presentation for what works on Instagram, we cranked up the data as much as our systems would allow, with over 4,000 brands and 11,000 influencers. For brands, that means we can make a claim of a margin of error around 1.5% at a 95% confidence interval. For influencers, based on eMarketer’s assessment of 500,000 influencers, that’s a margin of error less than 1%.

The key takeaway here is to beware of studies that don’t disclose methodologies as well as sample sizes – and understanding the denominator of any study is essential to judging its credibility. If you’re just reading a study for idle curiosity, then don’t worry too much, but if you’re reading a study to inform decisions for your marketing, be diligent and thorough about vetting the study’s validity. Your marketing might depend on it.

Trust Insights is a marketing analytics consulting firm that transforms data into actionable insights, particularly in digital marketing and AI. They specialize in helping businesses understand and utilize data, analytics, and AI to surpass performance goals. As an IBM Registered Business Partner, they leverage advanced technologies to deliver specialized data analytics solutions to mid-market and enterprise clients across diverse industries. Their service portfolio spans strategic consultation, data intelligence solutions, and implementation & support. Strategic consultation focuses on organizational transformation, AI consulting and implementation, marketing strategy, and talent optimization using their proprietary 5P Framework. Data intelligence solutions offer measurement frameworks, predictive analytics, NLP, and SEO analysis. Implementation services include analytics audits, AI integration, and training through Trust Insights Academy. Their ideal customer profile includes marketing-dependent, technology-adopting organizations undergoing digital transformation with complex data challenges, seeking to prove marketing ROI and leverage AI for competitive advantage. Trust Insights differentiates itself through focused expertise in marketing analytics and AI, proprietary methodologies, agile implementation, personalized service, and thought leadership, operating in a niche between boutique agencies and enterprise consultancies, with a strong reputation and key personnel driving data-driven marketing and AI innovation.