Statistical literacy: 7 questions to ask yourself when checking statistics

In the Age of Information, we are constantly blasted by an endless stream of opinions and news. To add weight to them, statistics are often used as supporting evidence. Here are some concepts and a framework you can use to prevent yourself from falling into statistical traps. 

Hi, I'm Mauricio. Connect with me on LinkedIn to share ideas and keep the conversation going!

Statistical literacy is a subset of data literacy, focused on the ability to read and interpret summary statistics and charts in our everyday life.

The information we get on the news and elsewhere can shape our world-view and the critical decisions we make about work, and our private and public lives, and that is why statistical literacy is so important.

There are some statistical literacy frameworks and guidelines out there, like this one from Iddo Gal:

There is a lot to unpack there, and of course, I can't tackle every one of those points in a single article, but I want to encourage you to have some healthy scepticism. So here is my own framework, focused on 7 aspects and relevant key questions you should ask yourself whenever you come across statistics.

Deep-dive: 7 aspects and related questions

- Credibility

Who collected, processed and reported this data, and for what purpose? 

Track record and motivation go a long way in trusting the statistics to begin with. Beware of intentionally satirical content. And don't equate statistics from reputable and serious institutions to a random person's rant on their social media.

One is not like the other.

- Data Coverage

Does the scope of the data match the claim being made? 

Credit: sketchplanations

What is/might be missing from the data? Is it representative?

One real example, with awful consequences, is the under-representation of women in medical research for decades. That led to women often getting less effective treatments than men. 🤦‍♀️

- Clarity

How are the terms defined, and are they used consistently? 

Without specifics, you might be misled. Watch out for “weasel words”, such as many, some, few, lots, little...

Always ask yourself: “What do they mean by X? How did they measure it?”. As an exercise, change X for harm, willingness, happiness, violence, worry, trust… notice how tricky it can get.

They should be clearly defined and consistent across. Comparing apples 🍎 to oranges 🍊 needs a lot more care and explanations in place, if it can be done well at all!

- Granularity

What is the level of detail given? 

It should match the level of the problem or goal at hand. Summary statistics drop details, so we can consume information, and that's great! But important info can be lost or hidden when doing so. Check the example below...

Anscombe's Quartet: four different datasets, with two variables but different values. Their summary statistics (mean, median, correlation coefficient, standard deviation) are all the same or very close but look at their shape  These charts would lead to different interpretations and actions.

Credit: heap.io

- Plausibility

Is the claim extraordinary?  

Then re-read everything carefully, and look for the (true) original source and methodology, if you can. Be extra sceptical. 🧐

This often happens with (bad) scientific reporting that forgets that correlation does not imply causation. 

- Self-Awareness

What do you feel when you come across this claim and/or statistic?

Check yourself before you wreck yourself. Our emotions and instincts make us react faster, which is great in certain situations, but can be terrible for processing information. Be sceptical with yourself, too. Try to acknowledge and mitigate your own biases.

- Counter-evidence

Is there contextual knowledge or counter-evidence that can challenge the claim?

Leaving out information that would have altered a conclusion about a topic is problematic. Perhaps think about this: "What evidence would invalidate this claim?"

Two important things here:

Empirical evidence (reproducible, first-hand and systematically collected) is a lot more valuable than anecdotal evidence (subjective, second-hand and casually collected).

Don't individualise from the general, and don't generalise from the individual. Take this original claim, for example: Europe is on average colder than South America. It does not mean that on a particular day, a particular South American location can't be colder than another in Europe. Or that observing that invalidates the original claim.

Find the full framework, with key questions and an editable canvas.

Click here to download a copy

Let's practice!

Use the provided framework to reflect on the following article: National Study Finds That Video Games Could Improve Cognitive Function in Kids.

Read through the full article, looking for the aspects described in the framework.

Refer to this microlearning here to clarify the aspects, if needed.

Edit the blank slide of the framework with your reflections.

Compare your reflections with mine. Remember: It's a reflective exercise, so my answers are not the only possible correct ones.

Dive deeper into the topic

"How To Make The World Add Up" by Tim Harford: This book teaches you how you can critically evaluate statistical claims, using 10 rules and many examples. Great read and a personal favourite of mine.

 "Operationalization: the art and science of making metrics": This article by Cassie Kozyrkov discusses the challenges in creating metrics, particularly for subjective and fuzzy concepts.

Cognitive Biases: The Decision Lab website has an extensive list of cognitive biases, with great explanations, examples and potential ways to mitigate them.

 "Statistical Literacy: A Short Introduction": This very objective paper by the Statistical Literacy Project presents some key concepts that a statistical literate person should know. An important source for this microlearning.

 "How Data Literate Is Your Company?": This Harvard Business Review article talks about the business impact of data literacy, presenting guidelines to improve it.

Wrapping Things Up

Here are the key points we covered:

Statistical literacy is super important: It profoundly impacts how we process information and make decisions in all aspects of our lives.

Many models and disciplines: There are a lot of different aspects and foundational knowledge about statistical literacy. 

A handy framework: You now have a framework with 7 aspects and key questions to remember when dealing with statistically-based claims.

I believe all of this will help you tremendously in figuring out “what is being said” and “how much you should trust it”. 

See you in the next one!