Anscombe's quartet, and why summary statistics don't tell the whole story

Let’s say you’re looking at a spreadsheet of your customers. You have data about how many times they’ve logged in, how much revenue you’ve earned from them, and so on. You can immediately calculate several compelling summary statistics: what’s the average number of logins per customer? What’s the average revenue? What’s the correlation between number of logins and revenue?

Summary statistics allow you to describe a vast, complex dataset using just a few key numbers. They give you something easy to optimize against and use as a barometer for your business.

But there’s a danger in relying only on summary statistics and ignoring the overall distribution. Read on to learn how summary statistics can be misleading—and why calculating summary statistics should only be one piece of your data analysis pipeline.

What is Anscombe’s quartet?

Perhaps the most elegant demonstration of the dangers of summary statistics is Anscombe’s quartet. It’s a group of 4 datasets that appear to be similar when using typical summary statistics, yet tell 4 different stories when graphed. 

Each dataset consists of eleven (x, y) pairs as follows:

All the summary statistics you’d think to compute are close to identical:

• The average x value is 9 for each dataset

• The average y value is 7.50 for each dataset

• The variance for x is 11 and the variance for y is 4.12

• The correlation between x and y is 0.816 for each dataset

• A linear regression (line of best fit) for each dataset follows the equation y = 0.5x + 3

So far these 4 datasets appear to be pretty similar. But when you plot them on an x/y coordinate plane, you get the following results:

Now the real relationships in the datasets start to emerge. Dataset 1 consists of a set of points that appear to follow a rough linear relationship with some variance. 

Dataset 2 fits a neat curve but doesn’t follow a linear relationship (maybe it’s quadratic?). 

Dataset 3 looks like a tight linear relationship between x and y, except for one large outlier, and dataset 4 looks like x remains constant, except for one outlier as well.

Computing summary statistics or staring at the data wouldn’t have told you any of these stories. Instead, it’s important to visualize the data to get a clear picture of what’s going on.

How summary statistics can mislead you: a real-world example

Let’s look at a real dataset that shows exactly how summary statistics can be dangerous.

A great example is the distribution of starting salaries for new law graduates. The National Association of Law Placement (NALP) reports that in 2012, lawyers made $80,798 on average in starting salary. However, examining the salary distribution shows what law salaries really look like:

It turns out, law graduates usually fall into 1 of 2 groups. The majority of new lawyers make somewhere between $35,000 and $75,000 per year, and a sizable minority earns $160,000 per year. 

This is a bimodal distribution: there are 2 peaks that arise from 2 distinct distributions happening within the same dataset. 

The $80,798 figure reported as the average falls into the trough between the 2 peaks, and few lawyers have salaries near that number. A much more accurate statement would be that most law graduates make around $50,000 on average, and those who go to one of the top law schools make $160,000 on average.

There’s also something else happening here that you wouldn’t have observed if you hadn’t plotted the data. There’s a giant spike at exactly $160,000 in starting salary, rather than a peak with some variance. Why is $160,000 such a popular number for law salaries? 

As it turns out, this data isn’t based on actual legal salaries, but based on what law schools report to the NALP as their students’ median starting salaries. There’s a lot of skepticism about the $160,000 figure, and third-party data shows that the distribution might not be so skewed.

Visualizing the data helped in 2 ways. It revealed a better picture of what realistic starting law salaries look like, and also prompted a follow-up question that exposed a potential flaw in the data.

When should you use summary statistics?

This isn’t to say that summary statistics are useless—they can just be misleading on their own. It’s important to use these as just one tool in a larger data analysis process.

Visualizing the data allows you to revisit summary statistics and recontextualize them as needed. 

For example, Dataset 2 from Anscombe’s quartet demonstrates a strong relationship between x and y; it just doesn’t appear to be linear. In this case, a linear regression was the wrong tool to use, and you can try other regressions. 

Eventually, you’ll be able to revise this into a model that does a great job of describing your data, and has a high degree of predictive power for future observations.