Don’t Let Numbers Deceive You

One of the reasons the recent presidential election was such a surprise was that all the major polls showed Hillary holding a sizable lead right up until election night. There are many theories on why the polls were wrong, but two ideas struck a chord with me.

First, what people say they believe can be very different than what they actually believe. If someone is asked a question (who are you going to vote for?) and they’re embarrassed by their truthful answer, they might lie (answer with another candidate) or downplay it and say undecided. In theory, lying in polls will balance out with a large enough sample, but I don’t think that was the case in this election (Trump supporters were less likely to admit who they were voting for). Second, surveys done online or by telephone will always be skewed in some way. I doubt the group of people who are willing to sit on the phone for 5-10 minutes to answer a survey is a random sample of the entire population.

Those two theories about the failure of the presidential polling prompted me to review my notes from a book I read a couple years ago called How to Lie with Statistics. Whether given to us by management teams or discovered through research, investors are always looking at numbers. I think it’s a good idea to remind myself every once in a while how easy it is to be deceptive with numbers.

Correlation does not equal causation

If B follows A, that doesn’t mean A caused B. It’s possible they’re correlated but something entirely differently causes both of them. Even real correlations may be worthless. People who go to college make more money than those who don’t, but I don’t think that’s as meaningful as it appears. People who go to college are more likely to be smart, motivated, or raised in a middle class (or wealthy) family. Those people are probably going to make more money no matter what.

I ran across this Spurious Correlations website recently and got quite a few laughs. The purpose of it is to find completely unrelated things that are correlated to each other. Below are two of my favorites.

How is it that every toothpaste company claims 9 out of 10 dentists recommend them?

The law of small numbers is how. Companies will do a bunch of “studies” of groups of ten experts (or customers or users or whatever) and eventually one of those groups will come back with the result they want. And that one study is the one they use. Sample size is the problem here and it’s incredibly important, but how big a sample needs to be depends on what’s being looked at. A medical disease that occurs in 1 out of 1,000 people would probably need a study done on tens or hundreds of thousands of people. Other things can be adequately sampled with 50 people.

Even studies that are done well can have inherent biases in them that aren’t always easy to spot. There are certain types of surveys that people tend to exaggerate in one direction more than the other (similar to Trump voters being less likely than Hillary supporters to admit who they support). Asking about wealth is another common one. I saw a poll once that showed income distribution and the number of people who claimed to make $100k per year was far greater than people who answered $90k. This doesn’t make any sense until you remember that six figures is a common threshold for what constitutes making good money in America. Because of this, a meaningful portion of people who make $90k round up and claim they make $100k. That could have been a perfectly random poll, but the answers people gave still contained an inherent bias in them.

Details that aren’t there are the hardest to notice.

Charts can be deceptive. I saw a company presentation last year that had a vaguely-titled slide with a chart on it that had no label on the y-axis. All it had was the x-axis label (time) and bars that got higher every year. There was no way to know what the chart was measuring or the scale of the y-axis—it could’ve been showing number of users cancelling per year for all I know. It looked good though!

When numbers are presented on a chart, scale makes all the difference. I’ve seen a few company charts that have a y-axis that doesn’t increase in even quantities. At a quick glance it looks like the company is generating explosive growth, until you realize the y-axis isn’t scaled evenly. But charts with correct scaling can be misleading as well. A large scale can make a 10% gain look like nothing, while the same 10% gain looks huge on a smaller scale. The below two charts show the exact same information with different y-axis scales. If you wanted to sell the idea that this stock was volatile, which chart would you use?

Averages are another way to be deceptive with numbers. When I see the word average, I assume the source has taken the total value of the sample and divided it by the sample size. But average is actually a broad title that can refer to mean (what I just described), median (another fairly common use in investing), or mode. I’ve caught a couple companies using the term average in their presentations in a way to deceive shareholders.

A similar example of this was Valeant using average growth across their acquisitions as opposed to weighted average growth (the AZValue blog deserves credit for catching it). Look at the below slide from Valeant and notice the largest acquisitions are performing poorly while the smallest acquisitions have the highest growth rates. Valeant wanted to make this appear better than what I just said so they took the average of all growth rates and highlighted that number (12%) at the bottom. But that is incredibly deceiving. A typical investor would look at that 12% number and assume it’s a weighted average growth rate because the large acquisitions should have a bigger effect on the bottom line number.

As I commented in a Valeant discussion, I think this stuff matters a lot: “It’s easy to say the non-weighted CAGR is one small issue but I view things like that as big red flags. Management knew perfectly well what they were doing and if they’re going to be deceptive about something that small it’s a bad sign for the bigger, more important things. If the management team is both unethical and smart, those small red flags are probably all we’ll get until everything blows up (but of course then it’s too late).”

When I look at something a company shows me, it’s natural to only look at what’s presented. It can be hard to actively think about what they are leaving out, but sometimes what they don’t say is just as important as what they do say. Like I said above, details that are left out are usually left out on purpose.