Faulty Analysis From FiveThirtyEight

Let’s talk about “Which NBA Team Is Wronged By The Refs The Most?”

We comb through a lot of NBA fan feedback on social media and blogs and articles from media outlets on a regular basis. Sometimes we respond, sometimes we just let it go. But one article published by FiveThirtyEight.com last week was so egregiously wrong in it’s conclusions it warrants a significant response. FiveThirtyEight describes itself as a site which “uses statistical analysis — hard numbers — to tell compelling stories about politics, sports, science, economics and culture.”

But last week’s story, “Which NBA Team Is Wronged By The Refs The Most?” uses faulty data and draws incorrect conclusions — certainly not the sort of statistical analysis you might expect at a site run by Nate Silver.

FiveThirtyEight’s article uses publicly available L2M, or Last 2 Minute, report data, which includes the“last two minutes of games that were within three points at any time in the last two minutes of the fourth quarter or the last two minutes of any overtime period.” They then ranked all NBA teams by how many incorrect non-calls and incorrect calls (“bad calls,” as they write) negatively affected them within these reports. The result shows teams with as many as 28 (Brooklyn Nets) and as few as 7 (Orlando Magic). The conclusion, as made clear in the headline, is that the Nets are the team “wronged by the refs the most.”

But this is incorrect analysis for several reasons. First, these numbers aren’t grounded in any context. Were there 28 incorrect calls that went against the Nets in 4 L2M reports, meaning 7 per L2M? Not even close. Instead, the Nets were involved in 25 games that had L2M reports, meaning the total number of incorrect calls that went against the Nets, based on this data, from the start of the season through February 28, was just 1.12 per game.

Of course, every team has played in a different amount of games that meet the criteria to have an L2M report made public. While most teams fell in the 21–27 game range, the most was the Heat, with 35, while the fewest was the Rockets, with 14. This greatly affects the numbers.

So when we divide the number of “bad” calls for each team by their number of L2M reports, our numbers are far different. We see that the team with the most was the Nuggets, still with only 1.14 per game, while the fewest was the Pistons, with .35 per game. Most teams fall in the .5 to .77 per game range.

In other words, the difference between the team that FiveThirtyEight concludes is the “most wronged” by the refs and the “least wronged” is less than a single play. How exactly are they drawing such definitive conclusions from such small variance?

Of course, we should also note the success rate here — based on our data analysis of all L2M reports this season, NBA refs are correct when they blow the whistle more than 98% of the time, and correct overall (including non-calls) more than 93% of the time.

Another important point when dealing with such a small sample size is that the data from which FiveThirtyEight is drawing it’s conclusions is simply capturing an arbitrary moment in time. Let’s say they decided to find the team “most wronged” by the refs after November, using data from the first two months of the NBA season. Let’s take the Nets as an example. The Nets played in 6 games with an L2M during that time period, and had 5 incorrect calls go against them during the period, for a .83 rate — putting them right back at the middle of the pack.

But let’s take those same 6 games in which the Nets played and we saw a publicly-available L2M report. The opposing team in that game — the team playing the Nets, saw 6 incorrect calls go against them, for a rate of 1 per game. In other words, using the FiveThirtyEight faulty analysis, the Nets received more favorable refereeing than their opponents did.

This out of context analysis could go on. Let’s not forget what exactly the L2M reports really are — a small snapshot of a game, for a small number of teams. We don’t see the results of the last two minutes of every game. We don’t see the results of the full games. In the context of FiveThirtyEight’s analysis, it’s simply arbitrary and meaningless. (And as FiveThirtyEight notes, we don’t agree with the concept of L2M’s in general — as we have previously noted, we don’t even think the conclusions there are necessarily always correct.)

In a perfect world, referees never miss a call. Our rate is 100% correct for every team. But that’s obviously unrealistic. In reality, our percentages are far higher than the shooting percentage of any player. And this faulty analysis hurts the perception of NBA refs — experts in our field, who only care about fairness and accuracy, upholding the rules and integrity of the game, on the court.