Debunking your favorite NFL officiating conspiracy theories

Ask most NFL fans what they think about referees, and they'll likely say that the refs have it out for their team. Or the officials don't know what they're doing. And they definitely want the Kansas City Chiefs to win.

Well, NFL fans, ESPN ran the numbers, and as an overall trend, these theories just don't stand up.

To assess whether officiating is routinely impacting game outcomes, ESPN compiled every penalty that referees called in regular-season and postseason games between 2001 and 2024, the type of penalty assessed, when it occurred during the game, and which team went on to win. ESPN also compiled every challenge a head coach made in the regular season from 2010 to 2024 as noted in official NFL Game Books. Dr. James J. Cochran, professor of applied statistics in the Culverhouse College of Business at the University of Alabama, helped ESPN run an analysis.

Here's what the numbers say:

• More-penalized teams do not lose more often.

• In the fourth quarter and overtime, teams that commit penalties that lead to large yardage swings or first downs -- defensive holding, defensive pass interference, illegal contact and offensive pass interference -- tend to win more often than when they don't commit those penalties.

• Coaches' challenges have decreased by 47% since 2010. In the 2024 season, challenges led to overturned calls about 38% of the time, the lowest success rate in five years.

• Data did not show that specific officiating crews were biased against particular teams in upholding or reversing challenges, or that less experienced crews routinely had more reversals.

• Of the current head coaches, Nick Sirianni and Andy Reid were most successful at using their challenges. Jim Harbaugh and Sean McDermott were the least.

The fix is not in.

While some teams were penalized more than others during a single season, the number of penalties called against individual teams evened out over multiple seasons, according to ESPN's analysis. Being penalized more -- in individual penalties or yards -- did not mean more wins or losses.

"It tells us that we do not have evidence of any systematic bias in penalties called in the NFL over the past several years," Cochran wrote in an email. "I am sorry for the fans who believe the NFL referees are biased against their favorite team, but it isn't so."

Fox rules analyst Dean Blandino, who was the NFL's vice president of officiating from 2013 to 2017, said the league's own analyses over time also found that penalties were not a "major factor" in wins and losses. Actions such as turnovers had a greater impact.

"I always said that to coaches. I was like, 'Listen, you can have a year where you were on the wrong end of a bunch of calls,'" he said. "Over time, it tends to even out. It just does."

One limitation in the dataset -- and it's a big one for the conspiracy theorists out there -- is that it does not include non-calls, when fans, analysts, reporters, coaches, players, etc. believe that a penalty should have been called or that coaches missed opportunities to challenge plays.

There are also plenty of angst-ridden fans who believe certain officiating crews are biased against their team -- or just aren't very good at their jobs. Using our database of coach challenges combined with data on officials and their years in the league, ESPN assessed numbers around reversed calls -- including the makeup and average years of experience of the officiating crew reversing the call. The numbers showed no bias or pattern.

"It all seemed random," Cochran said.

When the fine is worth the finish

As ESPN and Cochran analyzed the data, one significant relationship emerged around what we'll call impact penalties. ESPN looked at all games in which a team incurred any of four types of penalties that lead to large yardage swings or first downs -- defensive holding, defensive pass interference, illegal contact and offensive pass interference -- in the fourth quarter or overtime of regular-season and postseason games. Of those games, it examined how often a team won versus how often it lost.

Over the past five seasons, the numbers show something unexpected: Teams were overall more likely -- about 54% -- to win games when they committed an impact penalty late in the game than they were when they did not. Overall, in games where they committed those penalties, 23 teams won more often than they lost.

From a statistical standpoint, Cochran said that relationship was significant. The same trend held true overall going back to 2010, even though rule changes, penalty enforcement and coaches have changed substantially over that time.

This may seem counterintuitive. "You would think the opposite, right?" Blandino said.

But Cochran suggested that teams that commit those game-changing penalties in the fourth quarter and overtime could be rewarded for more aggressive play. "Maybe that is generally a winning strategy," he wrote in an email.

Teams could incur offensive pass interference penalties near the end of the game "because they are losing and they are desperate to try to make a big play to come back in the game," said Warren Sharp, an NFL analytics expert who runs a betting information site.

It's more likely that the team that is behind near the end of the game is throwing the ball, and the team that is ahead is then more inclined to commit a defensive pass interference or defensive holding to prevent the offensive team from scoring, according to NFL officiating analyst Joe Gibbs (no, not the Hall of Fame coach), who contributes to Sharp's site. Whatever position or yardage lost is worth it.

Blandino pointed to the successful Seattle Seahawks teams of the 2010s, who had among the highest volume of fouls in the defensive backfield, such as illegal contact and defensive holding, "because they played more aggressively."

"There's definitely times when you're playing the clock more than you're playing anything else in certain games," Sharp said.

"If you can commit a penalty that prevents the other team from having a bigger explosive play ... they don't put the time back on the clock. So they go downfield for a 10-second play, with a minute left in the game, and you're tackling those guys so they can't get a big 20- to 30-yard gain," he said. "Ten seconds run off the clock and now they get a first-and-10... but it's only 5 yards closer and it moves the ball from their 25 to their 30, you'd trade that all day."

Man and machine

In recent years, the NFL has continually expanded its use of replay assist, where officials in the stadium and at the league office monitor plays to help on-field officials make the right call and avoid stopping the game to review the play. Designed to decrease the time spent on debated calls, one of its main effects has been to reduce the number of times coaches challenge a call.

Since 2010, the number of coach challenges has dropped by about 47%, and while there hasn't been a lot of variability in their rate of reversals, the percentage has generally trended down. In 2024, 50 of the 133 coach challenges resulted in reversals.

That is exactly the trend the league wants to see, according to Mark Butterworth, NFL Vice President of Replay Training and Development.

"In a perfect world, we would have less replay challenges either from the booth or from teams because through assisting, we're cleaning up more and more and then keeping the game moving," Butterworth said. "It's that relentless pursuit, every play, to make sure we have it right. ... The hope would be that when coaches challenge they would have a lower reversal rate."

An NFL spokesperson told ESPN that the league used replay assist more than 300 times in the 2024 season. The increase in replay assist has not caused a significant decrease in replay challenges by NFL league officials in the booth, according to ESPN's analysis. But their success rate has improved dramatically, from 25% in 2010 to 76% in 2024. The league attributes that increase in part to better camera technology and a growing number of reviewable plays.

After expanding the list of fouls and plays that can be reviewed by replay assist in 2021 and 2024, NFL owners voted again in April to include fouls such as grabbing a face mask, horse-collar tackles and tripping.

Butterworth said 90% of replay assist interventions are for line to gain for both first downs and the goal line, catch/no catch calls, down by contact, and flag pickups -- when officials negate a penalty. He suspects flag pickup interventions will increase this season with the additional plays and fouls that can be reviewed.

Green Bay Packers head coach Matt LaFleur supported the increased usage of replay assist.

"I know you guys cringe every time I pull that red flag out, so the less I have to do that, the better I guess," he said.

Washington Commanders head coach Dan Quinn said he believed the increased use of replay assist would decrease the number of challenges. "It will speed up that process and will also help a lot with spotting of the football," he said.

Replay assists are not consistently noted in Game Books, and while Butterworth said the league is working to better track them, it does not plan to publicly release detailed statistics. However, he said replay assist interventions will be noted more prominently during broadcasts so people know when they are being used.

Blandino said he welcomed the increased information. "The more transparency you have, I think the less these conspiracy theories live, and I think there is less angst and negativity around officiating," he said, adding that replay assist also takes away some of the subjectivity.

"I always told people, I said, you're giving us way too much credit. To think there was a coordinated effort with all these people and everybody on the same page to be able to say, 'We want the Patriots and the Giants in the Super Bowl' ... it's impossible to create a scheme that is going to allow for that."

However, some fans have speculated that replay assist could be used to fix games or to help a favored team win, Sharp said. "But if we're watching it on TV and it's pretty clear that this shouldn't have been called, I don't care if they fix that in the course of action and don't go to the sideline to review it," he said. "Let's just keep the game going, but get the call right."

Broncos coach Sean Payton would go even further with making the decisions known. "Why not put the video on the process, let fans, everybody, hear the discussion, open the windows? What's the downside?"

Throw the flag, coach

The worst challenge San Francisco 49ers coach Kyle Shanahan said he ever made was against the Indianapolis Colts in the pouring rain at Levi's Stadium in October 2021. Colts running back Jonathan Taylor rushed for 3 yards on fourth-and-1 but appeared to fumble the ball. Shanahan threw the flag.

As he did, an assistant pointed out that another Colts player had recovered the ball. Shanahan tried to call it back, but the officials told him they had to play it out. "It was just two minutes of embarrassment," he said.

Despite that call, Shanahan is among the more successful current coaches when it comes to challenges, with a nearly 50% success rate, according to ESPN's analysis. Those with under four years of NFL head coaching experience were excluded because of the lack of data.

Going back to 2010 in ESPN's database, individual coaches showed patterns over their years in the league, in terms of the frequency of their challenges as well as their success rate.

Shanahan told ESPN that challenges are sometimes "educated guesses" and his success rate is due to "better camera angles" and not letting "your emotions get involved."

Quinn remembered challenging an incomplete pass ruling against the New York Giants last season in Week 9. He won the challenge, and the call was reversed and changed from an incomplete pass to a fumble that was recovered by the Commanders, who took possession and drove to their first touchdown of the game. They went on to win 27-22.

"This was a great example of our process working well and our players playing through the entire play in order to give us a chance to successfully win the challenge," Quinn told ESPN.

Payton said he thinks there are two types of challenges: Ones where there is great video and others where he goes by instinct because of a lack of clear evidence. "I'll hear 'not enough' from the booth," he said. "But I'm becoming more educated and more aware of even if I get it right, does it justify a challenge in that moment?"

LaFleur had a simple answer when asked if he regrets any challenges.

"Yeah, every one you don't win."

NFL reporters Robert Demovsky, Jeffrey Legwold, John Keim, Nick Wagoner and Kevin Seifert, and researcher John Mastroberardino contributed to this story.