Scouting vs. statistical analysis

Mock Draft 3.0 | Big Board 6.0 | Top 100

ESPN Insider's Chad Ford and Kevin Pelton return to provide the kind of discussions that are happening in front offices around the NBA -- where scouts and statistical experts are breaking down NBA draft prospects using their "eyes, ears and numbers."

Kevin Pelton: This week, we're going to do something slightly different from our usual dialogue about individual draft prospects. Instead, with the annual MIT Sloan Sports Analytics Conference beginning Friday in Boston, I'd like to take a step back to explain why both scouting and statistical analysis are helpful in projecting prospects to the NBA.

In his book "The Signal and the Noise," Nate Silver of FiveThirtyEight.com put PECOTA, his baseball projection system, to the test and found that it was not as effective at predicting major league success for young prospects as the scouting consensus. It's interesting to take a similar look at the NBA draft. For drafts between the introduction of the age limit in 2006 (before which many top prospects came out of high school with few if any meaningful stats) and 2011 (the most recent draft where players are in their fourth year or later), I looked at three groups of players:

• Those drafted in the top 10 who also were rated in the top 10 of my WARP projections (the consensus)

• Those drafted in the top 10 who were outside the top 10 of my WARP projections (scouting favorites)

• Those in the top 10 of my WARP projections who were drafted outside the top 10 (stat favorites)

Let's take a look at each of those groups. First up, the scouting favorites, along with their performance over the first four seasons of their career (the life of a rookie contract for first-round picks):

Chad, what stands out about this group?

Chad Ford: Well, it isn't pretty. With the exception of Russell Westbrook and possibly Brandon Knight (who strongly outperformed their statistical prediction), when the gap has been 10 points or more between where teams drafted and what analytics predicted, the stats were much better at predicting a player's NBA potential than the scouts.

I've always believed, based on the pre-draft analytics I've consulted, that stats were a better predictor of failure in the NBA than they were success. If analytics say you'll struggle in the NBA, they're usually right. Look at the guys drafted in the top 10 who were ranked 25th or above on Kevin's board -- Foye, Mayo, Alexander, Vesely, Udoh, Johnson and DeRozan. DeRozan has been solid (though his minus-9.0 WARP score indicates he was inefficient early in his career), and Mayo has been solid at times as well. However, Alexander, Vesely, Udoh and Johnson have all been major busts compared to where they were drafted. Foye has been a decent role player, but never lived up to where he was drafted.

Westbrook is the major outlier here. He was drafted fourth and was ranked just 21st by Kevin. But I think there's an easy explanation to why the stats got him wrong: Those WARP projections heavily favor players based on their freshman season. Westbrook played sparingly and averaged just 3.4 points per game. He wasn't exactly a dominant scorer as a sophomore either, averaging 12.7 PPG. Westbrook was a major gamble at No. 4 -- a classic upside guy who was drafted more on his physical abilities than on his offensive performance at UCLA over his two seasons in Westwood. I'm not sure anyone, including me, thought he'd be this dominant offensively after his first two years in the NBA. The Thunder were right -- he's an MVP candidate now. But at the time of the draft, everyone knew they were rolling the dice.

Pelton: Next up, the players who rated better statistically than in scouts' eyes.

Ford: Again, the stats did a better job than the scouts in most instances. There are some major misses here: DeJuan Blair was ranked No. 1, Mario Chalmers at No. 5, Cole Aldrich at No. 4, Justin Williams at No. 9 and Xavier Henry at No. 6. But of this group, only Henry, Aldrich, Brewer, Justin Williams and Sean Williams have been major busts compared to where they were drafted.

And look at the finds: Rajon Rondo, Kawhi Leonard, Ryan Anderson, Roy Hibbert, Paul Millsap, Kenneth Faried, Hassan Whiteside and Danny Green.

I will say, however, that for several players on this list, there were mitigating factors that forced them lower on draft night. Rondo had a well-deserved reputation of being difficult to coach. While he was regarded as a top-10 player going into draft night, teams were scared away by the background checks they had done. Essentially his head coach, Tubby Smith, warned teams he'd be difficult (he was right). Leonard had an injury coming into the draft that scared some teams away from drafting him high, though like Rondo he was considered a top-10 prospect before the draft. Millsap showed up to the NBA draft combine overweight and out of shape and struggled there. Faried had an asthma problem that made scouts question whether he could play 30 minutes a night.

All of those factors couldn't be quantified by statistics alone and, at times, the extra background scouting helped. Williams was a major knucklehead coming out of Boston College and flamed out in the NBA. So was Whiteside until this season; the talent was extraordinary, but 10 minutes into every interview he did with NBA teams was enough to scare everyone away. He's finally matured, but it took him five years to get there.

Pelton: Lastly, here are the consensus top prospects.

Ford: Fran Fraschilla has this saying I love: "Statistics accuse, analytics indict and videotape convicts." When box-score stats, advanced analytics and traditional scouting all agree, you're usually pretty safe either drafting a player or letting them go.

On this list are a host of All-Stars on whom both analytics and scouts agreed: Durant, Curry, Love, Harden, Cousins, Horford, Wall, Noah, Griffin, Irving and Conley. Those guys were the easy ones.

What is really interesting are the rare cases when both get it wrong. Usually there's an explanation that transcends scouting and analytics. These are human beings and their humanness can get in the way. Beasley, O'Bryant and Thomas were knuckleheads. The talent was there, but they were never committed to developing it. Injuries derailed the careers of Oden, Rose, Gallinari and Lopez. When they were healthy, they clearly belonged.

Three of the last four on the list are harder to quantify. Shelden Williams just never did any one thing well enough to stick. Derrick Williams was a tweener who never figured out a position. Fredette, an amazing scorer at BYU, may have been the victim of getting drafted by the wrong team -- if given a different coach, different offense and more minutes he may have been much better.

Pelton: Having looked at the individuals, let's compare the aggregate performance of these groups. The favorites of the scouts and the favorites of the stats end up rating remarkably similar. Both groups produced four All-Stars, and in Year 4 a similar percentage of players averaged at least 28 minutes per game (starters), 15 to 28 minutes (rotation) and fewer than 15 minutes (fringe). Both groups also had the same number of players (three apiece) out of the league by Year 4.

By comparison, the consensus group dominates. Discounting the injured Oden, 21 out of 29 players (72 percent) averaged at least 28 minutes in Year 4, and none were out of the league. The consensus group also produced 12 All-Stars, accounting for 40 percent of the players in this group.

The numbers (including the draft as a whole beyond the top 10) clearly back up your point: On their own, scouting and statistical analysis are both useful, but it's when we combine their strengths and weaknesses that they become overwhelmingly powerful.

Chad, in your conversations with talent evaluators, have you seen any changes in how they incorporate statistical analysis in the past five years or so as teams have beefed up their analytics departments? Will we see more players in the consensus group going forward and fewer favored by one method or the other?

Ford: As you noted in the Great Analytics Rankings this week, more and more front offices are incorporating analytics. However, I still believe they are widely misunderstood by both the public as well as many people in the NBA.

I think human nature is at fault here and the debate, too often, parallels similar arguments at the intersection between faith and science.

In almost every field, there is tension between those who believe that there is a formula that can explain everything (e.g. disease, war, economics, the origin of the universe, etc.) and those who believe that faith in its various forms is the only true explanation.

There is arrogance on both sides. Arrogance that our reason can uncover the mysteries of the universe (including the NBA draft). Arrogance that faith alone is the only avenue to providing answers. We both know "true believers" in analytics and "high priests" of scouting.

Both sides tend to ignore the data that says that just one explanation, one information stream alone, isn't as strong in predicting performance as multiple streams.

There are analytics folks who scoff at the idea, for example, that a seasoned scout has developed an instinct for finding talent. That scouts can "feel" when a player will or won't make it. That a "gut" feel from an experienced scout who knows basketball in their bones (through years of playing or scouting) is the best way to discover talent.

And we know that there are many scouts, ex-players and GMs who ridicule the idea that there are formulas out there that can see the game better than they can. They laugh at the idea that a game as complicated as basketball can be reduced to metrics by a bunch of nerds crunching numbers on their computers. Only people who have played the game at the highest levels can really understand the game, they argue.

The best teams (and they are out there) understand that those two paradigms can coexist together in a front office. That analytics and traditional scouting are sources of data that explain performance in ways that aren't always visible to the human eye. They give us clues, not necessarily answers, to why players may or may not be good NBA prospects that transcend traditional scouting. They each have strengths and weaknesses.

Teams that employ analytics know that they complement traditional scouting. They raise questions, sometimes give answers and sometimes force scouting departments to go back and re-watch tape. The best analytics people out there are relentless in watching tape, learning the game and collecting other sorts of data (background, psychological test, etc.) that complement the numbers. They learn to gather instincts about the game and trust them, even when the numbers don't always support them.

For teams that can give equal voices to both, the possibilities are endless. For those that insist that one way is the right way, they are playing the game with one hand willingly tied behind their back.

Pelton: Well said, Chad. Along those lines, I think sometimes people get confused that my statistical big board is exactly how I'd rank players for the draft, when really my assessments also factor in what I have seen and what professional scouts have seen -- how players compile their stats is important to projecting their future development, not to mention the off-court issues you've mentioned.

We can approximate this process statistically. Last season, draft analyst Layne Vashro introduced a "humble" rating that factored in the scouting consensus along with player statistics.

I did something similar with my WARP projections looking at results from the past decade. Using Chad's Big Board rankings in place of this year's draft slot -- which isn't known yet -- produces a consensus 2015 WARP projection that filters out some of the outliers by the numbers.

Four of the top five players are this year's consensus group, ranked in the top 10 by both stats and scouts. The other is Duke's Jahlil Okafor, whose relatively weak WARP projection is overcome by his spot atop the Big Board. The top of the consensus projections is close enough that any of Okafor, Karl-Anthony Towns or D'Angelo Russell could be No. 1 at season's end. How does this board look to you?

Ford: I've been saying there's a consensus top five and now I can shout it. Not only do most scouts agree, but so do the analytics! We have Okafor ranked No. 1, Towns ranked No. 2, Russell ranked No 4 and Porzingis ranked No. 5, so that fits perfectly. If we had a way to really gauge Emmanuel Mudiay statistically (apparently nine games in China isn't doing it), I think he'd fill out the top five. Turner and Looney are also in my top 10 and Portis is in my lottery. The only three who are a bit off are Kaminsky, Hunter and Jones. Kaminsky faces the problem of being a marginally athletic big man who's a senior. Scouts typically don't go nuts over players like that. Hunter would be in my top 10 if he could find his shooting stroke this season. I still think he reminds me a lot of Klay Thompson, but his shooting drought has hurt his stock. Jones, whom we talked about two weeks ago, faces a couple of key challenges; while his numbers scream star, his lack of elite size and athleticism yell "role player."