On Monday, I shared the results of my WARP (Wins Above Replacement Player) projections for the prospects in this year's NBA draft. Today, I'm taking a step back to explain more details about the process and look at what the same ratings said about past drafts.
Translated stats
I've been working on college stat translations for a decade now, and they've appeared as projections in the "Pro Basketball Prospectus" annuals. In theory, the method is simple. I take 14 statistical categories that sum up the box score, all of them pace-adjusted, and look at how much players drop off between the NCAA and the NBA. Players tend to see the biggest drop in their block rate (down by more than half), while they actually commit fewer turnovers at the NBA level because they tend to handle the ball less frequently.
There's one important additional layer, and that's accounting for strength of schedule (measured by KenPom.com's SOS) to ensure that players from small colleges aren't overrated. Schedule has historically been most important for translating usage rate and free throw rate, two places where players from small schools see much bigger declines than their major-conference counterparts. A smaller adjustment is applied to all other stats.
While the basics remain the same, in the course of preparing for this year's draft projections I overhauled the translations, including using multiple years of college stats rather than just the most recent season. Here were a few key findings:
• The game has changed: Because both the NBA and NCAA games are constantly evolving, it turns out that using 2000-07 multipliers to build the translations was creating problems for modern players. These mostly relate to the 3-pointer. Since the NCAA moved the 3-point line back, players have lost less of their 3-point percentage coming to the NBA. And with NBA teams shooting more 3s, college players are shooting free throws and drawing fouls less frequently relative to their college performance. To solve this problem, I now base the multipliers on the previous five seasons, weighted more heavily to recent years.
• Small schools, big game: Some of the players most underrated by the translations in recent years were products of the mid-majors such as Stephen Curry and Damian Lillard. When I updated the database, I had to shrink the strength of schedule adjustment to improve accuracy.
• Early performance matters: When I began incorporating multiple seasons, I anticipated that the most recent campaign would predict NBA performance best. Instead, I found the opposite. In the past, Per Diem predecessor John Hollinger noted that players coming off a down season tended to do better than expected, but I found the effect both ways. How prospects perform at a young age appears more telling than whether they dominate younger opponents as upperclassmen.
Together, the changes doubled the amount of variation in rookie-year performance predicted by college translations among players from the past four seasons while simultaneously improving the predictive power of previous translations.
WARP projections
To put together the rookie projections and age, I use a regression involving all the players in my database, which features some players going all the way back to the 2003 draft and everyone from the 2006 draft onward.
These factors are regressed against players' WARP averages over their first five seasons, adjusted to weight early performance more heavily. Future seasons are discounted by a factor of 10 percent per year, which makes it slightly easier to compare players at different stages of their careers and rewards immediate contributions.
Age ends up making up about 40 percent of the final WARP projection, with translated performance accounting for the other 60 percent.
Past draft boards
Let's take a look at how the updated version of the system has rated past drafts. To be clear, this is not an indicator of how well WARP projections will assess this year's draft. Although my method makes it difficult to overfit the data, I'm still building the model around how these players have actually performed, meaning the system knows the results when it guesses how they will do.
That said, looking at past drafts can give an idea of what the WARP projections look for in prospects and the range of typical scores. I've also included players' actual five-year WARP totals (again, adjusted to discount future performance) to give an idea of how close the projections were. For each draft, I've also highlighted a pair of players whose stats were better indicators than where they were taken and two more the scouts got right.
2012 WARP Projections
Hits: Austin Rivers, Meyers Leonard
Two lottery picks were projected for negative WARP over their first five seasons. There's still a ways to go, but both Leonard and particularly Rivers struggled as rookies. Additionally, Thomas Robinson's middling translation nearly exactly pegged his stats last season.
Misses: Jae Crowder, Draymond Green
Crowder and Green contributed as rookies, but -- Green's playoff run aside -- weren't the instant successes their numbers indicated. Harrison Barnes may eventually find his way on this list, but his advanced stats during the regular season were no better than his pessimistic translation suggested.
2011 WARP Projections
Hits: Kenneth Faried, Kawhi Leonard
Reducing the schedule adjustment bumped Faried to the top of this list. (Kyrie Irving was great when he played at Duke, but the injury that limited his minutes hurt his projection slightly.) That's a lot closer to where he would go if we redrafted than 22nd overall. Leonard, a top-10 pick by the stats, dropped to 15th to the benefit of San Antonio and detriment of the rest of the Western Conference.
Misses: Jimmer Fredette, Chandler Parsons
While the adjustments helped overall, they also turned Fredette into a top-10 prospect, which he's yet to demonstrate in the NBA. Parsons and fellow second-round steal Isaiah Thomas have both outplayed their projections.
2010 WARP Projections
Hits: Evan Turner, Wesley Johnson
Consider Turner and Johnson exhibits 1 and 1A when it comes to the importance of succeeding early in college. Both players developed into lottery picks late, but have disappointed in the NBA.
Misses: John Wall, Hassan Whiteside
John Calipari point guards have proven tough to evaluate statistically, but John Hollinger's Draft Rater did have Wall third, just after Turner. Of the run of busts near the top of the rankings, which also includes Cole Aldrich and Xavier Henry, Whiteside was the one correctly figured out by scouts.
2009 WARP Projections
Hits: Stephen Curry, Ty Lawson
Had David Kahn been using the current WARP projections, he surely would have taken Curry and maybe even held on to the rights to Lawson, drafted with a pick that originally belonged to the Timberwolves. Oh, and he might have considered Danny Green in the second round.
Misses: Nick Calathes, Taj Gibson
The other PG drafted by Minnesota (as part of a prearranged trade with Dallas) was a numbers favorite, but he has yet to arrive in the NBA after a solid but unspectacular career in Europe. Gibson is the ultimate outlier when it comes to the importance of age. Nearly 24 at the time he was drafted, Gibson has continued to develop with the Bulls.
2008 WARP Projections
Hits: Ryan Anderson, Mario Chalmers
Anderson and Chalmers correctly rated as two of the draft's steals. From the other perspective, Joe Alexander's college numbers revealed him as nothing more than workout hype.
Misses: Derrick Rose, Russell Westbrook
J.J. Hickson was the only prospect badly overrated by the stats, and even he has posted decent individual numbers while struggling defensively. However, several young guards exceeded poor projections. Rose falls under the aforementioned Calipari exception, while Westbrook looked like nothing more than a defensive specialist before evolving into a dominant scorer in Oklahoma City.
2007 WARP Projections
Hits: Acie Law, Rodney Stuckey
An older prospect who came on late in his career, Law never could find a home in the NBA. Stuckey, who put up good numbers against inferior competition in the Big Sky conference, proved the better combo guard.
Misses: Sean Williams, Aaron Brooks
The same kinds of off-court issues that derailed Williams' college career also prevented him from sticking in the NBA. The analytical Rockets successfully went against the numbers in taking Brooks, another four-year player who boosted his draft stock as a senior.
2006 WARP Projections
Hits: Paul Millsap, Adam Morrison
Along with former Utah teammate Carlos Boozer, Millsap is one of the great triumphs for statistical scouting. Considered undersized for a power forward, Millsap has developed into a quality starter at the position. The numbers also picked out Rajon Rondo as undervalued. At the top of the draft, the one-dimensional Morrison (and Randy Foye) was doomed from the start.
Misses: Patrick O'Bryant, Brandon Roy
Sure, Tyrus Thomas as the No. 1 prospect (a spot he also held in the original version of Hollinger's Draft Rater) looks bad, but Thomas was at least a useful player in Chicago. O'Bryant rated below replacement level. Roy, meanwhile, was the rare four-year collegian who made massive improvement as a pro before knee problems prematurely ended his career.