As the main source of NHL player acquisition, the NHL draft is a huge and often debated area of the game and -- as we discussed recently -- the most critical source of good players in the salary cap era.
People often discuss hits and misses in an NHL team's draft record: finding that gem in the fourth round, striking out on a top-10 pick, etc. As we head to the 2015 NHL draft, where many teams' futures will be heavily influenced, this column asked one important question: Can individual NHL teams show skill in being \good or bad at the NHL draft? The answer is yes; skill at the draft is about 25 percent of the results. But for the most part, the outcome is not due to the team selecting. This column explains why.
The method
Typically when we ask which are the best- and worst-drafting NHL teams, the question is answered by looking at statistics like total games played or total points scored by a team's draft picks.
The method employed here differs from this line of thinking because we don't ask for the raw output of a team's draft picks. Rather, we ask: Given the draft position of a particular pick, what was the expected output of such pick, and did the selected player go above or below that benchmark? With this data, we will ask how much of any observed difference from the expected benchmark is due to team-scouting skill or randomness.
The statistic used to gauge performance is NHL games played. Depending on where a player ranked among his draft class in games played, he is given a particular "draft results" number based on that, which is then subtracted from the draft results number a player selected at his slot would get. The most games played gets a draft result number of 20, fourth-most games played gets a value of 10, the 50th gets a value of three, etc. This is formed from a basic curve we expect based on most studies on the NHL draft.
For example, Scott Gomez has been top-10 in games played from the 1999 NHL draft class, but his 27th overall selection usually plays about half the NHL games he does. That provides twice as much value to the Devils from that draft pick. This method is then employed for every draft pick selected from 1991-2008. 1991 was arbitrarily chosen, and 2008 was chosen to end because the next draft had no players with more than 500 games played. To keep the data uniform over different collective bargaining agreements, only the top 210 picks in every draft class were considered.
One expected counter to this particular method is that it will lead to some misleading value numbers for some drafts. I am sure fans can dig up drafts where great players got hurt for a few years, or didn't play their best games for their teams, or were called up late, etc. We don't claim that the method is perfect in every instance, but it passes the smell test in most instances.
NHL teams don't draft well or poorly consistently
The first question I asked with this data was whether NHL teams showed any persistent skill at drafting. The force of the fancy-stats movement in hockey was around this type of question, when we asked whether teams persistently outshot attempted teams, or finished chances at a high percentage. The former was yes; the latter was no.
When it comes to the NHL draft, while teams on a performance level have separated themselves over a 20-year span, the level of year-to-year performance in terms of consistency has been very low. "Performance" in this context is the total surplus or negative value they got from their draft picks in one draft relative to what an average team would get with the very same draft picks.
Very few teams actually showed significant consistency by this metric from 1991-2008. One of the very few that did was the Dallas and the Minnesota North Stars franchise. Detroit is shown below as an example of a team that was consistently inconsistent -- similar to almost every other NHL team.
You can see that in Dallas, there is grouping in the right and upper portions of the graph, representing consistent positive drafting value. In Detroit, there are a roughly equal number of data points in each of the four quadrants. Keep in mind how hard it is to get two straight good drafts. In Detroit's case, the Zetterberg draft was considered very positive, but the Datsyuk one was negative. The Datsyuk pick obviously provided a ton of value, but because Detroit got negative value on the 25th, 55th and 56th selections -- and no NHL games from several other picks -- the team ended up with an overall negative draft. The NHL as a whole from 1991 to 2008 followed this pattern.
What this shows is that over a nearly 20-year span, NHL teams showed no ability to repeat their performance, either good or bad, in terms of the prospects they drafted relative to their draft position gauged by NHL games played.
Like clockwork, most of the great drafts, such as when a team selected a top-10 player from the draft class and found a midround gem, tend to be followed by a mediocre draft. The poor drafts, where teams struck out horribly and maybe got one NHL player who did OK over his career, were usually followed by a mediocre draft.
This does not mean that the draft is random and anybody could do it. What I'm saying is that among experienced, trained NHL evaluators, there was no identifiable difference at the NHL draft in terms of repeating draft performances the next year.
However, examining the long run is a different story.
Ranking NHL draft records and gauging scouting skill
While the repeatability numbers presented above show a gloomy picture of the distribution of NHL scouting skill, there was actually a decent spread over the nearly 20-year span. Moreover, when you adjust for placement in the standings, more skill shows up. That is because the most unpredictability happens at the very top of the NHL draft, where teams either get the best players or tend to have the worst drafts on record. Since those top-pick opportunities are not evenly distributed, we need to create a level playing field to account for that.
Upon making this adjustment, we get a draft score that shows how well a team has picked from 1991 to 2008 given the expected value of all of its draft picks, and adjusted for its finish in the standings.
Now, although we have a clear ranking of teams with a spread of results, some of it is due to chance. Let me try to explain with an example. If 30 people took a fair coin and flipped it 100 times each, you would not expect 50 percent heads for every person. Instead you'll get a bell-curve distribution of results because of randomness. Any deviation from that would indicate particular skill, which we expect none at flipping coins.
We thus can learn through the variance in the NHL teams' draft performance, versus what was expected, how much of the draft results are due to the teams picking. The answer is 25 percent.
That means 25 percent of the draft results in terms of performing better or worse than the average team would at a particular draft slot. The other 75 percent is due to randomness.
The takeaway points
The NHL draft is a highly debated area of hockey, especially around this time of year, where fans, media and scouts go back and forth on who they think the best prospects are and who would be a great or horrible decision at slot X.
What is not often discussed, however, is exactly how much the skills of the evaluator actually matter. In the grand scheme of things, the evaluator's abilities account for only 25 percent of the total picture.
This is not due to the fault of particular NHL personnel. Most teams, full of trained expert evaluators, tend to have roughly a similar top 30-50 draft board, and yet the top 20 players in a draft class are rarely those who all go in the top 30. Some tend to point to certain teams being better at developing players, and although that certainly matters, the reality is the external forces that play on a draft-decision result are massive.
Players can be affected by injury, huge unexpected leaps forward or backward in development, or non-hockey factors. I always make the analogy to look at a draft ranking from September to the final one in June to see the variance at play in just an eight-month span. Now multiply that by years.
When I look back on my previous draft rankings, these effects are so apparent to me. There are countless examples of guys I think I rated too low at the time and ended up not panning out, or rated too high and they ended up being very good players.
Just because the draft is extremely random does not mean a scout is not valuable -- players just need their value put into context. Think about how much a good coach is worth. We know a coach who runs an NHL team will be subjected to tons of randomness. In fact, 40 percent of NHL game results are due to such. You can make the best game plans, have the best execution with the right line matching, but with one bad bounce -- you lose.
Over the long run, if you look at the above draft scores, good scouting clearly does matter. Heck, if Atlanta weren't so bad at it, there might still be a team there.
When looking at individual years and especially picks, though, I urge the hockey community to keep this in context. Over the long run, we see true evaluation skill, but year to year and pick to pick, most of what we observe is just due to random variation. During these past playoffs, we all heard talk about Chicago picking Brandon Saad or Tampa Bay's selection of Ondrej Palat. While those success stories may be due in part to the result of a progressive, analytical process and a ton of hard work, more likely than not they are just due to chance. This applies on the flip side of criticism to, say, Edmonton, and how some of its top picks, such as Nail Yakupov and Magnus Paajarvi, have not lived up to expectations. NHL evaluators have only so much control over the progression of teenagers.
Keep in mind as well, as we enter the 2015 NHL draft, to avoid any broad proclamations of success or failure. History has shown that the result could go in any direction, most of which we can't predict the week of the draft anyway. While the draft is incredibly important, it is also incredibly unpredictable. Sometimes it just pays to be lucky.
Special thanks to professor A.C. Thomas at the University of Florida and ESPN contributor Tom Awad for their help with this column.