Goaltending evaluation has been a troubling area in the world of hockey scouting and analytics. Even with the major advances we've made in statistical analysis, the goalie position holds a divisive and often misunderstood place in the game, as there are issues in the way they're scouted, to the way they are analyzed numerically.
But there are some who believe we have the tools to properly evaluate and scout goalies, or that this could happen in the near future. And there is a significant reward to those who master this realm: There is no position that makes a bigger impact on a hockey team's success than the goalie. So finding (and retaining) the right one can be a pivotal landmark in a franchise's history.
Where do things stand now, and what happens next?
Scouting goalies vs. scouting skaters
When it comes to scouting goaltenders, the complexities of their position is evident. "So much of it is mental -- and technical," one NHL scout said in discussing the difficulties of scouting the position. "[Goalies] are so hard to project," another scout said. "There are so many variables and the history of results speaks for itself."
When goalies come up in conversations with hockey people, it is almost like speaking a different language. There is also a striking difference among evaluators in this area.
According to top executives, or even an area scout, the focus of the goalie's scouting report is on the core fundamentals. They determine the strength of the prospect based on attributes like athleticism, hockey sense, reflexes and competitiveness, among other generic characteristics. On the other hand, a goalie scout or a goalie coach will have an entirely different viewpoint.
Goalie scouts will spot details in a goalie's game that even a seasoned hockey evaluator might miss, such as the way one of his leg moves on certain shots or how he twists his upper body. It's apparent there are two sports being played simultaneously: skater hockey and goalie hockey.
Issues in evaluations can stem from the fact the position creates a fundamentally different game for a goalie than the one in which the skaters participate. You'd be surprised at how often some experienced, high-level evaluator's main point of analysis comes down to: "Did he stop the puck?"
Sometimes ignorance is just admitted. "I don't do goalies," said one NHL executive. "That's what we have our goalie scout for."
Another executive discussed his team's draft board process noting how they will build a board of just skaters, and will decide where to prioritize the goalie prospects based on how much their goalie scout is advocating for particular players.
Some disagree with that approach. "I don't think they're harder to scout. There are only 30 starters in the NHL, so there's less opportunity, and thus it takes longer," one NHL scout said. "GMs don't want to take goalies, and for scouts, it's easier to fight for a skater."
While there is a little more variance than for skaters, the draft table isn't the Wild West with regard to goaltenders either, since the highest regarded players in an age group tend to be selected first. For example, the two top European goalies in the 2015 NHL draft -- Ilya Samsonov (22nd overall, by Washington) and Felix Sandstrom (70th overall, by Philadelphia) -- have been among the most impressive 18-year-olds in the KHL and SHL, respectively, in recent memory to start this season.
There is usually a moderately tight consensus leading into every draft season about who the top goalie prospects are, but once they get closer to (or reach) the NHL level, the evaluation becomes splintered. At that point, there is a much higher failure rate for top prospects among goaltenders than for skaters. One could see the argument that goalies making the leap to the NHL are a tougher climb than the other positions, or one could critique the original evaluations.
What makes good goalies good?
Goalie statistics have always suffered from being too limiting. With teams and skaters, we use shot metrics as a process stat, because over time they predict the output stats (goals, and wins) better than using the result stats as the predictors. With goalies, we have stuck with save percentage, a results stat, and tinkered with it. We have used things like even-strength save percentage and penalty-kill save percentage, but we haven't been able to extract the question: What makes good goalies good?
A common argument is to adjust for shot quality. The term has been used very often during the past decade during the hockey analytics movement. Most analysis has shown its effect to be minimal. In "Hockey Abstract 2014," Tom Awad took an in-depth look at this issue. He showed that while shot quality is somewhat stable for teams from season to season, it is a very small fraction of a team's success.
There isn't a ton of deviation on shot location difficulty per shot across teams. Rather, the quantity of shots, which is spread widely across teams, increases at the same proportional rate for low- and high-danger shots.
One NHL analytics consultant said they've found at the Canadian junior and NCAA levels that shot quality is somewhat more steady -- but still not a gigantic issue. Shot quality is either a narrative for randomness from watching a goalie in a short sample, or a mirage from seeing a guy getting barraged from all locations at a high frequency.
While skater analysis has determined scouts should be looking for possession-based plays -- skills that can gain the offensive zone or prevent defensive zone entries -- there hasn't been a similar development for goalies.
The most common proposal to the statistical issues with goalies is to look at save percentage in the "danger area." However, the nature of save percentage is one of extreme randomness. Former NHL analytics consultant Gabriel Desjardins has discussed how you need up to four seasons of data for even-strength save percentage to normalize. It should be no surprise when you slice it down into "dangerous" shots, the sample leaves almost no relationship season to season.
Chris Boucher from Sportslogiq, a new analytics company using video tracking technology, feels that in the near future "the cream will rise to the top" when we're able to isolate things like tough saves and dangerous rebounds allowed. There have been many others who have approached this issue similarly, and feel that goalie skill is measurable through looking at what kind of saves they're making.
There are enough very smart hockey people who believe a solution is either here (or not far away) to being able to put a number on a goalie. These debates can dodge the big issue, however.
Are goalies worth all of this trouble?
While every position in every sport has its challenges with regard to evaluation, goaltenders are particularly interesting for a different reason. They are extremely valuable game to game, yet there are big questions about how much should be invested into their evaluations. This is primarily due to how long it takes for their talent level to be evident.
The usual argument to adjust for a goalie's statistical issues will be to increase the sample size to several seasons. But that's a massive issue in an evaluation -- especially when you have to particularize -- and at the amateur level. That period of time is often something teams don't have, and you don't get a cookie for saying Tuukka Rask is a good goaltender in a game tomorrow. Sure there are situations where there is some uncertainty and a large enough sample, but it still leaves out many other situations.
Blueshirt Banter's Nick Mercadante showed at a recent analytics conference that most goalies don't last in the league for three seasons, but then after that they tend to perform at a quality level. Desjardins showed decisions on goalies typically lag, with a small hot stretch buying the goalie an extra season to be not good before the cord is cut. This all shows how long it takes to get a read on a goalie. If you get that read eventually, more power to you, but for restricted free agents, prospects or guys who get called up in their mid-20s, that's not very reassuring.
Furthermore, the extreme variance season to season is the nature of the position. If a goalie stops 90 out of 100 pucks, he's a backup or AHLer; if he stops 91, he's roughly NHL average; if he stops 92, he is an All-Star, and 93, a Hall of Famer.
Despite the shortcomings of getting a great read on them statistically, the reason why goalies continue to be valued so highly is that no position has a greater impact on the game. If you look at any version of wins above replacement across positions in hockey, the top of every list is littered with goaltenders. While they only produce at one end of the rink, they are the only player on the ice for 60 minutes, so their impact is massive. The up and downs of the 2014-15 Minnesota Wild -- before and after they traded for Devan Dubnyk -- is Example A of this effect.
Views are divided across the industry on whether goalies are worth all the trouble.
"What's the point?" said one NHL analytics consultant, "It takes too long for a goalie's talent level to even out to care too much about who the best goalies are."
Another consultant was a little more extreme, stating, "Teams should just fire all their goalie scouts, the results probably won't change at all. Predicting goalies is voodoo."
However, there is a camp within the NHL that feels predicting goalies' future performance is possible. They will point to multivariable models, and looking at very specific parts of a goaltender's game. Scouts will point to success stories, and say that it's not impossible, just harder.
"As long as you've done your homework, your scouting reports are accurate, and you're not guessing, then there's nothing else you can do," one NHL scout said.
The bottom line
There are a lot of resources spent trying to determine the merits of particular goaltenders at various levels of the game. In both the scouting and statistical realm, we have real, significant issues with how goaltenders are evaluated.
These issues are important, because goalies are extremely important, at least on a results level. However, their payoff isn't always evident for a long time.
"Even if you somehow found out who, on true talent, the best goalies are, they may not produce at that level for years," an NHL consultant noted.
Given how poorly everyone across the industry have done for years at projecting this position, I think there are fair questions to ask in terms of how we scout and quantify goalie performance at any level. We don't have all the answers yet, and those answers could prove quite valuable to whomever figures them out.