Each year the NFL scouting combine offers troves of rigorously measured data on prospects -- data we would expect really matters in terms of how good a pro athlete can become. Certainly being faster, stronger and more agile than your competition makes it more likely that a player will succeed in the pros. Given the importance of selecting the right players, and with all the advances in sports analytics, you might think this problem would be one of the first and easiest to be solved. So it may surprise you just how difficult it is to find meaningful connections between combine performance and success in the pros.
Many attempts have been made to discover a formula for the relationship between combine data and player success, and they consistently return the same results, finding very little relevance in the combine measurements. Perhaps the most surprising result is that the combine drills that are seemingly most similar to what a position does on the field -- like the shuttle drill for running backs or the bench press for offensive linemen -- often have the least connection to career outcomes. Of the many analyses I've come across in 10 years of crunching football numbers, the first analysis to find any meaningful connection uses advanced methods with modest success. We've recently been able to improve upon that research here at ESPN, but not without a great deal of effort. It's much harder to detect the signal among the noise than you might expect.
The tremendous difficulty of finding a connection between athletic abilities and performance in the NFL is something of a mystery. We should naturally expect better athletes to become, well ... better pro athletes. After all, if you and I and other random fans were invited to the combine, we'd likely do quite poorly at the drills. And our chances of making an NFL roster would be accordingly very poor, so certainly there must be a strong underlying link between athleticism and on-field performance.
But that's just it! You and I aren't invited to the combine. That's the key to unlocking the mystery.
To be invited to the combine, a prospect must have measurable talent, nonmeasurable skills, or likely some combination of both, and must be strong enough to play well in college and catch the eye of NFL scouts. Measurable talent includes things like speed, strength or agility, while the nonmeasurable skills involve things like technique, work ethic, ability to read keys and anticipate plays, or the ability to learn the playbook. When I hear scouts and analysts using trite terms like "football IQ," "vision" or "ball skills," they're most likely referring to the vague nonmeasurable part of a player's set of attributes.
The fact that a prospect is invited creates a selection bias in the combine results. The guys with relatively low measurables tend to have higher nonmeasurables, or else they likely wouldn't have been invited. And the guys with low nonmeasurables tend to have high measurables for the same reason. Measurable talent and nonmeasurable skills tend to be cross-wired, given the fact that someone has been invited to the combine. Once players have been drafted and enter the league, the low-measurable ones will tend to outperform their combine stats while the high-measureable ones will tend to underperform them. This phenomenon is why we don't see a clear correlation between measurable combine performance and pro success.
This diagram may help explain what I mean. It's symbolic of the underlying concept, and not meant to depict exact numbers. (They're called "nonmeasurables" for a reason.) The horizontal axis represents measurable (combine) talent, and the vertical axis represents nonmeasurable skills. To be invited to the combine, a player must have some combination of both types of abilities, which is depicted by the white area. Prospects with low measurables (blue) tend to have higher nonmeasurables, while the prospects with high measurables (red) tend to have lower nonmeasurables.
In statistics, this phenomenon is known as Berkson's paradox, but I tend to think of it as the Boldin paradox, as Anquan Boldin might be the epitome of high nonmeasurables and low measurables. Boldin's 40-yard dash was clocked at 4.72, dead last for wide receivers at the 2003 combine, yet he continues a fantastically successful career to this day. Among his 2003 classmates, only Andre Johnson's career production exceeds Boldin's. Boldin compensates for his lesser speed with his renowned route-running and pass-catching skills -- his nonmeasurable (but observable) skills. It's likely why he can continue to play the receiver position well into his mid-30s. His nonmeasurable skills don't diminish with age nearly as quickly as his measurable talents, especially speed.
One common manifestation of Berkson's paradox is in the dating pool. Suppose you'd date someone who has only some minimal combination of being attractive and having a nice personality. The paradox helps explain why the ones with the best personalities don't appear so attractive, and the attractive ones don't always seem so pleasant.
So when poring over the combine results, don't get too excited about the blazing fast numbers some players put up. Chances are their skills aren't as strong as the players with more modest numbers. And the receiver who ran a 4.7 40-yard dash might just have the best hands and run the best routes in a generation. After all, there's a reason he was invited.
For more from ESPN Analytics, visit the ESPN Analytics Index.