Note: This is the second of a two-part series on the NFL's issues evaluating quarterbacks, and how we can use those issues to better assess the top 2018 prospects. Go here to see Part 1.
We could come close to an NFL record this year before any players take a snap. There are five quarterbacks who could come off the board on Day 1 of the draft, which would tie the 1999 draft for the second most since the merger. The only draft to post six first-rounders is the legendary Class of 1983, which delivered a trio of Hall of Famers: John Elway, Jim Kelly and Dan Marino.
As much as the league seems to be struggling to pick between the prospects in this year's class, though, the coaches and executives of 1983 weren't able to separate the wheat from the chaff until well after the fact. Elway was the No. 1 pick, but the Chiefs still managed to draft Todd Blackledge seven picks before Kelly. Blackledge threw 29 career touchdowns. Kelly topped 29 in 1991 alone. Tony Eason was taken one pick after Kelly and 12 picks before Marino, who would post the greatest passing season in league history to that point during his second campaign.
A league full of coaches and personnel executives who had spent and would go on to spend the majority of their lives working in the game weren't able to pick between a trio of future Hall of Famers and two guys who would fail to make a single Pro Bowl. (Ken O'Brien, drafted after Eason and before Marino, at least made two Pro Bowls over his 10-year career.)
Thirty-five years later, I'm not entirely convinced we've gotten much better at evaluating quarterbacks. The league has access to more information than ever before, but the job has become tougher. A wider range of passing offenses at the collegiate level have made it more difficult for obstinate coaches to translate amateur success into bland professional schemes. Passers come better prepared for the pre-draft process than ever before and are far more selective about throwing at the combine.
As a result, the range of opinions -- anonymous and otherwise -- we hear about these players before they enter the league is truly remarkable. The error bars are impossibly large. Ask around about Wyoming quarterback Josh Allen and you'll hear that he'll turn into budding MVP candidate Carson Wentz or Titans washout Jake Locker. You'll hear that Heisman Trophy winner Baker Mayfield will turn into either Johnny Manziel or Russell Wilson. This doesn't happen in other sports. Jaylen Brown didn't enter the NBA draft only to be compared to both Jimmy Butler and Bill Murray in "Space Jam."
Picking the right quarterback is the most important thing an NFL organization can do. The Browns famously didn't believe in Wentz and traded the No. 2 pick in 2016 to the Eagles, who did. The rest is history. You can basically get everything else wrong and still repeatedly make it to the playoffs with the right quarterback, as we saw in the first few years of Andrew Luck's career during Ryan Grigson's reign as general manager in Indianapolis. It is not hyperbole to suggest getting this decision right is worth hundreds of millions of dollars.
And yet history tells us that the league will wrongly evaluate these prospects. Chances are that one or two of these five passers will turn into superstars, but it's unlikely that those one or two will be the first quarterbacks taken on draft night. Some fans are about to buy authentic team jerseys they'll quickly regret. Thousands of scouts spent tens of thousands of nights in hotels around America for decisions nobody will want to claim three years from now. We should be able to do better than this.
So what has gone wrong? Why can't we reliably figure out which quarterbacks will turn into superstars? And can we fix it in time to evaluate this year's class?
Over the past two days, I've looked into how and why we struggle with the most important part of the draft process. I started Monday with what recent history can tell us, and will try to answer the "why" in Part 2 on Tuesday, as well as evaluate whether the problems are fixable and apply what we know (and don't know) to the Class of 2018.
Jump to Part 1 on the history of the NFL's poor track record of evaluating quarterbacks.
Why is it so difficult?
All those examples aren't to suggest that NFL personnel executives are stupid and don't understand football, or that only a few select experts have any idea what's going on, both before and after quarterbacks have entered the pros. Longtime Ravens GM Ozzie Newsome traded up to grab Kyle Boller. Bill Belichick, who was on the other end of that swap, famously went back and forth between Tom Brady and Tim Rattay during the 2000 draft. Howie Roseman and the Eagles traded up to grab Wentz, but they also handed $12 million in guarantees to Chase Daniel, who threw one pass in an Eagles uniform before being released and replaced by Nick Foles. Even for the best, this is tough.
The simple response is usually just to peg players who underperform expectations as "busts" and assume that they would have failed regardless of where they ended up. That's a naive way to evaluate draft picks. Think about Jared Goff, who looked entirely lost during his debut season under Jeff Fisher and offensive coordinator Rob Boras. What if the Rams had passed up the opportunity to hire Sean McVay and held on to Fisher for another interminable season? Would Goff have improved? If he had been truly dismal for a second consecutive campaign, would the Rams have given up on him and gone after yet another quarterback this offseason? Goff very well could have been written off as a bust.
There are more realistic reasons why coaches and executives end up making decisions that seem curious with the benefit of hindsight. Let's understand why:
The guys who drafted the QBs don't get to stick around for long. The coaches and general managers who fall in love with a quarterback don't always get to stick around for long to actually mold their would-be star. In Goff's case, that was clearly a good thing, but other new coaches don't click with their inherited starter. Matt Leinart's career went south after Dennis Green was replaced by Ken Whisenhunt in the desert.
This is increasingly becoming a problem. Take a look at every top-10 passer over the past 20 years and their first four professional seasons, which form the bulk of their rookie deal. Of the 20 quarterbacks drafted under the old CBA, 10 played under the coach who drafted them for the first four years of their career.
Thirteen quarterbacks have been drafted since the league went to a new rookie scale in 2011, and of those 13, just four -- Cam Newton, Andrew Luck, Carson Wentz and Patrick Mahomes -- have held on to their coach for the first four years of their careers. Even that number is generous, given that Luck lost interim coach Bruce Arians after one season once Chuck Pagano returned, while Wentz is only two years into his career and Mahomes just finished his debut campaign.
Highly drafted quarterbacks don't get to enjoy much stability. Mitchell Trubisky is already on his second coach after John Fox was fired. The same is true for Goff. Jameis Winston is on his second after Dirk Koetter was promoted from an offensive coordinator's role. Marcus Mariota is about to start his fourth season with his third coach; the coach who drafted him, Whisenhunt, was fired after seven games.
The numbers are even more notable when you look at offensive coordinators. Not a single quarterback taken in the top 10 since Matthew Stafford in 2009 has managed to make it four years with the same offensive coordinator at the helm. Each of the 15 top-10 passers drafted since -- a group that includes Wentz and Mahomes -- has been forced to work with at least two offensive coordinators during their first four pro seasons.
Translating between schemes is difficult. NFL teams have adopted some plays from the more wide-open college game. Heck, the Eagles ran a concept that descends from the Air Raid -- Mesh -- for a number of their big plays in the Super Bowl. For various reasons both practical (the wider college hashmarks) and obstinate (ask Titans fans about the "exotic smashmouth"), teams are often stuck interpreting college tape of passers into their professional scheme.
NFL offenses do look more like modern college offenses than ever before, given how frequently they throw the ball and how often they line up in the shotgun. Teams that run more traditional attacks have to account for a quarterback's ability to line up under center when they typically took snaps out of the Pistol or shotgun. Successful NFL passers make their throw after one read more frequently than some might suggest, but coaches who will ask their new starter to make deeper progressions also have to project what they think their passer might be able to do versus seeing him do it on a drive-by-drive basis in college.
Players who are drafted for their tools don't necessarily end up in schemes that play to their strengths. Every NFL team covets a quarterback who can "make all the throws." Offenses don't throw downfield as much as they used to -- the average pass traveled 8.0 yards in the air in 2017, down from 8.3 yards per throw a decade ago -- but when you are in third-and-long or driving for a score late in the fourth quarter, you want a quarterback capable of planting his foot and driving an out route 24 yards downfield to the sideline.
The problem is that teams don't necessarily end up using their quarterbacks in that way. The two starters in the league who were drafted more for their arm strength than anything else are Stafford and Joe Flacco, who have absolute howitzers on their shoulders. They also, curiously, work within two of the most conservative schemes in the league.
Over the past two years with Jim Bob Cooter, Stafford has averaged 7.6 air yards per throw, which is 24th in the league. Flacco is even further down the stat sheet; at 6.8 yards per pass, the only quarterback who has thrown his typical pass shorter in the air is Sam Bradford. The most famous comeback of the 2017 season came on a deep out thrown by Case Keenum -- who wasn't drafted in part because of a lack of arm strength -- to Stefon Diggs in the playoffs.
Analytics haven't helped all that much (yet). In 2008, college student David Lewin looked back at college quarterbacks from the past and found that completion percentage and games started were the only two meaningful predictors of professional success, and that only worked for quarterbacks selected in the top two rounds of the draft. (Lewin now works for the Boston Celtics as their director of scouting.)
The idea had a couple of fitting issues -- the data set hadn't been backtested, and the endpoint of the top two rounds was arbitrary -- but the concepts were insightful. Quarterbacks who struggled with their accuracy in college rarely developed it at the professional level, and as passers spent more time in college, scouts gained a better sense of their actual skills and were able to provide a more meaningful evaluation of their talents.
Since then, the results have been mixed. Passers with sub-60 percent completion percentages like Stafford and Matt Ryan have found success at the pro level, while hyper-accurate darlings like Brian Brohm and Robert Griffin III haven't been able to parlay college completions into regular NFL work. The disparity in schemes between schools naturally makes it easier to complete passes in some places as opposed to others, and a more robust version of the Lewin model would incorporate air yard data, which is beginning to become available to quantitatively inclined organizations. Teams have done their own internal studies to measure which elements of college success actually do translate to professional staying power, but that data might not be convincing enough to get a traditional owner, general manager or coach to listen.
Furthermore, "analytics" is an overly broad catch-all, because one team's research may be totally different from another's. The Browns were the league's most numbers-friendly team under the stewardship of Sashi Brown, but Cleveland famously didn't believe that Wentz would turn into a franchise quarterback. (It's worth nothing that, contrary to what might be said today, their coach also felt like they should have passed on Wentz with the second pick.)
The Eagles felt differently and traded up to grab Wentz, and while it's tempting to contrast their subsequent success to that of the Browns, Philadelphia has been influenced by analytics for more than a decade, going back to the days of Joe Banner and Andy Reid. Both teams were using a mix of analytics and traditional scouting to evaluate Wentz, and in the end, the Eagles appear to have been right.
Unqualified evaluators can use pseudo-psychology to talk themselves into anything. Statistics and tape aren't enough to scout players, as teams rightfully want to know a quarterback might fit in their locker room and how he'll command a huddle. Off-field problems have sunk passers like Ryan Leaf, Manziel and Todd Marinovich. Organizations have to do their due diligence.
The problem, though, is that they often take a leap past those evaluations and adopt rules or administer tests that can't realistically prove anything. The Giants were famous for administering a 200-page psychological exam to potential draftees, at which Deion Sanders summarily laughed and walked away.
This year, one NFL organization asked LSU running back Derrick Guice if he was gay, marking at least the third time in recent memory a team has reportedly asked a player this irrelevant (and possibly illegal) question. Another asked wide receiver prospect Braxton Berrios to explain both ISIS and Bitcoin to them; given the age of NFL executives, you might expect they were looking for a primer on the latter. You can infer just about whatever you want, positive or negative, from asking Berrios those questions.
Another popular theory is the marriage test, as espoused by Texans owner Bob McNair when Houston drafted David Carr as opposed to Joey Harrington with the opening pick of the 2002 draft. "David has been married for three years," McNair said at the time. "He's got a wife, Melody, and a 23-month-old son, Austin. He's already settled down. He can handle success."
Carr might very well have been settled down, but it wasn't enough for him to succeed behind a dismal offensive line in Houston. Plenty of quarterbacks weren't married at the time they were drafted. Brady was years away from dating -- let alone marrying -- supermodel Gisele Bündchen. Aaron Rodgers remains unmarried. Troy Aikman didn't get married until his final year in the NFL.
Teams aren't much smarter about this stuff now. It might not be the single determining factor in deciding who a team selects, but there is plenty of useless information being strewn around this year. Hue Jackson is impressed that Baker Mayfield got his teammates to chant "Hee hee." Giants owner Steve Tisch was blown away by Josh Rosen's table manners. Two years ago, the Jets were impressed that Christian Hackenberg could make easy conversation about the Masters at a local bar.
Others want to believe they can see whether a quarterback will succeed or not by looking into their eyes or figuring out whether a passer has the "it" factor. I wrote about the "it" factor fallacy back in 2014, noting how quarterbacks like Mark Sanchez and Vince Young had been pegged with the moniker early in their careers, only for "it" to disappear when they were exposed.
Everyone from Jimmy Clausen to Tim Tebow has been regarded as someone with the "it" factor at one point or another during their careers, but "it" hasn't been remotely predicative of pro success. Two of the quarterbacks who we would say most clearly have "it" are Brady and Russell Wilson, who fell past the first two rounds of the draft when everyone who swears they can spot "it" should have seen "it" from a mile away. Intangibles clearly exist, but we're not very good at identifying who possesses "it" until well after the fact.
Underclassmen quarterbacks get to decide when we stop seeing them, which is often at a high point. While there are exceptions, many of the quarterbacks who are taken in the first round of the draft are underclassmen who declare for the draft with eligibility still remaining. Over the past four drafts, 10 of the 11 quarterbacks who were chosen in the first round left school before exhausting their eligibility, with Wentz -- who went to FCS North Dakota State -- as the lone exception.
By definition, quarterbacks who declare for the draft are almost always going to be leaving after what they consider to be a career season. Few quarterbacks are going to forgo their remaining eligibility if they feel like they aren't going to be a first-round pick or didn't play well in their final collegiate season. They have the ability to cut off the information spout when they've made a great impression as opposed to quarterbacks who play out their amateur eligibility and don't live up to expectations during their final year. Leinart is a good example of the latter.
Take Trubisky, who declared for the pros after his junior season, which was his only season as a starter. It's hardly out of the question that he could have kept up his excellent play and continued to perform at a high level, which would have kept him in the top five of the draft. There's also a chance that he would have suffered without Mack Hollins and Ryan Switzer, thrown a few more picks, and fallen into the second round as a project.
It's difficult to imagine a year we didn't actually see from Trubisky, but young quarterbacks see their stock fluctuate all the time. Think about the differences in how we perceive Derek Carr and Dak Prescott between 2016 and 2017. Quarterbacks who play well enough to leave early get to be judged at their best tape as opposed to simply their most recent tape.
We're not really good at drafting in general, let alone quarterbacks. As many folks have documented, including Neil Paine for 538, no organization has proven to be better than another at actually picking players. The best teams in the league have succeeded by amassing extra picks via compensatory selections and by trading down, which has worked for the Patriots under Bill Belichick.
The Steelers famously had what was widely regarded as the best draft in league history in 1974, when they drafted four future Hall of Famers, but their drafts in 1973 and 1975 saw them select 41 players who combined to make two trips to the Pro Bowl. The next Hall of Famer Pittsburgh drafted was Rod Woodson in 1987, 13 years later. Study after study tells us that after you account for the expected return from a given position in the draft, no executive consistently outscores the opposition.
Organizations get impatient. Teams that say one thing during the pre-draft process don't actually follow through on their development plans. The Jaguars wanted to give Blake Bortles a redshirt year in 2014 to rebuild his mechanics, but after a solid preseason and three mediocre games from backup Chad Henne, an 0-3 Jags team named Bortles as their starter in Week 4. Bortles' mechanics never took and eventually collapsed altogether in 2016.
The Eagles, meanwhile, intended to sit Wentz behind Bradford in 2016, but when Philadelphia got a great trade offer from the Vikings for Bradford, they inserted Wentz into the lineup a year earlier than expected. Things obviously went fine for the second overall pick one year later, but teams generally push their new quarterbacks into the lineup earlier than they suggest in April.
In some cases, this might not be the decision of the people who seem to be involved. Ownership might encourage -- or insist -- that a team draft a quarterback or insert their new prospect into the lineup. A coach who is in danger of losing his job might prematurely bench a struggling young quarterback in the hopes of finding a higher floor with an experienced veteran, representing the moral hazard of when a team's incentives to win aren't in line with that of their head coach.
What does that mean for the 2018 draft class?
For one, anyone who says they're sure that any of these prospects is going to turn into a star or a mistake is kidding themselves. It's pretty clear that much of what makes a quarterback is what happens after he arrives in the NFL. How is his team at developing draft picks? What sort of infrastructure does he have to succeed? How much will his coach, offensive coordinator and quarterbacks coach do to ease the transition to the professional level? Educated eyes can certainly pick up elements of a quarterback's game he'll need to improve or rely upon at the next level, and history may suggest some members of the quarterback pool are more likely to succeed than others, but nobody's a surefire hit or a guaranteed disaster.
There are question marks about each of these prospects, such as the pre-draft debate on ESPN comparing Wyoming quarterback Josh Allen and Louisville passer Lamar Jackson. Bill Polian, the former GM of the Colts who now works as an NFL analyst for ESPN, suggested earlier this year that Jackson should consider a move to wide receiver owing to his athleticism and size, but Jackson measured in at the NFL combine at 6-foot-2 and 216 pounds, which is nearly identical to the listed height and weight of Andy Dalton. Dalton, who was a surprisingly influential running quarterback in terms of the the zone-read during his time at TCU, does not appear to have been asked to switch positions before or during his pro career.
Allen, meanwhile, is a quarterback prospect straight out of the 1970s. The 21-year-old has ideal height (6-5) and a cannon for an arm, but he completed just 56.2 percent of his throws at Wyoming, the lowest completion percentage for a would-be first-rounder since Locker (54.0 percent). ESPN draft guru Mel Kiper Jr. has suggested Allen played in an offense that frequently threw the ball down the field, deflating his completion percentage, with Jackson playing in a scheme in which it was easier to find safe completions.
In talking to NFL coaches and executives at the combine, there are certainly plenty of people within the league who share Kiper's sentiments about the Louisville and Wyoming offenses. It's also impossible to break down every single one of the playcalls for each offense to figure out whether Allen turned down easy completions to take big shots downfield. Remember the air yards statistic I mentioned earlier, though? We have that data for Allen Jackson, and the rest of the quarterbacks in this year's draft, thanks to ESPN Stats & Information research.
It's true that Allen didn't throw many short passes relative to your typical college quarterback, but Jackson actually threw fewer "layups" than Allen when you account for the fact that he has an extra year on tape. Here are the numbers for each of the five passers, split by the distance over which the ball traveled in the air. Note that these numbers don't include the six passes Allen threw in 2015 or Baker Mayfield's stats from his lone season at Texas Tech in 2013:
Josh Rosen threw more screens and quick hitches than any of the other candidates, as more than 40 percent of his passes were thrown within 4 yards of the line of scrimmage. Jackson didn't throw a ton of screens, but 42.8 percent of his passes were in the midrange between 5 and 14 yards, which was comfortably the most of any of these five signal-callers. And then, it's certainly true that Allen was chucking it downfield more than anybody else. Nearly a full one-third of his passes, 33.2 percent, traveled 15 or more yards downfield. Rosen, who was playing in a version of the West Coast offense under the tutelage of former NFL coach Jim Mora Jr., threw downfield on just 22.9 percent of his passes.
Let's now consider how accurate each quarterback was in each respective zone. Here is each quarterback's completion percentage, split out by those same air yard figures:
Allen and Jackson, ironically, have roughly similar profiles. They were both well below the NCAA average for completion percentages on extremely short throws (73.3 percent), but were better than the typical passer on throws 15-plus yards downfield.
It's difficult to not notice Mayfield, given that the Heisman Trophy winner was significantly ahead of the pack in most categories. It's staggering that Mayfield's completion percentage on throws traveling 20-plus yards in the air (51.8 percent) wasn't all that far behind that of Allen on throws at or behind the line of scrimmage (58.7 percent). Mayfield's passer rating on those throws was 297.6, and he posted a 99.7 QBR on those throws. Allen was second-best in those same categories among the five would-be first-rounders, but his passer rating was a mere 185.8, with a 90.8 QBR mixed in.
One more chart. We can use NCAA-average completion rates and the rate at which each passer threw each particular type of pass to project what his completion percentage "should" have been, given those range of throws.
Jackson actually underperformed his completion percentage by the largest amount, although it's not materially different from Allen and doesn't lend itself to the layups argument. Rosen was essentially about where we would expect given his range of passes. Sam Darnold was well ahead of expectation, but Mayfield was in another universe.
This isn't the whole story. A deeper analysis would need to account for the types of specific throws each passer was asked to make and the game situation. QBR does exactly that and gives Mayfield an 89.0 mark over the last three years, with Jackson at 82.3, Darnold at 80.2, Allen at 66.6, and Rosen holding up the rear at 65.4. QBR doesn't know the playcall and can only say much, but it's a further reflection of how these quarterbacks actually impacted both drive and win expectancy.
Jackson ranks second despite decent passing numbers because he was simply dominant as a scrambler and runner. Over his three seasons at Louisville, Jackson ran for 4,132 yards and 50 touchdowns. Forget quarterbacks; Jackson would be third in the country in rushing yards and has more rushing touchdowns than any running back in college football over the last three seasons. He had 171 runs of 10 yards or more, 33 more than any other player over that timeframe.
To put his rushing dominance in context, QBR suggests Jackson generated 180.4 points for Louisville over the last three seasons as a runner. The other four quarterback hopefuls in this year's draft generated 113.8 points combined with their feet over that same timeframe. Jackson is good, but not great as a passer -- strip out the runs and his QBR on throwing plays is 73.8, which is just ahead of Allen (73.7) and only good for 26th among passers with 600 attempts or more over the last three seasons.
If a team drafts Jackson to play quarterback and discourages him from running the football, he might disappoint for reasons entirely out of his control. Likewise, though, if a team drafts Allen and tells him to never throw the ball any farther than 10 yards downfield, they're not going to be happy with the results. Draft Mayfield, line him up under center, and tell him to never leave the pocket? You'll end up playing to your quarterback's weaknesses as opposed to his strengths.
It sounds downright stupid for a team to do that, but organizations make these sorts of mistakes all the time. Mora inherited Michael Vick and tried to turn him into a West Coast quarterback. Mariota went from spending all of his time in the shotgun at Oregon to an offense that took 45.1 percent of its snaps under center over the last two years under Mike Mularkey, the 10th-highest rate in football. Imagine if Wilson had been drafted by a team that insisted he develop as a quarterback under center as opposed to the Seahawks, whose obsession with competition led Pete Carroll to award Wilson the job over free-agent signing Matt Flynn before the season began.
History tells us not to expect each of these five passers to turn into stars. Just seven draft classes since the AFL-NFL merger of 1970 have produced four or more first-round quarterbacks, and their track record is mixed at best. Of the 31 passers who were taken across those seven classes, five -- Elway, Kelly and Marino from 1983 and both Eli Manning and Ben Roethlisberger from 2004 -- are likely to make the Hall of Fame. The jury's still out on Luck and Newton, the first overall picks from the 2011 and 2012 drafts.
Thirteen of the 31 passers failed to produce more than one season of above-average football as a starter. That's 42 percent of the quarterbacks chosen, and it's a group that includes passers selected with high picks. Quarterbacks went 1-2-3 in the 1999 draft, and while second overall selection Donovan McNabb worked out well for the Eagles, the guys chosen first (Tim Couch) and third (Akili Smith) were disappointments. They combined to make zero Pro Bowls, while the guys chosen with the next five picks -- Edgerrin James, Ricky Williams, Torry Holt, Champ Bailey and David Boston -- combined for 25 trips to Hawaii.
There's significant risk in taking a quarterback in the first round, but the potential reward is enormous. The Eagles had Wentz signed for 2017 at a cap hit of just over $6.0 million, peanuts in a league in which the top young quarterbacks were signing extensions at $25 million per season. Wentz's cap hit won't rise above $8.5 million until 2020, creating a savings that has allowed the Eagles to squeeze in deals for the likes of Alshon Jeffery and Tim Jernigan. The Seahawks were able to add veterans such as Cliff Avril and Michael Bennett to an already-stacked team because Wilson's four-year rookie deal cost only a combined $3 million.
Under the new CBA, teams are heavily incentivized to draft and develop rookie quarterbacks into useful contributors. To say that drafting and developing passers is an inexact science, though, would be an understatement. Instead of focusing on the players and what we can gauge about their possible success before they get drafted, it seems way more appropriate to evaluate the infrastructure surrounding those quarterback opportunities, since they may be a far better indicator of whether each passer is likely to develop into the franchise icon any first-round pick is expected to become.
As I mentioned earlier this year in talking about Case Keenum, the old trope about how there aren't 32 good starting quarterbacks to go around is naive. The problem is that there aren't 32 good quarterback situations to play within, a reality that was reinforced by the success of Keenum and Foles after they looked disastrously bad under Jeff Fisher with the Rams. Teams that are spending every waking hour worrying about picking the right guy are likely kidding themselves to think that they can reliably pick the best quarterback from this group. They should worry about making themselves the right fit for whoever they end up choosing.
Part 1
How bad is the problem?
Pretty bad. We're about to hit the 20-year anniversary of the 1998 draft, which famously saw Colts general manager Bill Polian agonize over whether to take Peyton Manning or Ryan Leaf, as Peter King documented for Sports Illustrated. Polian got his call right. The Colts grabbed Manning while the Chargers traded up to No. 2 and happily settled for Leaf, who threw 36 interceptions over 18 starts and was out of football by 2001.
Since then, it hasn't been quite as easy. We're still judging the most recent classes, and there are different ways to evaluate quarterbacks, but the first quarterback taken in the 19 drafts since 1998 has been the most productive and/or successful passer from that class only five times. It hasn't happened since 2011, when the Panthers controversially chose Cam Newton over Blaine Gabbert and a bevy of dominant defenders with the first overall pick. (More on that later.) That was the fourth consecutive year in which the league correctly chose the most productive pro passer first, but the success rate wasn't good before that and hasn't been good since.
You can quibble with a couple of those choices. Maybe you prefer Luck over Wilson, even though Wilson has won a Super Bowl and has been far more productive in Seattle. Perhaps you think Jimmy Garoppolo has already proved to be better than Derek Carr. And there are some cases in which there really were no winners: The Bills would probably prefer to have Geno Smith over EJ Manuel, but you suspect they would rather have stayed out of the quarterback market altogether in 2013. You get the idea, though: The first guy often isn't the best guy.
It's fair to note that this doesn't tell the whole story. Teams that pick first are often some of the worst organizations in football, and they're among the worst because they're bad at talent evaluation and subsequent development. It can become a self-fulfilling prophecy -- in other words, the Browns stay the Browns.
From what we know, though, even some of the league's most decorated talent evaluators struggle with evaluating quarterbacks. When he was deciding between Leaf and Manning in 1998, Polian paid the late Bill Walsh $5,000 to evaluate both passers, a price that I suspect most teams would love to pay for a quarterback evaluation today. If you were making a list of the best quarterback evaluators in league history, the architect of the 49ers dynasty might be a good place to start.
Quotes from the report made their way into a USA Today story at the time by Richard Weiner, and surprisingly, Walsh raved about Leaf. "He is gifted, in just a natural throwing motion that is so quick," Walsh reportedly said in his evaluation. "With a flick of his wrist, he can get the ball just about anywhere he wants. He is a good competitor, amazingly agile and smooth and graceful in his movement as a big man can be. He handled the Washington State offense beautifully. In a sense, it was an aerial circus." Mike Shanahan, then the coach of the Super Bowl-winning Denver Broncos, said the "excellent" Leaf was "a big, strong kid with unbelievable arm strength. He's played in a system very similar to a lot of NFL teams. That cannot be underestimated."
At the same time, we don't know what Walsh's report said about the future Hall of Famer on the other side of this comparison, and most evaluators preferred the Tennessee star. A poll of executives at the time found that 20 of 25 observers preferred Manning, while three preferred Leaf and two couldn't decide. Then-Steelers general manager Tom Donahoe, a member of the Eagles' front office staff when it chose Wentz in 2016, suggested that you couldn't go wrong with either guy. Colts coach Jim Mora said that both Leaf and Manning would be good players in the NFL, while Seahawks coach Dennis Erickson believed that Leaf would turn into one of the "future stars of the league."
We're lucky to have that many quotes about the Leaf-Manning debate on the record. Usually, pre-draft scouting evaluations are the domain of the dreaded anonymous scout or personnel executive, whose comparisons are often facile and occasionally prejudicial. It's only after the fact that those executives come out and reveal that they had a first-round grade on that franchise quarterback they passed on three times and let someone else take in the fourth round.
Fortunately, to get a better sense of what the league actually thought about quarterbacks, there's Bob McGinn. The legendary Packers beat writer took an annual poll of anonymous evaluators before the draft during his time with the Milwaukee Journal-Sentinel, and asked them to rank the quarterbacks from each year's draft class. McGinn then scored each of those ballots with five points for the top-ranked quarterback, four for the second-ranked quarterback and onward.
I was able to find polls going back to the 2005 draft, which was a fateful session for the team McGinn covered. Let's run through the top five from those polls to see just how difficult it is to project quarterbacks.
2005
1. Alex Smith (63 points, 11 first-place votes)
2. Aaron Rodgers (53, two first-place)
3. Jason Campbell (25)
4. Charlie Frye (17)
5. David Greene (16)
While Rodgers challenged to be the first overall pick in the draft, the 49ers chose Smith, with Rodgers falling all the way to the Packers at 24. Many observers were concerned that Rodgers was a product of Cal coach Jeff Tedford, who previously sent Akili Smith, Kyle Boller and Joey Harrington to the pros with limited success. One NFL personnel director told McGinn that Tedford quarterbacks "all throw the ball the same way," while another said Rodgers "is very rigid mechanically." It's difficult to think of a quarterback in NFL history who is more fluid in getting rid of the football off-schedule than Rodgers.
2006
1. Matt Leinart (71 points, 11 first-place votes)
2. Vince Young (61, six first-place votes)
3. Jay Cutler (55, four first-place votes)
4. Brodie Croyle (10)
5. Charlie Whitehurst (9)
Opinions were split on the first three quarterbacks, although the legendary Rose Bowl battle between Leinart and Young loomed heavily on every scout's mind. Leinart lost his job to Kurt Warner in Arizona, while Young mixed game-changing plays, bad decisions and inconsistency during his run in Tennessee. Young made two Pro Bowls but lacked the longevity of Cutler. As then-Broncos general manager Ted Sundquist documented in 2015, Shanahan wanted to draft Leinart, only for the Cardinals to beat them to the punch. Cutler would be the last highly touted quarterback to hit the NFL with a significant losing record in college until the Titans drafted Locker in 2011.
2007
1. JaMarcus Russell (69 points, 15 first-place votes)
2. Brady Quinn (54, three first-place votes)
3. Trent Edwards (25)
4. Drew Stanton (15)
5. John Beck (8)
Kevin Kolb, who had the most success of any quarterback in this class, received only six points. Russell's pro day was the stuff of legend, with both draftniks and ex-pros alike raving about his future. Bucs coach Jon Gruden compared it to "Star Wars," while ESPN draft expert Mel Kiper Jr. compared Russell's athleticism to Elway's. Texans coach Gary Kubiak said he was sure Russell would turn into a great player, while Hall of Famer Terry Bradshaw said Russell was a "pretty easy" choice for the Raiders with the first overall pick.
Others preferred Quinn, although he fell to the bottom of the first round before the Browns traded up to snatch him away with the 22nd pick. Polian raved about Quinn, dismissing a 58.0 completion percentage while noting that Quinn had no protection during his time at Notre Dame. Others disagreed, but Quinn certainly had his backers. McGinn asked 18 scouts if they would prefer to have Quinn or Rodgers, who had thrown only 31 passes with limited success over his first two seasons backing up Brett Favre in Green Bay. Just four of those 18 scouts chose Rodgers while 12 chose Quinn and two said it was too close to call.
2008
1. Matt Ryan (71 points, 12 first-place votes)
2. Brian Brohm (53, one first-place vote)
3. Chad Henne (38, two first-place votes)
4. Joe Flacco (36)
5. John David Booty (11)
There were concerns at the time about Ryan's accuracy, but after completing just 59.9 percent of his passes at Boston College, Ryan is up to 64.9 percent as a pro. Brohm, a quantitative darling who completed 65.8 percent of his passes at Louisville, failed to unseat Rodgers and never developed into a successful pro. Flacco rose late up draft boards by virtue of his arm strength, although McGinn astutely noted that the famously quiet Flacco could be characterized as an introvert.
2009
1. Matthew Stafford (46 points, 11 first-place votes)
2. Mark Sanchez (39, five first-place votes)
3. Josh Freeman (21, two first-place votes)
4. Rhett Bomar (1)
5. Pat White (1)
Stafford was another quarterback with a below-average 57.1 completion percentage in college, but he has improved his mechanics and accuracy as a pro, with the move to a short passing scheme under Jim Bob Cooter pushing him closer to 66 percent over the past several seasons in Detroit. Sanchez nearly parlayed his cool demeanor and lone year of college success into coming off the board second to the Rams, only to instead go to the Jets at No. 5 as part of a trade that the Browns somehow still lost.
2010
1. Sam Bradford (80 points, 20 first-place votes)
2. Jimmy Clausen (57)
3. Colt McCoy (30)
4. Tim Tebow (21)
5. Dan LeFevour (5)
The scouts got this one right, although Bradford never developed into the superstar some projected he would. Falcons general manager Thomas Dimitroff told McGinn that Bradford was "a tall-stature guy who has that prototypical stand in the pocket," which seems like a roundabout way of saying that the 6-foot-4 Bradford was tall. The flashpoint was obviously Tebow. McGinn polled 21 scouts about his future, and the anonymous ones mostly got it right -- 16 said he wouldn't turn into an NFL starter, but eight of them suggested they would consider taking Tebow in one of the first two rounds of the draft.
2011
1. Blaine Gabbert (105 points, 16 first-place votes)
2. Cam Newton (100, 11 first-place votes)
3. Jake Locker (43)
4. Ryan Mallett (43)
5. Christian Ponder (37)
You'll note that this list doesn't include Andy Dalton (15 points) or Colin Kaepernick (11). Opinions were split on the class. One scout rather hilariously told McGinn that there were only second- and third-rounders at the position in the pool and bemoaned the fact that there was no Bradford or Freeman. McGinn polled the scouts on Newton's future, and only two of the 24 said he would become a perennial Pro Bowler. Nine said he would be a solid starter, while another nine said he would play without ever becoming effective. Four said he would be a bust.
2012
1. Andrew Luck (95 points, 19 first-place votes)
2. Robert Griffin III (75)
3. Ryan Tannehill (57)
4. Brandon Weeden (18)
5. Brock Osweiler (14)
Luck was the highest-regarded prospect of this generation, so it's no surprise he received every first-place vote from the queried scouts. Among the guys who didn't make it to the top five were Kirk Cousins (13 points), Russell Wilson (6) and Nick Foles (1). McGinn himself ranked Osweiler over Wilson, with one scout saying Osweiler was "like the CEO of a company." That's another roundabout way of saying someone's tall. Many were concerned about the 5-foot-11 Wilson's ability to throw from the pocket, but the only thing that's prevented him from doing so as a pro has been the Seahawks' offensive line.
2013
1. Geno Smith (74 points, 11 first-place votes)
2. Matt Barkley (58, three first-place votes)
3. EJ Manuel (35, two first-place votes)
4. Mike Glennon (28)
5. Ryan Nassib (22)
The scouts rightly thought this was a dismal quarterback class, although it's clear to see the difference between the consensus and what actually happened on draft day. While most of the league preferred Smith, the Bills were one of the exceptions and took Manuel with the 16th pick. The Florida State passer was the only quarterback taken in the first round.
2014
1. Johnny Manziel (68 points, eight first-place votes)
2. Blake Bortles (61, four first-place votes)
3. Derek Carr (49, two first-place votes)
4. Teddy Bridgewater (41, two first-place votes)
5. Jimmy Garoppolo (16)
It's still unclear exactly who will end up as the best quarterback from this class, but it does seem safe to say it won't be Johnny Football. Twelve of the 20 scouts McGinn polled suggested Manziel's career would go down as a miss, with scouts expressing concerns about his off-field behavior even before hitting the pros. Zach Mettenberger received one first-place vote, with his six points placing him just ahead of AJ McCarron's five.
2015
1. Marcus Mariota (85 points, nine first-place votes)
2. Jameis Winston (84, 10 first-place votes)
3. Bryce Petty (45)
4. Brett Hundley (39)
5. Garrett Grayson (19.5)
Winston was seen as a far more polarizing prospect at the time of the draft, owing in part to his off-field troubles. One scout compared Winston to a less athletic version of JaMarcus Russell. Eight of the 19 executives on McGinn's anonymous committee suggested Winston would be a disappointment or a bust, while just one said the same thing about Mariota.
2016
1. Jared Goff (73 points, eight first-place votes)
2. Carson Wentz (71.5, seven first-place votes)
3. Paxton Lynch (52, one first-place vote)
4. Connor Cook (29, one first-place vote)
5. Christian Hackenberg (11)
Dak Prescott finished behind Hackenberg with six points. While that seems comical now, remember that the Cowboys tried to trade up for Lynch and then were pipped to Cook by the Raiders before settling on the guy who would quickly become their franchise quarterback. One scout told McGinn that Prescott had "no accuracy and no vision," while saying the Mississippi State product wasn't an NFL quarterback. Another suggested that the Eagles would need to sit Wentz for a year or two before he could become a real starter, while others alternately compared him to both Bortles and Ben Roethlisberger.
2017
1. Mitchell Trubisky (61 points, four first-place votes)
2. Deshaun Watson (58, six first-place votes)
3. Patrick Mahomes (56, five first-place votes)
4. DeShone Kizer (32, one first-place vote)
5. Davis Webb (23)
It's still far too early to draw any conclusions about this class, but it does seem likely that the person who submitted a first-place vote for Kizer probably won't be bragging about it publicly anytime soon.
On Tuesday, we'll try to figure out why teams struggle to evaluate quarterbacks, figure out whether the problems are fixable, and apply what we know (and don't know) to the Class of 2018.