METAIRIE, La. -- New Orleans Saints quarterback Drew Brees often describes himself as "annoyingly optimistic."

It comes from experience.

Brees' recovery from a major shoulder injury in 2006 has been well chronicled. But his resilience with the San Diego Chargers in 2004 when rookie backup Philip Rivers was breathing down his neck was also vital to catapulting his Hall of Fame career.

I recently talked to Brees about just how close he came to losing his job permanently while we discussed the development of young quarterbacks -- a topic profiled in the latest issue of ESPN The Magazine.

Brees said he was actually one series away from being replaced by Rivers during a Week 4 game against the Tennessee Titans.

San Diego was 1-2, and Brees was off to a somewhat slow start, with the score tied 7-7 in the second quarter. Making matters worse, he had just suffered a Grade 2 AC separation in his non-throwing shoulder while being sacked. Brees insisted that he wanted to keep playing. But coach Marty Schottenheimer made it clear he was on a short leash.

"He comes over to me and he goes, 'You have one more chance. You have one more drive,'" Brees said. "If he plays Philip, Philip was the fourth pick in the draft. You don't put that guy in for a few games and then [switch back]. No, that's your guy now, you're sticking with him.

"I think history could be much different."

Brees said veteran fullback Lorenzo Neal -- who had been one of his biggest supporters -- delivered a message to him on the sideline.

"He saw me kind of grimacing, and he was like, 'C'mon, man. C'mon,'" Brees said. "He called me PB [for Pro Bowler]. He said, 'Hey PB, you gotta make a decision. This is a turning point for you. Either you're gonna be a great player in this league for a long time and you're gonna be the man, or you're gonna be a career backup. Right now, you gotta make that decision.'

"So, stiff upper lip, let's go. And then we score touchdowns on our next [two] drives, and the rest is history."

That situation was dire because the Chargers had just drafted Rivers. But Brees had been benched three times before -- and he went into detail on all of those instances, as well.

The first time came in 2002, during his first season as a starter. Schottenheimer told him they "just needed a spark" during a bad game and that Brees was still the starter.

The second time came a year later, when the Chargers were off to a 1-7 start and Brees was "really struggling" in a game at Chicago. Doug Flutie replaced him. Brees said he knew it was coming, and when it happened, he looked at Schottenheimer and said, "I understand."

But Brees said doing that was "heartbreaking," and he vowed he would never let it happen again.

Sure enough, after Brees had regained the job late in the season and Schottenheimer benched him again in Week 16, Brees refused to accept it.

"I lost it. I lost it," said Brees, who had turned the ball over three times in four drives but explained that the last interception tipped off LaDainian Tomlinson's hands. "I said, 'No, I'm not going out of the game.' And we went back and forth, back and forth, and then finally Doug Flutie ended up going out there. But it went from there into the locker room into his office the next day. But then he said I was the starter the next week and we won the game and kind of finished the season on a high note."

Brees said sitting out for those five weeks "really gave me perspective."

"I sat back and I realized just how much pressure I was putting on myself, just how difficult I was making the game," Brees said. "'I can really simplify this game if I just relax and trust my instincts.' Because I've always been a bit of a feel player. So when I came back, I felt a big difference in just my approach and my poise and that kind of thing."

Brees said he thinks it's rare for young quarterbacks like Ben Roethlisberger and Russell Wilson to come into the NFL and have instant success.

"All I can really do is look at my career. I came in, and my first year as a starter ... I was a bit of a gunslinger," Brees said. "You know, like, 'I'm just gonna kind of drop back and I'm gonna rip it. Good things are gonna happen a lot of times, but some things I just don't know any better.'

"There's things you can get away with in college and overcome in college that are much harder to overcome at this level, just because of the level of talent. And you feel like there's less opportunities for you to just kind of 'make a play' at this level. In college, we'd throw it 60 times a game. So if you threw three picks, it didn't matter, because you were gonna throw five touchdowns."