HOUSTON -- In the week leading up to the Houston Texans’ 25-22 loss to the New England Patriots, coach David Culley had one message for his rookie quarterback.

You’ve got to protect the football.

Davis Mills was coming off a poor performance in Buffalo -- a 40-0 loss -- where he threw four interceptions and had a Total QBR of 0.8, the fourth-lowest Total QBR in the last five seasons.

But while Mills took a step forward with zero turnovers and by far the best game of his young career, the Texans once again couldn’t overcome a myriad of mistakes they made, falling to 1-4 on the season. With the way the roster is built, and the way the team has played thus far, the Texans need to stop shooting themselves in the football, Culley said.

“I don’t think we have to be perfect, but we have to be more consistent than what we’ve been,” Culley said Monday. “And when you’re 1-4 and you look back and see why those things happen, you look back and you want to say, ‘Was it the team you’re playing?’ That hasn’t been the case with us. We’ve been the enemy sometimes. A lot of those miscues that we had, it wasn’t because of what they did, it was what we didn’t do.

“We were in a ballgame yesterday that we could have very easily won, and they made a play in the end to win the game and we didn’t put ourselves in a situation to do that.”

For weeks, Culley has lamented the penalties the Texans were committing that was forcing the offense to play behind the chains. On Sunday, the Texans had eight penalties for 70 yards -- an improvement over the week before in Buffalo -- but still something Culley said "kept us from being able to keep the ball moving along."

But while penalties were a factor Sunday, it was Houston’s special teams that played a major role.

The Texans entered Sunday minus-8.4 expected points added on special teams this season, which ranks 29th in the NFL, according to ESPN Stats & Information. And the unit made crucial mistakes Sunday.

Kicker Ka'imi Fairbairn missed his first two extra point attempts of the game. After his first missed extra point, his kickoff went out of bounds to give the Patriots (2-3) the ball at the 40-yard line. New England scored a touchdown on the drive to tie the game.

In the third quarter, the Texans tried to confuse the Patriots on a punt, resulting in a 0-yard punt after the ball hit the back of a Texans player. The Patriots started the drive at the Houston 36 before kicking a field goal.

And then in the fourth quarter, Fairbairn missed a 56-yard field goal. The Patriots got the ball at their own 46 and scored a touchdown to tie the game at 22.

And while players took the blame all the way around for the loss to the Patriots, a common theme was that the team let one get away from them after leading for three quarters.

“This one hurt a lot," safety Justin Reid said. "The game was right there. We had control of the game, and we let it slip through our fingertips. Couple unfortunate plays, couple plays that they made, and credit to them to go and close the game out. Just wish we could have some of them back. You always do whenever games end the way they do like that.”

Now, the Texans' offense that was held scoreless for the final 28 minutes and 29 seconds of the game, has to take a step forward against the Indianapolis Colts on Sunday (1 p.m. ET, CBS) if Houston wants to find themselves in the win column again. On Monday, Culley said he didn't think quarterback Tyrod Tylor would be activated from injured reserve for Week 6, the first game he is eligible to return from his left hamstring injury.

And if Mills plays against the Colts on Sunday, he said he thinks the loss to the Patriots the week before will help get him closer to his first win in the NFL.

“I think just kind of going out there and feeling the feeling of making plays and being right there close to winning games at the end [will help moving forward]," Mills said. "I think there's really a feeling of just learning how to win games and finish. And I think we're close. But just didn't do enough."