In an exclusive tell-all interview on Seven's Sunday Night, Mr Howard turned the criticism on himself, saying he should have bowed out during his final term.
"It was a mistake, a miscalculation, lost my nerve or whatever description you want to give it," Mr Howard said of his decision to remain party leader after polling badly ahead of the 2007 election.
"It was a big negative, that was very careless of me"
In his first major interview since his unseating seven years ago, the 75-year-old said his biggest political regret was in losing the 2007 election after a leadership debate during the APEC conference week.
"I asked Alexander Downer, 'why don’t you get cabinet ministers together and find out what they are thinking, because if all of you want me to go and make that plain, then maybe I should'."
"Now that was a mistake... because it was too late for leadership change."
Mr Howard also opened up about his simmering feud with Peter Costello over the party leadership.
"It is only natural that he would be disappointed um that I did not retire at a time that suited him," Mr Howard said.
"I didn’t make a deal... I said to him 'Peter I will only stay one and a half terms'. That was my feeling at the time but he didn’t say to me at the time 'lets shake on that, I will support you for the leadership'."
"Interestingly enough, he never said in 2003, 'Well hang on, we made a deal back in 1994 that you were going to go after one term… here you are eight years later'.”
He conceded that he was the wrong party leader to see out the 2007 election but that the realisation came too late.
"I mean he wanted me to go, I understood that, there was nothing wrong with that, his ambition was thoroughly legitimate."
After committing Australian troops to the Iraq war based on evidence of weapons of mass destruction, Mr Howard revealed to Sunday Night the mistake was 'embarassing' but not critical.
"I felt embarrassed, I couldn’t believe it cause I had genuinely believed it. So, incidentally, did Kevin Rudd.
"Kevin Rudd made a speech saying that It was 'an empirical fact' that Iraq had weapons of mass destruction. He later on said that I’d taken the country to war based on a lie."
Reflecting on his leadership, Mr Howard said he had benefited from his working-class roots.
"Australians are very intelligent, they’re very savvy. They don’t like humbug they don’t like phonies."
"They don’t mind a person being passionate about something they don’t necessarily support provided they can get on with their own lives but when somebody demonstrates a lack of conviction and consistency they lose interest."