The Post Office chairman was sacked because it "just wasn't working", the business secretary says.
Kemi Badenoch told the BBC that Henry Staunton's departure was about more than just the Horizon scandal, in the wake of public outcry over the wrongful convictions of sub-postmasters.
Ms Badenoch said she removed him after concerns about Post Office governance and "the entire business model".
Labour called on the government to give more specific reasons for its decision.
Mr Staunton left the role - which he had been appointed to in December 2022 - on Saturday. The BBC has approached him for comment.
The Post Office said Mr Staunton had been asked to go by Ms Badenoch.
Appearing on the Sunday with Laura Kuenssberg programme, Ms Badenoch said it was very difficult for someone to be asked to stand down from a position.
But, she continued: "I decided given all of the difficulties the Post Office is having, it's not just about Horizon, it's about the entire business model, how we make it work, that we needed someone who could chair a board that was able to deal with these things effectively."
She explained that one of her priorities was the governance of the Post Office, and said: "That was where my deciding that we needed a new chair of the organisation was important. It just wasn't working."
Ms Badenoch added that there were "various disagreements" within the board "and when I looked at it I thought change of personnel was what was required".
She declined to rule out further personnel changes when appearing on Sky's Sunday Morning with Trevor Phillips and referred to "difficulties" and other board members.
On Sunday with Laura Kuenssberg, she declined to set a deadline to deliver full compensation for those affected - echoing Prime Minister Rishi Sunak's position. But Ms Badenoch promised "we couldn't move any faster than we already have been".
"Setting a deadline is not the priority," she said.
"Getting the money out, getting fair compensation, sorting out the governance of the Post Office, is the critical thing."
It comes as the Post Office is reeling from the fallout from the Horizon scandal, which has been called the biggest miscarriage of justice in UK history.
More than 700 sub-postmasters and sub-postmistresses were prosecuted between 1999 and 2015 after faulty accounting software Horizon made it look as though money was missing from their shops.
Many of those affected were financially ruined.
As part of the role, advertised with a salary of up to £150,000, he had been tasked with leading the board of directors and working to right the wrongs of the Horizon dispute.
Before that, he had worked on the board of companies ranging from ITV to WH Smith.
A Post Office spokesperson said: "On Saturday afternoon, the Post Office was informed that the business and trade secretary had asked Henry Staunton to stand down as chairman of the Post Office.
"We have been advised by the government that they will appoint an interim chairman shortly."
They later said the Post Office was "actively supportive of efforts to compensate postmasters fairly and quickly, and determined in its wish to help right the wrongs of the past".
"As part of this process, in 2021 Post Office changed its governance structure, appointing two serving postmasters to the board, and is making progress on its commitment to replace the upgraded Horizon system with a new and appropriately funded modern solution," they said.
"While progress has been made with over £150m paid in compensation, there remains much work to do to ensure that full justice and compensation are delivered, and to make sure that this can never happen again."
Also appearing on Sunday with Laura Kuenssberg, Labour's shadow business secretary branded the government's decision to ask Mr Staunton to go "quite unusual".
"The government have to tell us why they made this decision seemingly yesterday", Jonathan Reynolds said.
"The person who's going wasn't actually there for the scandal, so there must be specific reasons why they don't have confidence in that person going on."
The Horizon scandal recently came to wider public attention after it was depicted in an ITV drama. and public anger led to the former Post Office boss Paula Vennells handing back her CBE.
Senior figures from the Post Office and the technology firm Fujitsu, which developed the Horizon software, are currently facing questions at a public inquiry over what happened.
The Conservative peer James Arbuthnot, who has campaigned on behalf of sub-post masters, told the BBC he was surprised that Mr Staunton had left the role.
He said the departure was an opportunity to change the culture and governance of the organisation.
Some of his fellow campaigners have accused the Post Office leadership of being too slow to deliver compensation, and too quick to hand out bonuses to senior managers.
Their hope is that the appointment of a new chairman will signal the start of more wide-ranging changes.
Have you been personally affected by the Post Office scandal? Share your experiences by emailing email@example.com.
Please include a contact number if you are willing to speak to a BBC journalist. You can also get in touch in the following ways:
WhatsApp: +44 7756 165803
If you are reading this page and can't see the form you will need to visit the mobile version of the BBC website to submit your question or comment or you can email us at HaveYourSay@bbc.co.uk. Please include your name, age and location with any submission.