Malcolm Turnbull has moved to clarify his comments in a speech widely interpreted as having a shot at disgruntled conservatives in the Liberal Party such as Tony Abbott.
In the speech to the centre right think tank Policy Exchange in London, Mr Turnbull questioned the use of labels "conservative" and "liberal".
But he noted Liberal founder Sir Robert Menzies "went to great pains not to call his new political party, consolidating the centre right of Australian politics, 'conservative' - but rather the Liberal Party which he firmly anchored in the centre of Australian politics".'
The comments have come at a time of tension between Liberal conservative MPs and moderates, especially over issues such as same-sex marriage and climate policy.
Mr Abbott, who is understood to be on leave overseas, recently vowed to be a strong conservative voice and former Liberal senator Cory Bernadi split earlier this year to set up his own Australian Conservatives party.
Liberal conservative Eric Abetz called for mutual respect between party conservatives and "small-l liberals".
"The Liberal party is and has always been a train running on small-l liberal and conservative tracks - unless both are tended to the whole train will derail," he said in a statement.
Senator Bernadi's Australian Conservatives and One Nation's Pauline Hanson have used the comments to launch recruitment campaigns as the "true" conservative parties.
Mr Turnbull clarified his comments in the wake of media reports of a leaked copy of his speech and a torrent of social media outrage.
He noted he had deliberately used the phrase "sensible centre" coined by Tony Abbott in the speech and believed most Liberal Party members - including himself - embraced both the the terms liberal and conservative.
"They are brought together and indeed they are shared by most of us, we share both traditions, they are not exclusive," Mr Turnbull said.
"But the focus has got to be on delivering for the people you represent ... not ideology and politics."
Mr Shorten said the Liberal party was in crisis.
"It's getting to a point where I think when it comes down to the fight between Tony Abbott and Malcolm Turnbull, one of them has got to go," he said.
Liberal campaign guru Lynton Crosby said Mr Turnbull was right to put an end to labels.
"It's who you govern for rather than what you call yourself that's most important of all," Sir Lynton told a luncheon in Sydney.
Meanwhile, the prime minister will have an audience with the Queen at Buckingham Palace on Tuesday .
The meeting will be the final formal event for the prime minister before he heads home.
Mr Turnbull will meet with business leaders to discuss trade and investment, as well as hold a roundtable discussion on education and innovation.