Senior Labor ministers have brushed off the government's latest slump in the opinion polls which has wiped out its recent gains.