Treasurer Wayne Swan is refusing to give up on his budget surplus promise despite growing calls to scrap it.