A Sydney court has refused bail to the man known as the Bondi caveman, who is accused of raping a woman last night.

Peter James Millhouse has lived in a makeshift camp on the cliffs overlooking Bondi Beach for almost 10 years.

He has been charged with sexual assault and faced Waverley Local court today.

The court heard that he met a woman who was visiting the beach for the Sculptures by the Sea exhibition and invited her back to his camp.

It is alleged he raped her, but the defence said he strongly denies the charge.

Magistrate Lee Gilmour says it seems the issue is one of consent.

The 54-year-old, who has a long greying beard, said nothing as his lawyer applied for bail, arguing Millhouse has never been charged with an offence like this before.

The court heard that Millhouse has no mental health problems and survives on handouts of food and poetry that he writes.

The magistrate refused bail, citing protection to the community.

Millhouse made worldwide headlines two-and-a-half years ago when Waverley Council backed down from plans to evict him from the cliff.

The prosecution said today that the accused has never been given permission to live in his camp.

The magistrate replied: "Maybe they should have removed him long before now."

But acting police superintendent Christine George says it is now up to Waverley Council to decide what will happen to Millhouse's camp.

"Eviction of people who are living homeless in this area isn't really a police matter; we only get involved when it becomes a criminal matter," she said.

"All we're concerned about is the safe environment of Bondi and we provide regular taskings down the Bondi Beach area to ensure the environment's secure for anyone who visits Bondi Beach."