Olympic chief Thomas Bach warned against allowing the Games to "descend into a marketplace of demonstrations" on Saturday after criticism of a rule banning protests by athletes.
The International Olympic Committee president has come under fire for the ruling, announced in a year which has seen widespread support by athletes for the Black Lives Matter movement.
But Bach, who won a team fencing gold medal at the 1976 Olympics, said he had learned first-hand about the "political impotence of sport" when West Germany boycotted the 1980 Games over the Soviet invasion of Afghanistan.
"The athletes personify the values of excellence, solidarity and peace. They express this inclusiveness and mutual respect also by being politically neutral on the field of play and during the ceremonies," Bach wrote in British newspaper The Guardian.
"At times, this focus on sport needs to be reconciled with the freedom of speech that all athletes also enjoy at the Olympic Games. This is the reason there are rules for the field of play and the ceremonies protecting this spirit of sport.
"The unifying power of the Games can only unfold if everyone shows respect for and solidarity to one another. Otherwise, the Games will descend into a marketplace of demonstrations of all kinds, dividing and not uniting the world."
Earlier this month, World Athletics chief and newly enshrined IOC member Sebastian Coe backed athletes' right to "take the knee" on the Olympic podium, as long as it was done with "complete respect" for other competitors.
Former NFL quarterback Colin Kaepernick made headlines when he began kneeling during the US national anthem in 2016 to protest police brutality against Black Americans and other minorities.
He was ostracised by the league and criticised by US President Donald Trump, but in June the NFL sanctioned "peaceful protests" and world football's governing body FIFA has urged leagues to use "common sense" when deciding whether to discipline political activism.
In June, US athletes and 1968 Mexico Games icon John Carlos, famously kicked out of the event for raising his fist in a black power salute, called on the IOC to scrap the ban on protests.
"Athletes will no longer be silenced," they wrote.
Last month, hundreds of human rights and civil society groups urged the IOC to strip China of the 2022 Beijing Winter Olympics following its clampdown in Hong Kong and mass internment of Uighur Muslims in Xinjiang.