An anti-racism activist has said he was forced to leave the Belgian town of Ath on Sunday after a row with local authorities about a folkloric festival.

A carnival float featuring a white man in blackface called “the savage” featured on Sunday as part of Ath’s annual festival, despite calls to abandon the practice, which campaigners say is an act of “symbolic violence” towards black people in Belgium.

Mouhad Reghif, a spokesman for the Brussels Panthers, who is leading a campaign against blackface, said he had been expelled from the town on the mayor’s orders after being spotted by the police. He said officers had followed him for 30km on the motorway to make sure he left.

The mayor of Ath, Bruno Lefèbvre, said Reghif was spotted on Ath’s main square by security services at 9.30am on Sunday. “Given the comments that have been made on the web these past weeks, comments that have shocked the Athoise population, I explained to him that we could not guarantee his safety. The interview passed well and he left the town,” Lefèbvre told Belgian media.

The row blew up after the Brussels Panthers petitioned Unesco to remove the centuries-old Ath street festival from its “intangible cultural heritage of humanity” list, unless organisers abandoned the practice of parading a white man in blackface wearing chains.

Local organisers insist the savage character is not intended to depict a black man and is held in great affection. On the streets of Ath on Sunday the character was applauded by locals, while some wore T-shirts that said: “I am savage”.

A local historian, Laurent Dubuisson, said charges of racism at Ath were misplaced.

In a recent interview with the Guardian, he suggested that the “savage” – as well as a devil character in black makeup – could “evolve”, if townspeople wished it. “This is no more a medieval festival but a festival of the 21st century. Perhaps in 15 years or 20 years, the population will decide that the savage no longer has a place, or that the devil no longer has a place.”

In a letter to the Brussels Panthers, Unesco said every event inscribed on the cultural heritage list had to conform to a charter ensuring “mutual respect between communities, groups and individuals”.

The UN organisation said it was planning in December to review the Unesco status of the Aalst carnival after a furore over antisemitism. Two Jewish organisations filed complaints to Unesco after a float in the Flemish city festival in March featured grotesque caricatures of Jewish men surrounded by money.

Unesco did not reveal its views on the Ath festival, but said the complaint had been passed on to Belgian and French delegations within its organisation.