French authorities say the migrants must move to cleaner and safer shipping containers on another part of the site.
But many fear this will mean they will have to stay in France, and give up their hopes of travelling to Britain.
Overnight, riot police fired tear gas at migrants who were throwing stones at the demolition squads.
Here's some advice if anything in the news upsets you.
The authorities believe some 1,000 migrants will be affected by the removal plan while aid agencies say the number of people living there is much higher.
A lot of migrants and refugees have been leaving their homes in places like Syria, Afghanistan and Africa, where there is fighting, over the last year.
Many travel great distances in search of a better life in Europe and some have ended up living in the makeshift camp in Calais hoping to make their way to the UK.
Workers in high-visibility jackets could be seen tearing down structures and dumping material in a skip as police stood by on Monday.
About 100 huts were taken down.
Earlier, Good Chance, a theatre group which works in the camp, said police were preventing protestors from entering the camp.
French officials say public areas in the camp, such as places of worship and schools, will not be affected and describe the clearance as a "humanitarian operation".
Conditions in the southern sector are filthy but many residents have told the BBC that they do not want to leave.
Demolition teams have moved in to the port of Calais in France on Tuesday to break down more shelters in the migrant camp known as the "Jungle".