The Portuguese built a fort in the town . Many other countries also built forts at Accra during the 17th century . Later the Portuguese , followed by the Swedish , Dutch , French , British and Danish built forts in the town by the 17th century .