The expansion of Hohenneuffen Castle into a fortress began in the 14th century. However, the most important alterations to the castle structure were conducted by Duke Ulrich of Württemberg in the first half of the 16th century. Barbicans, round towers, bastions, a building for the commanding officer, casemates, stables, an armoury as well as two cisterns were built. Essentially, these fortification did not change for the next two hundred years. While the fortress had to surrender to troops from the Swabian League in 1519, it withstood the insurgent peasants' attempt to take it during the German Peasants' War of 1524/25.