Duke Karl Alexander of Württemberg planned to have Hohenneuffen altered into a fortress following the French model. Yet he died before the task was completed and his successor, Karl Eugen abandoned the plan due to the high costs and the doubtful military benefit. In 1793 it was decided to raze the castle and to sell off the building materials. The castle went out of use in 1795 and was finally destined for destruction in 1801. The inhabitants of the surrounding area were happy to utilise the cheap building materials. Only from 1830 onwards the remains of the castle were safeguarded from further destruction and in the 1860s public access to the ruin was allowed. In 1862 an inn was established in one of the buildings in the upper bailey.