There are no records which confirm that the Byzantines used the fort. In the first half of the 15th century, during the reign of Serbian despot Stefan Lazarević, it was expanded into a proper fortress. The data on the fort is scarce. When German naturalist and traveler Felix Philipp Kanitz visited the area in the second half of the 19th century, he made several drawings and Žrnov and wrote that Serbian emperors built the fort on the foundations of the Roman outpost on the top of the mountain, without specifying which emperors or when. During the Middle Ages, the cinnabarite was used for the fresco paintings and was exported to Greece. The expanded fortress became known as Žrnov Grad (Žrnov Town). It is believed that it got its name because of the large grinding stones for crushing the ores, which are called žrvanj in Serbian. The mountain Avala itself was called Žrnovica in the Middle Ages.