The temple began its slow decline in the Heian period. With the rise of the Tendai sect on Mount Hiei with its own kaidan, and with the lack of a supporting sect of its own to provide followers, the Shimotsuke Yakushi-ji's role as primarily a seminary gradually diminished, although it retained the prestige of one of the highest ranking temples in eastern Japan through the end of the Heian period. Although receiving support from Minamoto no Yoritomo at the start of the Kamakura period as a Shingon sect temple, the growing popularity of Japanese Zen among the elites of the Kamakura and subsequent Muromachi period led to the further decline of the temple. In 1339, Ashikaga Takauji ordered that the temple change its name to Ankoku-ji, with the intention that it would be one of the Ankoku-ji that he was building nationwide dedicated to the memory of the dead of the Genkō War of 1333 and to enhance his control over the country. However, the temple continued to use the name Yakushi-ji. The temple was destroyed in the Sengoku period in the wars between the Later Hōjō clan and the Yūki clan.