Later in 724, Wang Shouyi was accused of using witchcraft to try to regain Emperor Xuanzong's favor for his sister Empress Wang. Empress Wang was deposed, and Wang Shouyi was first exiled and then ordered to commit suicide. Zhang Jiazhen, because of his association with Wang Shouyi, was demoted to be the prefect of Tai Prefecture (臺州, roughly modern Taizhou, Zhejiang). However, he was eventually recalled to serve successively as minister of public works (工部尚書, Gongbu Shangshu) and then prefect of Ding Prefecture (定州, roughly modern Baoding, Hebei). He was also created the Marquess of Hedong. After he got to Ding Prefecture, on one occasion, he commissioned a stele for a temple dedicated to the god of Mount Heng, and personally wrote the text on the stele, described to be very beautiful in form. As there were much money donations made to the temple, he believed himself to be responsible for those donations, and so took a small part of it.