Later in 710, Emperor Zhongzong died suddenly-a death that traditional historians believed to be a poisoning carried out by his powerful wife Empress Wei and Li Guo'er, so that Empress Wei could eventually be "emperor" like Wu Zetian and Li Guo'er could be crown princess.
Shortly after Emperor Ruizong took the throne, Zhao was demoted to be the prefect of Jiang Prefecture (絳州, part of modern Yuncheng, Shanxi), but was almost immediately recalled to again serve as Zhongshu Shilang and chancellor.