Yang's sisters—the Ladies of Guo, Han, and Qin—and Yang Xian and Yang Qi were to attend Emperor Xuanzong and Consort Yang on tours to Huaqing Palace (華清宮, known for its hot springs), their trains would gather at Yang Guozhong's mansion, with each of the five households distinguished by a different color, causing a glamorous display of colors. Yang Guozhong would lead the train ahead with his own guards and the banners as the military governor of Jiannan. In spring 754, Yang suggested to Emperor Xuanzong that An was set on rebelling, and that if Emperor Xuanzong summoned him to Chang'an, he would surely not come—but when Emperor Xuanzong then did summon An, An immediately arrived at Chang'an and accused Yang of false accusations. Thereafter, Emperor Xuanzong would no longer believe any suggestions that An was plotting rebellion, despite Li Heng agreeing with Yang's assessment on this issue. Meanwhile, he considered promoting An to be chancellor, but Yang opposed, and this did not occur, and An subsequently returned to Fanyang. Meanwhile, Yang was having conflicts with Chen as well, and he forced Chen to resign and then recommended Wei Jiansu to replace Chen. During a subsequent flood that caused much damage, Yang, not wanting any reports of damage, insisted that there was no damage—and when the officials Li Xian and Fang Guan nevertheless submitted damage reports, he had them demoted. He also viewed the official Wei Zhi () as a threat and had Wei accused of corruption—and when Wei subsequently bribed Ji, who was by then allied with An, for help, Yang also discovered this and reported it. As a result, Wei and Ji were demoted, and An subsequently submitted a petition on their behalf, claiming that the accusations from Yang were false, but Emperor Xuanzong took no actions against either An or Yang. In spring 755, however, the matters were beginning to come to a head. When An submitted a petition to have thirty-two non-Han generals under him replace Han generals, Emperor Xuanzong immediately agreed, despite stern arguments from Yang and Wei Jiansu that this was a sign of impending rebellion. Yang and Wei then suggested that An be promoted to be chancellor, and that his three commands be divided between his three deputies; Emperor Xuanzong initially agreed, but after the edict was drafted, Emperor Xuanzong tabled it and instead sent the eunuch Fu Qiulin () to send fresh fruits