 In or around 646, Fang was again, for reasons unknown in history but described to be minor reasons, removed from his post and returned to his mansion.After Chu Suiliang submitted a petition stating that Fang's faithfulness should not be forgotten, Emperor Taizong visited Fang's mansion – a visit that Fang anticipated and had cleaned his house for – and he had Fang accompany him on the imperial wagon back to palace.Fang was said to be so influential in Emperor Taizong's decision-making by this stage that an amusing incident occurred in 647, involving the official Li Wei (李緯).