The Xiaotaoyuan Mosque was originally built in 1917 by Jin Ziyun, the Director of the Shanghai Islamic Board of Directors, who had bought the land.
After the founding of the People's Republic of China on 1 October 1949, Shanghai Mayor Chen Yi inspected the mosque and approved appropriate funds for its maintenance.