Recently, with the maturity of cloud services and the development of distributed computing, increasing images and video data are stored in the cloud. However, cloud services are generally provided by the third party entities. In addition, the images and video which are stored in the cloud mostly depend on the security of the cloud servers, while the data are not encrypted, which is a great threat to the users' privacy. In this paper, we propose a method of privacy preserving background extraction of video surveillance in multiple cloud servers based on Chinese Remainder Theorem (CRT). The client uses the CRT to divide a video frame into several encrypted frames, which are sent to the corresponding cloud servers separately. The cloud servers conduct the algorithm of Visual Background Extractor (ViBe) in the encrypted video frames, hereafter, the results obtained are transmitted to the end server, in which the results are decrypted and combined to get the final extraction results in original videos. Finally, the results are transmitted back to the client. The proposed method has several advantages: (1) Based on our encryption method, the original extraction method in the original videos need not be changed; (2) Each cloud server learns only a portion of the video frame information, yet they cannot recover the original video even if the data is leaked; (3) Multiple cloud servers can improve the security of data and enhance the processing efficiency. The experimental results on several video surveillance datasets shows that our method can correctly extract the background as the original ViBe algorithm does while protecting the privacy of the client video data.