Python ftp.retrbinary() does not work when called parallelized in function
Question: I am using a script to retrieve data from a ftp-server. As I want to
parallelize the download, `ftp.retrbinary` is called within a function.

Atm the working code looks like this:

    
    
    from ftplib import FTP
    
    def download_file(file_in, target_file):
        ftp.retrbinary('RETR '+file_in, open(target_file, 'wb').write)
        return 0
    
    ftp = FTP(FTP_HOST)
    ftp.login(FTP_USER, FTP_PASS)
    ftp.cwd(FTP_PATH)
    
    for file_input in files_to_check:
        download_file(target_dir,file_input)
    

As soon as I want to download parallelly, the download just gets stuck and no
data is transferred:

    
    
    from ftplib import FTP
    from joblib import Parallel, delayed
    
    def download_file(file_in, target_file):
        ftp.retrbinary('RETR '+file_in, open(target_file, 'wb').write)
        return 0
    
    ftp = FTP(FTP_HOST)
    ftp.login(FTP_USER, FTP_PASS)
    ftp.cwd(FTP_PATH)
    
    Parallel(n_jobs=2)(delayed(download_file)(target_dir,file_input) for file_input in files_to_check)
    

Does anybody have an idea why `ftp.retrbinary` does not work for parallel
downloads?


Answer: You cannot use one FTP session for multiple parallel transfers. The FTP
protocol does not support that (contrary to the SFTP for example).

You have to open a separate FTP session for each parallel job.

