Multiprocessing Error when Downloading files from FTP
Question: I have been stuck on this particular beauty of a code for a while and I can
figure out why it isn't working. When I run the code below I get a pickling
error and it is always on a different file.

This will download a random number of files and then magically stop. For some
reason the ith file name (or wherever it chooses to stop) is suddenly not
pickleable whereas the other ones before it were. I could see _all_ of them
not being pickleable or none of them but having an arbitrary number not be
pickleable is just weird.

    
    
    n=10
    urls = ["ftp://ftp.sec.gov/{0:s}".format(f) for f in flist[:n]]
    print urls
    from multiprocessing import Pool
    from urllib import urlretrieve
    
    def download(url):
        try:
            file_name = str(url.split('/')[-1])
            print file_name
            return urlretrieve(url, file_name), None
        except Exception as e:
            return None, e
    
    if __name__ == "__main__":
       p = Pool(10) 
       p.map(download, urls) 
    

The error I am getting is:

    
    
    Traceback (most recent call last):
      File "<stdin>", line 3, in <module>
      File "/usr/local/lib64/anaconda/lib/python2.7/multiprocessing/pool.py", line 250, in map
        return self.map_async(func, iterable, chunksize).get()
      File "/usr/local/lib64/anaconda/lib/python2.7/multiprocessing/pool.py", line 554, in get
        raise self._value
    multiprocessing.pool.MaybeEncodingError: Error sending result: '[(('0000950144-94-000788.txt', <mimetools.Message instance at 0x6333878>), None)]'. Reason: 'PicklingError("Can't pickle <type 'cStringIO.StringI'>: attribute lookup cStringIO.StringI failed",)'
    

Does anyone know why the ith element is suddenly not pickleable? Whereas the
other ones were before? I can see it is because it thinks the ith name is a
stringIO but that makes no sense because the ones before it should have been
as well then.


Answer: You're returning a
[mimetools.Message](https://docs.python.org/2/library/mimetools.html#mimetools.Message)
object representing the FTP headers. Either zap it, or turn into an ordinary
string before returning it.

Here's an example of the latter:

    
    
    n=10
    urls = ["ftp://ftp.sec.gov/{0:s}".format(f) for f in flist[:n]]
    print urls
    from multiprocessing import Pool
    from urllib import urlretrieve
    
    def download(url):
        try:
            file_name = str(url.split('/')[-1])
            print file_name
            filename,headers = urlretrieve(url, file_name)
            return (filename, repr(headers), None
        except Exception as e:
            return None, e
    
    if __name__ == "__main__":
       p = Pool(10) 
       p.map(download, urls) 
    

