Fastest way to extract tar files using Python
Question: I have to extract hundreds of tar.bz files each with size of 5GB. So tried the
following code:

    
    
    import tarfile
    from multiprocessing import Pool
    
    files = glob.glob('D:\\*.tar.bz') ##All my files are in D
    for f in files:
    
       tar = tarfile.open (f, 'r:bz2')
       pool = Pool(processes=5)
    
       pool.map(tar.extractall('E:\\') ###I want to extract them in E
       tar.close()
    

But the code has type error: TypeError: map() takes at least 3 arguments (2
given)

How can I solve it? Any further ideas to accelerate extracting?


Answer: Define a function that extract a single tar file. Pass that function and a tar
file list to `multiprocessing.Pool.map`:

    
    
    from functools import partial
    import glob
    from multiprocessing import Pool
    import tarfile
    
    
    def extract(path, dest):
        with tarfile.open(path, 'r:bz2') as tar:
            tar.extractall(dest)
    
    if __name__ == '__main__':
        files = glob.glob('D:\\*.tar.bz')
        pool = Pool(processes=5)
        pool.map(partial(extract, dest='E:\\'), files)
    

