Python 中最快解压 zip 文件的方法
zip
![](/data/attachment/album/201802/24/213511kqgmemzlqmcbw15n.jpg)


假设现在的上下文（LCTT 译注：context，计算机术语，此处意为业务情景）是这样的：一个 zip 文件被上传到一个[Web 服务](https://symbols.mozilla.org)中，然后 Python 需要解压这个 zip 文件然后分析和处理其中的每个文件。这个特殊的应用查看每个文件各自的名称和大小，并和已经上传到 AWS S3 上的文件进行比较，如果文件（和 AWS S3 上的相比）有所不同或者文件本身更新，那么就将它上传到 AWS S3。


[![Uploads today](/data/attachment/album/201802/24/213540t8v50jgvvtt8vqgg.png)](https://cdn-2916.kxcdn.com/cache/e6/dc/e6dc20acd37d94239edbbc0727721e4a.png)


挑战在于这些 zip 文件太大了。它们的平均大小是 560MB 但是其中一些大于 1GB。这些文件中大多数是文本文件，但是其中同样也有一些巨大的二进制文件。不同寻常的是，每个 zip 文件包含 100 个文件但是其中 1-3 个文件却占据了多达 95% 的 zip 文件大小。


最开始我尝试在内存中解压文件，并且每次只处理一个文件。在各种内存爆炸和 EC2 耗尽内存的情况下，这个方法壮烈失败了。我觉得这个原因是这样的。最开始你有 1GB 文件在内存中，然后你现在解压每个文件，在内存中大约就要占用 2-3GB。所以，在很多次测试之后，解决方案是将这些 zip 文件复制到磁盘上（在临时目录 `/tmp` 中），然后遍历这些文件。这次情况好多了但是我仍然注意到了整个解压过程花费了巨量的时间。**是否可能有方法优化呢？**


### 原始函数


首先是下面这些模拟对 zip 文件中文件实际操作的普通函数：



```
def _count_file(fn):
 with open(fn, 'rb') as f:
   return _count_file_object(f)

def _count_file_object(f):
 # Note that this iterates on 'f'.
 # You *could* do 'return len(f.read())'
 # which would be faster but potentially memory
 # inefficient and unrealistic in terms of this
 # benchmark experiment.
 total = 0
 for line in f:
   total += len(line)
 return total

```

这里是可能最简单的另一个函数：



```
def f1(fn, dest):
 with open(fn, 'rb') as f:
   zf = zipfile.ZipFile(f)
   zf.extractall(dest)

 total = 0
 for root, dirs, files in os.walk(dest):
   for file_ in files:
     fn = os.path.join(root, file_)
     total += _count_file(fn)
 return total

```

如果我更仔细地分析一下，我将会发现这个函数花费时间 40% 运行 `extractall`，60% 的时间在遍历各个文件并读取其长度。


### 第一步尝试


我的第一步尝试是使用线程。先创建一个 `zipfile.ZipFile` 的实例，展开其中的每个文件名，然后为每一个文件开始一个线程。每个线程都给它一个函数来做“实质工作”（在这个基准测试中，就是遍历每个文件然后获取它的名称）。实际业务中的函数进行的工作是复杂的 S3、Redis 和 PostgreSQL 操作，但是在我的基准测试中我只需要制作一个可以找出文件长度的函数就好了。线程池函数：



```
def f2(fn, dest):

    def unzip_member(zf, member, dest):
        zf.extract(member, dest)
        fn = os.path.join(dest, member.filename)
        return _count_file(fn)

    with open(fn, 'rb') as f:
        zf = zipfile.ZipFile(f)
        futures = []
        with concurrent.futures.ThreadPoolExecutor() as executor:
            for member in zf.infolist():
                futures.append(
                    executor.submit(
                        unzip_member,
                        zf,
                        member,
                        dest,
                    )
                )
            total = 0
            for future in concurrent.futures.as_completed(futures):
                total += future.result()
    return total

```

**结果：加速 ~10%**


### 第二步尝试


所以可能是 GIL（LCTT 译注：Global Interpreter Lock，一种全局锁，CPython 中的一个概念）阻碍了我。最自然的想法是尝试使用多线程在多个 CPU 上分配工作。但是这样做有缺点，那就是你不能传递一个非可 pickle 序列化的对象（LCTT 译注：意为只有可 pickle 序列化的对象可以被传递），所以你只能发送文件名到之后的函数中：



```
def unzip_member_f3(zip_filepath, filename, dest):
    with open(zip_filepath, 'rb') as f:
        zf = zipfile.ZipFile(f)
        zf.extract(filename, dest)
    fn = os.path.join(dest, filename)
    return _count_file(fn)



def f3(fn, dest):
    with open(fn, 'rb') as f:
        zf = zipfile.ZipFile(f)
        futures = []
        with concurrent.futures.ProcessPoolExecutor() as executor:
            for member in zf.infolist():
                futures.append(
                    executor.submit(
                        unzip_member_f3,
                        fn,
                        member.filename,
                        dest,
                    )
                )
            total = 0
            for future in concurrent.futures.as_completed(futures):
                total += future.result()
    return total

```

**结果: 加速 ~300%**


### 这是作弊


使用处理器池的问题是这样需要存储在磁盘上的原始 `.zip` 文件。所以为了在我的 web 服务器上使用这个解决方案，我首先得要将内存中的 zip 文件保存到磁盘，然后调用这个函数。这样做的代价我不是很清楚但是应该不低。


好吧，再翻翻看又没有损失。可能，解压过程加速到足以弥补这样做的损失了吧。


但是一定记住！这个优化取决于使用所有可用的 CPU。如果一些其它的 CPU 需要执行在 `gunicorn` 中的其它事务呢？这时，这些其它进程必须等待，直到有 CPU 可用。由于在这个服务器上有其他的事务正在进行，我不是很确定我想要在进程中接管所有其他 CPU。


### 结论


一步一步地做这个任务的这个过程感觉挺好的。你被限制在一个 CPU 上但是表现仍然特别好。同样地，一定要看看在`f1` 和 `f2` 两段代码之间的不同之处！利用 `concurrent.futures` 池类你可以获取到允许使用的 CPU 的个数，但是这样做同样给人感觉不是很好。如果你在虚拟环境中获取的个数是错的呢？或者可用的个数太低以致无法从负载分配获取好处并且现在你仅仅是为了移动负载而支付营运开支呢？


我将会继续使用 `zipfile.ZipFile(file_buffer).extractall(temp_dir)`。这个工作这样做已经足够好了。


### 想试试手吗？


我使用一个 `c5.4xlarge` EC2 服务器来进行我的基准测试。文件可以从此处下载：



```
wget https://www.peterbe.com/unzip-in-parallel/hack.unzip-in-parallel.py
wget https://www.peterbe.com/unzip-in-parallel/symbols-2017-11-27T14_15_30.zip

```

这里的 `.zip` 文件有 34MB。和在服务器上的相比已经小了很多。


`hack.unzip-in-parallel.py` 文件里是一团糟。它包含了大量可怕的修正和丑陋的代码，但是这只是一个开始。




---


via: <https://www.peterbe.com/plog/fastest-way-to-unzip-a-zip-file-in-python>


作者：[Peterbe](https://www.peterbe.com/) 译者：[Leemeans](https://github.com/leemeans) 校对：[wxy](https://github.com/wxy)


本文由 [LCTT](https://github.com/LCTT/TranslateProject) 原创编译，[Linux中国](https://linux.cn/) 荣誉推出


OK
So the context is this; a zip file is uploaded into a [web service](https://symbols.mozilla.org) and Python then needs extract that and analyze and deal with each file within. In this particular application what it does is that it looks at the file's individual name and size, compares that to what has already been uploaded in AWS S3 and if the file is believed to be different or new, it gets uploaded to AWS S3.


The challenge is that these zip files that come in are huuuge. The average is 560MB but some are as much as 1GB. Within them, there are mostly plain text files but there are some binary files in there too that are huge. It's not unusual that each zip file contains 100 files and 1-3 of those make up 95% of the zip file size.

At first I tried unzipping the file, in memory, and deal with one file at a time. That failed spectacularly with various memory explosions and EC2 running out of memory. I guess it makes sense. First you have the 1GB file in RAM, then you unzip each file and now you have possibly 2-3GB all in memory. So, the solution, after much testing, was to dump the zip file to disk (in a temporary directory in `/tmp`

) and then iterate over the files. This worked much better but I still noticed the whole unzipping was taking up a huge amount of time. **Is there perhaps a way to optimize that?**

### Baseline function

First it's these common functions that simulate actually doing something with the files in the zip file:

```
def _count_file(fn):
with open(fn, 'rb') as f:
return _count_file_object(f)
def _count_file_object(f):
# Note that this iterates on 'f'.
# You *could* do 'return len(f.read())'
# which would be faster but potentially memory
# inefficient and unrealistic in terms of this
# benchmark experiment.
total = 0
for line in f:
total += len(line)
return total
```


Here's the simplest one possible:

```
def f1(fn, dest):
with open(fn, 'rb') as f:
zf = zipfile.ZipFile(f)
zf.extractall(dest)
total = 0
for root, dirs, files in os.walk(dest):
for file_ in files:
fn = os.path.join(root, file_)
total += _count_file(fn)
return total
```


If I analyze it a bit more carefully, I find that it spends about 40% doing the `extractall`

and 60% doing the looping over files and reading their full length.

### First attempt

My first attempt was to try to use threads. You create an instance of `zipfile.ZipFile`

, extract every file name within and start a thread for each name. Each thread is given a function that does the "meat of the work" (in this benchmark, iterating over the file and getting its total size). In reality that function does a bunch of complicated S3, Redis and PostgreSQL stuff but in my benchmark I just made it a function that figures out the total length of file. The thread pool function:

```
def f2(fn, dest):
def unzip_member(zf, member, dest):
zf.extract(member, dest)
fn = os.path.join(dest, member.filename)
return _count_file(fn)
with open(fn, 'rb') as f:
zf = zipfile.ZipFile(f)
futures = []
with concurrent.futures.ThreadPoolExecutor() as executor:
for member in zf.infolist():
futures.append(
executor.submit(
unzip_member,
zf,
member,
dest,
)
)
total = 0
for future in concurrent.futures.as_completed(futures):
total += future.result()
return total
```


**Result: ~10% faster**

### Second attempt

So perhaps the GIL is blocking me. The natural inclination is to try to use multiprocessing to spread the work across multiple available CPUs. But doing so has the disadvantage that you can't pass around a non-pickleable object so you have to send just the filename to each future function:

```
def unzip_member_f3(zip_filepath, filename, dest):
with open(zip_filepath, 'rb') as f:
zf = zipfile.ZipFile(f)
zf.extract(filename, dest)
fn = os.path.join(dest, filename)
return _count_file(fn)
def f3(fn, dest):
with open(fn, 'rb') as f:
zf = zipfile.ZipFile(f)
futures = []
with concurrent.futures.ProcessPoolExecutor() as executor:
for member in zf.infolist():
futures.append(
executor.submit(
unzip_member_f3,
fn,
member.filename,
dest,
)
)
total = 0
for future in concurrent.futures.as_completed(futures):
total += future.result()
return total
```


**Result: ~300% faster**

### That's cheating!

The problem with using a pool of processors is that it requires that the original `.zip`

file exists on disk. So in my web server, to use this solution, I'd first have to save the in-memory ZIP file to disk, then invoke this function. Not sure what the cost of that it's not likely to be cheap.

Well, it doesn't hurt to poke around. Perhaps, it could be worth it if the extraction was significantly faster.

But remember! This optimization depends on using up as many CPUs as it possibly can. What if some of those other CPUs are needed for something else going on in `gunicorn`

? Those other processes would have to patiently wait till there's a CPU available. Since there's other things going on in this server, I'm not sure I'm willing to let on process take over all the other CPUs.

### Conclusion

Doing it serially turns out to be quite nice. You're bound to one CPU but the performance is still pretty good. Also, just look at the difference in the code between `f1`

and `f2`

! With `concurrent.futures`

pool classes you can cap the number of CPUs it's allowed to use but that doesn't feel great either. What if you get the number wrong in a virtual environment? Of if the number is too low and don't benefit any from spreading the workload and now you're just paying for overheads to move the work around?

I'm going to stick with `zipfile.ZipFile(file_buffer).extractall(temp_dir)`

. It's good enough for this.

### Want to try your hands on it?

I did my benchmarking using a `c5.4xlarge`

EC2 server. The files can be downloaded from:

wget https://www.peterbe.com/unzip-in-parallel/hack.unzip-in-parallel.py wget https://www.peterbe.com/unzip-in-parallel/symbols-2017-11-27T14_15_30.zip

The `.zip`

file there is 34MB which is relatively small compared to what's happening on the server.

The `hack.unzip-in-parallel.py`

is a hot mess. It contains a bunch of terrible hacks and ugly stuff but hopefully it's a start.

## Comments

Post your own commentFebruary 1, 2018AnonymousConsider using bzip2, xz or something similar which can be both compressed and decompressed in parallel. See pbzip2 and pixz. (Zip does not lend itself to this, so pigz would not help you too much under most circumstances.)


Choose the right UNIX tool, subprocesses and pipes are the way to go here and pipes can go across network boundaries as well - rely on the deep history of computing.

Peter BengtssonFebruary 1, 2018Zip files can be decompressed in parallel too. You first ask the zip file for its file contents (fast), then you start parallel processes (or threads or whatever) that each get a file name to extract.


My multiprocess example above is a proof of that.

Martin BammerFebruary 1, 2018You should not use ThreadPoolExecutor, because it has a relatively high overhead and is very slow. Especially if you use submit. Use a different thread pool implementation or if you want to stick with ThreadPoolExecutor use map. A much faster solution is the good old multiprocessing.pool.ThreadPool. Or you could also try my fastthreadpool module (https://github.com/brmmm3/fastthreadpool). This module is faster than ThreadPool and much faster than ThreadPoolExecutor. It also has the advantage that you can use generator functions as worker which is very useful in certain situations. For example code please look into benchmark.py. The doc directory contains some benchmark files which show the overhead difference between the 3 thread pool implementations. I'm also working on a fastprocesspool module but this is still not finished and buggy, First tests have shown that this module is also faster than multiprocessing.Pool and much fast than ProcessPoolExecutor.

Peter BengtssonFebruary 4, 2018What's the alternative to submit?

And multiprocessing.Pool might be marginally faster, but isn't the bulk of the computation still in the actual unzipping?

Martin BammerFebruary 5, 2018The bulk is indeed in unzipping. But if you've an archive with many small files the overhead of the pool can be 10% or more. And this is much for just handling a pool of threads. The alternative is to use map, where you have to prepare an iterable before calling map. Another alternative is to switch to a faster pool implementation.

The module zipfile is completely written in Python, which comes with a relatively big overhead at Python level, which in turn means the GIL is locked relatively long. The result is a low level of parallelisation. I'm currently writing an archiving tool which uses msgpack and zstd. Both libraries have a very thin Python layer and parallelisation with threads is very good. I get nearly 100% CPU load. The results currently are ~4 faster than zip and compression ratio between zip and lzma. When the tool is finished I'll release it for the public.

Peter BengtssonFebruary 6, 2018If you have an example of using fastthreadpool that I can use instead of concurrent.futures.ThreadPoolExecutor or concurrent.futures.ProcessPoolExecutor then I'll try to run a benchmark with that too.

MasklinnFebruary 4, 2018Peter, if you don't need the entire content of the file in memory (which I guess you don't since you're apparently fine with reading it from disk despite in-memory decompression blowing up) *and* you don't need seekability, you may have missed the `open` method on zipfiles. You'll still be paying for the decompression work, and as noted previously you can only access data sequentially (no seeking backwards) but it *is* lazy, you ask for a 10k chunk of a file, you get 10k in memory (plus some overhead from Python).

Peter BengtssonFebruary 4, 2018Doing an open requires all things to be in-memory. It's a bit faster but would require me to have a much beefier server, which might be the best course of action actually.

MasklinnFebruary 5, 2018> Doing an open requires all things to be in-memory.




It only requires that the zip file be accessible, either in-memory or on-disk.

Your introduction says that you originally had the zip file *and* the decompressed files in memory, ZipFile.open avoids the latter. Furthermore the small bit of code you post in the conclusion seems to hint that you still have the zip file in memory when you do the extraction.

And again, you can use ZipFile.open just fine if you have the zip file on-disk, it's still lazy.

jfsFebruary 7, 2018It might be more efficient to count bytes by reading chunks instead of lines: `_count_file_object = lambda file: sum(map(len, iter(lambda: file.read(1<<15), b'')))` (assuming you don't want `os.path.getsize(filename)` or similar). Related: https://stackoverflow.com/questions/9371238/why-is-reading-lines-from-stdin-much-slower-in-c-than-python

mikeSeptember 12, 2018awesome! thanks a lot!

October 21, 2020AnonymousPeter, Can we do similar improvements with ZipFile.write() if it irequired to zip folders and files in parallel?

Thanks.

May 6, 2022AnonymousThanks for this code. I have something similar based on it running in production and I hit a wall.



Decompressing archives containing 65688 files will result to be very very slow.

To have a significant speed boost you can try something similar to this https://github.com/ITISFoundation/osparc-simcore/blob/cd9640c0072fcd2536a0ae11cd602e7b7a9ea3ee/packages/service-library/src/servicelib/archiving_utils.py#L45

It basically avoids reading the entire file list from the zipfile inside the background process!

ViolaOctober 30, 2022How should I increase the instance number or thread number?because I have many zip file to extract.

Peter BengtssonOctober 31, 2022From https://docs.python.org/3/library/concurrent.futures.html#concurrent.futures.ProcessPoolExecutor


"If max_workers is None or not given, it will default to the number of processors on the machine."

So by default, it will use as many CPUs as your computer possibly has.