Python and zlib: Terribly slow decompressing concatenated streams
Question: I've been supplied with a zipped file containing multiple individual streams
of compressed XML. The compressed file is 833 mb.

If I try to decompress it as a single object, I only get the first stream
(about 19 kb).

I've modified the following code supplied as a answer to an [older
question](http://stackoverflow.com/questions/12147484/extract-zlib-compressed-
data-from-binary-file-in-python) to decompress each stream and write it to a
file:

    
    
    import zlib
    
    outfile = open('output.xml', 'w')
    
    def zipstreams(filename):
        """Return all zip streams and their positions in file."""
        with open(filename, 'rb') as fh:
            data = fh.read()
        i = 0
        print "got it"
        while i < len(data):
            try:
                zo = zlib.decompressobj()
                dat =zo.decompress(data[i:])
                outfile.write(dat)
                zo.flush()
                i += len(data[i:]) - len(zo.unused_data)
            except zlib.error:
                i += 1
        outfile.close()
    
    zipstreams('payload')
    infile.close()
    

This code runs and produces the desired result (all the XML data decompressed
to a single file). The problem is that it takes several days to work!

Even though there are tens of thousands of streams in the compressed file, it
still seems like this should be a much faster process. Roughly 8 days to
decompress 833mb (estimated 3gb raw) suggests that I'm doing something very
wrong.

Is there another way to do this more efficiently, or is the slow speed the
result of a read-decompress-write---repeat bottleneck that I'm stuck with?

Thanks for any pointers or suggestions you have!


Answer: It's hard to say very much without more specific knowledge of the file format
you're actually dealing with, but it's clear that your algorithm's handling of
substrings is quadratic-- not a good thing when you've got tens of thousands
of them. So let's see what we know:

You say that the vendor states that they are

> using the standard zlib compression library.These are the same compression
> routines on which the gzip utilities are built.

From this we can conclude that the component streams are in **raw zlib
format,** and are _not_ encapsulated in a gzip wrapper (or a PKZIP archive, or
whatever). The authoritative documentation on the ZLIB format is here:
<http://tools.ietf.org/html/rfc1950>

So let's assume that your file is exactly as you describe: **A 32-byte header,
followed by raw ZLIB streams concatenated together, without any other stuff in
between.** (**Edit:** That's not the case, after all).

Python's [zlib documentation](http://docs.python.org/2/library/zlib.html)
provides a `Decompress` class that is actually pretty well suited to churning
through your file. It includes an attribute `unused_data` whose
[documentation](http://docs.python.org/2/library/zlib.html#zlib.Decompress.unused_data)
states clearly that:

> The only way to determine where a string of compressed data ends is by
> actually decompressing it. This means that when compressed data is contained
> part of a larger file, you can only find the end of it by reading data and
> feeding it followed by some non-empty string into a decompression object’s
> decompress() method until the unused_data attribute is no longer the empty
> string.

So, this is what you can do: Write a loop that reads through `data`, say, one
block at a time (no need to even read the entire 800MB file into memory). Push
each block to the `Decompress` object, and check the `unused_data` attribute.
When it becomes non-empty, you've got a complete object. Write it to disk,
create a new decompress object and initialize iw with the `unused_data` from
the last one. This just might work (untested, so check for correctness).

**Edit:** Since you do have other data in your data stream, I've added a
routine that aligns to the next ZLIB start. You'll need to find and fill in
the two-byte sequence that identifies a ZLIB stream in _your_ data. (Feel free
to use your old code to discover it.) While there's no fixed ZLIB header in
general, it should be the same for each stream since it consists of [protocol
options and flags,](http://tools.ietf.org/html/rfc1950) which are presumably
the same for the entire run.

    
    
    import zlib
    
    # FILL IN: ZHEAD is two bytes with the actual ZLIB settings in the input
    ZHEAD = CMF+FLG  
    
    def findstart(header, buf, source):
        """Find `header` in str `buf`, reading more from `source` if necessary"""
    
        while buf.find(header) == -1:
            more = source.read(2**12)
            if len(more) == 0:  # EOF without finding the header
                return ''
            buf += more
    
        offset = buf.find(header)
        return buf[offset:]
    

You can then advance to the start of the next stream. I've added a
`try`/`except` pair since the same byte sequence might occur outside a stream:

    
    
    source = open(datafile, 'rb')
    skip_ = source.read(32) # Skip non-zlib header
    
    buf = ''
    while True:
        decomp = zlib.decompressobj()
        # Find the start of the next stream
        buf = findstart(ZHEAD, buf, source)
        try:    
            stream = decomp.decompress(buf)
        except zlib.error:
            print "Spurious match(?) at output offset %d." % outfile.tell(),
            print "Skipping 2 bytes"
            buf = buf[2:]
            continue
    
        # Read until zlib decides it's seen a complete file
        while decomp.unused_data == '':
            block = source.read(2**12)
            if len(block) > 0:       
                stream += decomp.decompress(block)
            else:
                break # We've reached EOF
    
        outfile.write(stream)
        buf = decomp.unused_data # Save for the next stream
        if len(block) == 0:
            break  # EOF
    
    outfile.close()
    

PS 1. If I were you I'd write each XML stream into a separate file.

PS 2. You can test whatever you do on the first MB of your file, till you get
adequate performance.

