Python - Extracting files from a large (6GB+) zip file
Question: I have a `Python` script where I need to extract the contents of a ZIP file.
However, the zip file is over 6GB in size.

There is a lot of information about `zlib` and `zipfile` modules, however, I
can't find a single approach that works in my case. I have the code:

    
    
    with zipfile.ZipFile(fname, "r") as z:
            try:
                log.info("Extracting %s " %fname)
                head, tail = os.path.split(fname)
                z.extractall(folder + "/" + tail)
            except zipfile.BadZipfile:
                log.error("Bad Zip file")
            except zipfile.LargeZipFile:
                log.error("Zip file requires ZIP64 functionality but that has not been enabled (i.e., too large)")
            except zipfile.error:
                log.error("Error decompressing ZIP file")
    

I know that I need to set the `allowZip64` to `true` but I'm unsure of how to
do this. Yet, even as is, the `LargeZipFile` exception is not thrown, but
instead the `BadZipFile` exception is. I have no idea why.

Also, is this the best approach to handle extracting a 6GB zip archive???

Update: Modifying the `BadZipfile` exception to this:

    
    
    except zipfile.BadZipfile as inst:
            log.error("Bad Zip file")
            print type(inst)     # the exception instance
            print inst.args      # arguments stored in .args
            print inst
    

shows:

    
    
    <class 'zipfile.BadZipfile'>
    ('Bad magic number for file header',)
    

Update #2:

The full traceback shows

    
    
    BadZipfile                                Traceback (most recent call last)
    <ipython-input-1-8d34a9f58f6a> in <module>()
          6     for member in z.infolist():
          7         print member.filename[-70:],
    ----> 8         f = z.open(member, 'r')
          9         size = 0
         10         while True:
    
    /Users/brspurri/anaconda/python.app/Contents/lib/python2.7/zipfile.pyc in open(self, name, mode, pwd)
        965             fheader = struct.unpack(structFileHeader, fheader)
        966             if fheader[_FH_SIGNATURE] != stringFileHeader:
    --> 967                 raise BadZipfile("Bad magic number for file header")
        968 
        969             fname = zef_file.read(fheader[_FH_FILENAME_LENGTH])
    
    BadZipfile: Bad magic number for file header
    

Running the code:

    
    
    import sys
    import zipfile
    
    with open(zip_filename, 'rb') as zf:
        z = zipfile.ZipFile(zf, allowZip64=True)
        z.testzip()
    
    doesn't output anything.
    


Answer: The problem is that you have a corrupted zip file. I can add more details
about the corruption below, but first the practical stuff:

You can use [this code snippet](http://pastebin.com/ab3JjsHW) to tell you
which member within the archive is corrupted. However, `print z.testzip()`
would already tell you the same thing. And `zip -T` or `unzip` on the command
line should also give you that info with the appropriate verbosity.

* * *

So, what do you do about it?

Well, obviously, if you can get an uncorrupted copy of the file, do that.

If not, if you want to just skip over the bad file and extract everything
else, that's pretty easy—mostly the same code as the snippet linked above:

    
    
    with open(sys.argv[1], 'rb') as zf:
        z = zipfile.ZipFile(zf, allowZip64=True)
        for member in z.infolist():
            try:
                z.extract(member)
            except zipfile.error as e:
                # log the error, the member.filename, whatever
    

* * *

The `Bad magic number for file header` exception message means that `zipfile`
was able to successfully open the zipfile, parse its directory, find the
information for a member, seek to that member within the archive, and read the
header of that member—all of which means you probably have no zip64-related
problems here. However, when it read that header, it did not have the expected
"magic" signature of `PK\003\004`. That means the archive is corrupted.

The fact that the corruption happens to be at exactly 4294967296 implies very
strongly that you had a 64-bit problem somewhere along the chain, because
that's exactly 2**32.

* * *

The command-line `zip`/`unzip` tool has some workarounds to deal with common
causes of corruption that lead to problems like this. it looks like those
workarounds may be working for this archive, given that you get a warning, but
all of the files are apparently recovered. Python's `zipfile` library does not
have those workarounds, and I doubt you want to write your own `zip`-handling
code yourself…

However, that does open the door for two more possibilities:

First, `zip` might be able to _repair_ the zipfile for you, using the `-F` of
`-FF` flag. (Read the manpage, or `zip -h`, or ask at a site like SuperUser if
you need help with that.)

And if all else fails, you can run the `unzip` tool from Python, instead of
using `zipfile`, like this:

    
    
    subprocess.check_output(['unzip', fname])
    

That gives you a lot less flexibility and power than the `zipfile` module, of
course—but you're not using any of that flexibility anyway; you're just
calling `extractall`.

