Create and stream a large archive without storing it in memory or on disk
Question: I want to allow users to download an archive of multiple large files at once.
However, the files and the archive may be too large to store in memory or on
disk on my server (they are streamed in from other servers on the fly). I'd
like to generate the archive as I stream it to the user.

I can use Tar or Zip or whatever is simplest. I am using django, which allows
me to return a generator or file-like object in my response. This object could
be used to pump the process along. However, I am having trouble figuring out
how to build this sort of thing around the zipfile or tarfile libraries, and
I'm afraid they may not support reading files as they go, or reading the
archive as it is built.

This answer on [converting an iterator to a file-like
object](http://stackoverflow.com/questions/6657820/python-convert-an-iterable-
to-a-stream) might help. `tarfile#addfile` takes an iterable, but it appears
to immediately pass that to `shutil.copyfileobj`, so this may not be as
generator-friendly as I had hoped.


Answer: You can do it by generating and streaming a zip file with no compression,
which is basically to just add the headers before each file's content. You're
right, the libraries don't support this, but you can hack around them to get
it working.

This code wraps zipfile.ZipFile with a class that manages the stream and
creates instances of zipfile.ZipInfo for the files as they come. CRC and size
can be set at the end. You can push data from the input stream into it with
put_file(), write() and flush(), and read data out of it to the output stream
with read().

    
    
    import struct      
    import zipfile
    import time
    
    from StringIO import StringIO
    
    class ZipStreamer(object):
        def __init__(self):
            self.out_stream = StringIO()
    
            # write to the stringIO with no compression
            self.zipfile = zipfile.ZipFile(self.out_stream, 'w', zipfile.ZIP_STORED)
    
            self.current_file = None
    
            self._last_streamed = 0
    
        def put_file(self, name, date_time=None):
            if date_time is None:
                date_time = time.localtime(time.time())[:6]
    
            zinfo = zipfile.ZipInfo(name, date_time)
            zinfo.compress_type = zipfile.ZIP_STORED
            zinfo.flag_bits = 0x08
            zinfo.external_attr = 0600 << 16
            zinfo.header_offset = self.out_stream.pos
    
            # write right values later
            zinfo.CRC = 0
            zinfo.file_size = 0
            zinfo.compress_size = 0
    
            self.zipfile._writecheck(zinfo)
    
            # write header to stream
            self.out_stream.write(zinfo.FileHeader())
    
            self.current_file = zinfo
    
        def flush(self):
            zinfo = self.current_file
            self.out_stream.write(struct.pack("<LLL", zinfo.CRC, zinfo.compress_size, zinfo.file_size))
            self.zipfile.filelist.append(zinfo)
            self.zipfile.NameToInfo[zinfo.filename] = zinfo
            self.current_file = None
    
        def write(self, bytes):
            self.out_stream.write(bytes)
            self.out_stream.flush()
            zinfo = self.current_file
            # update these...
            zinfo.CRC = zipfile.crc32(bytes, zinfo.CRC) & 0xffffffff
            zinfo.file_size += len(bytes)
            zinfo.compress_size += len(bytes)
    
        def read(self):
            i = self.out_stream.pos
    
            self.out_stream.seek(self._last_streamed)
            bytes = self.out_stream.read()
    
            self.out_stream.seek(i)
            self._last_streamed = i
    
            return bytes
    
        def close(self):
            self.zipfile.close()
    

Keep in mind that this code was just a quick proof of concept and I did no
further development or testing once I decided to let the http server itself
deal with this problem. A few things you should look into if you decide to use
it is to check if nested folders are archived correctly, and filename encoding
(which is always a pain with zip files anyway).

