One of our current work projects involves working with large ZIP files stored in S3. These are files in the BagIt format, which contain files we want to put in long-term digital storage. Part of this process involves unpacking the ZIP, and examining and verifying every file. So far, so easy – the AWS SDK allows us to read objects from S3, and there are plenty of libraries for dealing with ZIP files.
This is what most code examples for working with S3 look like – download the entire file first (whether to disk or in-memory), then work with the complete copy.
Where this breaks down is if you have an exceptionally large file, or you're working in a constrained environment. Some of our BagIt files are tens of gigabytes, and the largest might be over half a terabyte (even if the individual files are small). And if you've gone serverless and you're running in AWS Lambda, you only get 500 MB of disk space. What are we to do?
To process a ZIP file (like many formats), you don't need to load the entire file at once – it has a well-defined internal structure, and you can read it a bit at a time. In a ZIP, there's a table of contents that tells you what files it contains, and where they are in the overall ZIP. If you want to extract a single file, you can read the table of contents, then jump straight to that file – ignoring everything else. This is easy if you're working with a file on disk, and S3 allows you to read a specific section of a object if you pass an HTTP Range header in your GetObject request.
So if we construct a wrapper for S3 objects that passes the correct Range headers, we can process a large object in S3 without downloading the whole thing.
I couldn't find any public examples of somebody doing this, so I decided to try it myself. In this post, I'll walk you through how I was able to stream a large ZIP file from S3. But fair warning: I wrote this as an experiment, not as production code. You're welcome to use it, but you might want to test it first.
In Python, there's a notion of a "file-like object" – a wrapper around some I/O that behaves like a file, even if it isn't actually a file on disk. It responds to calls like read() and write(), and you can use it in places where you'd ordinarily use a file. The docs for the io library explain the different methods that a file-like object can support, although not every file-like object supports every method – for example, you can't write() to an HTTP response body.
Many libraries that work with local files can also work with file-like objects, including the zipfile module in the Python standard library. If we can get a file-like object from S3, we can pass that around and most libraries won't know the difference!
So although StreamingBody is file-like, it doesn't support the methods we need. We'll have to create our own file-like object, and define those methods ourselves.
Note: the constructor expects an instance of boto3.S3.Object, which you might create directly or via a boto3 resource. This means our class doesn't have to create an S3 client or deal with authentication – it can stay simple, and just focus on I/O operations.
This hints at the key part of doing selective reads: we need to know how far through we are. What part of the object are we currently looking at? When we open a file on disk, the OS handles that for us – but in this case, we'll need to track it ourselves.
We've added the position attribute to track where we are in the stream, and that's what we update when we call seek().
The content_length attribute on the S3 object tells us its length in bytes, which corresponds to the end of the stream.
So we know what to implement next!
Read up to size bytes from the object and return them. As a convenience, if size is unspecified or -1, all bytes until EOF are returned. Otherwise, only one system call is ever made. Fewer than size bytes may be returned if the operating system call returns fewer than size bytes.
If 0 bytes are returned, and size was not 0, this indicates end of file. If the object is in non-blocking mode and no bytes are available, None is returned.
To implement this method, we have to remember that we read from the position set by seek() – not necessarily the start of object. And when we've read some bytes, we need to advance the position.
To read a specific section of an S3 object, we pass an HTTP Range header into the get() call, which defines what part of the object we want to read.
This is a little more complicated than seek().
I've added a size property that exposes the length of the stream, and wraps the content_length attribute. This is for convenience.
If the user doesn't specify a size for read(), we create an open-ended Range header and seek to the end of the file. Note that I'm calling seek() rather than updating the position manually – it saves me writing a second copy of the logic for tracking the position.
If the caller passes a size to read(), we need to work out if this size goes beyond the end of the object – in which case we should truncate it! If it is too big, we fall back to reading the entire object, by making a second call to read() – we don't need to duplicate that logic. If it's not, we create a Range header (and note that byte ranges include the upper value, hence the -1), and seek to the appropriate position.
Then we call the get() method on the object, pass the Range header, and read all the bytes that come back.
And that's all you need to do selective reads from S3.
There's a small cost to making GetObject calls in S3 – both in money and performance.
This wrapper class uses more GetObject calls than downloading the object once. In my brief experiments, it took 3 calls to load the table of contents, and another 3 calls to load an individual file. If you can, it's cheaper and faster to download the entire object to disk, and do all the processing locally – but only if you have the resources to do so! This wrapper is useful when you can't do that.
In practice, I'd probably use a hybrid approach: download the entire object if it's small enough, or use this wrapper if not. I'd trade some extra performance and lower costs for a bit more code complexity.
At work, we write everything in Scala, so I don't think we'll ever use this code directly. (At best, we'll use the ideas it contains.) But this post proves the general idea: you can process a large object in S3 without having to download the whole thing first.
As I said at the top, I wrote this as an experiment, not as production code. Feel free to use it (MIT licence), but you probably want to do some more testing first!