Getting a MemoryError because list/array is too large
Question: ## Problem

I have to download `object_x`. For simplicity's sake, `object_x` comprises a
series of `integers` adding up to `1000`. The download is irregular. I receive
groups or `chunks` of integers in seemingly random order, and I need to keep
track of them until I have all `1000` to make up the final `object_x`.

The incoming chunks can also overlap, so for instance:

    
    
    Chunk 1: integers 0-500
    Chunk 2: integers 600-1000
    Chunk 3: integers 400-700
    

## Current method

Create `object_x` as a `list` containing all of its comprising integers
`0-1000`. When a `chunk` is downloaded, remove all of the integers that
comprise the `chunk` from `object_x`. Keep doing this until `object_x` is
empty (known to be complete then).

    
    
    object_x = range(0,1000)
    
    # download chunk 1
    chunk = range(0, 500)
    
    for number in chunk:
        if number in object_x:
            object_x.remove(number)
    
    # repeat for every downloaded chunk
    

## Conclusion

This method is very memory intensive. The script throws a MemoryError if
`object_x` or `chunk` is too large.

I'm searching for a better way to keep track of the chunks to build the
`object_x`. Any ideas? I'm using Python, but language doesn't matter I guess.


Answer: This is the kind of scenario where streaming is very important. Doing
everything in memory is a bad idea because you might not have enough memory
(as in your case). You should probably save the chunks to disk, keep track of
how many you downloaded, and when you reach 1000, process them on disk (or
load them into memory one by one to process them).

"[C# Security: Computing File
Hashes](http://www.programmersranch.com/2014/05/c-security-computing-file-
hashes.html)" is a recent article I wrote - it's a different subject, but it
does illustrate the importance of streaming towards the end.

