Writing a big file with np.save in Python in a while True Loop
Question: I am scraping a website with a while True loop, and then saving all of the
data to a file with np.savez. I want to process the npz file, but the file
updates faster than I can copy it. Here's my code:

    
    
    while True:
      time.sleep(1.5)
      for post in new:
        all_posts.append(post)
      np.savez('records.npz', posts)
      new = other_site.get_next()
    

Initially to process the data I was scraping I just would copy the file, but
now the file is too big and it gets corrupted every time. I could restart this
process from the beginning and save less often so I would have more time to
copy, but I'd like to know if there's a way I can recover the data I've
written. Another idea I had was to truncate the end of the file such that it
still looks like an npz file and python can read it, but I don't know if
that's possible.


Answer: To avoid your file getting trampled or over-written, why not write some python
code to avoid this? For example, you could save to a new file for each site,
and collect these files in a directory;

    
    
    import os
    
    os.mkdir('scraped_sites')
    
    while True:
       time.sleep(1.5)
       for post in new:
          all_posts.append(post)
    
       # create a unique file path
       save_file = os.path.join('scraped_sites', 'records_%s.npz' % other_site)
       np.savez(save_file, all_posts)
    
       new = other_site.get_next()
    

This way your file will never be trampled and so you needn't worry about
processing it before it gets written again. If you don't like the idea of
naming your files, check out
[`tempfile`](http://docs.python.org/2/library/tempfile.html)

Also, `while True` can be dangerous as your loop will never exit - I assume
that you've just written this for brevity, but it would be good to have a
`break` or `while <conditional` so that you don't accidentally force the loop
to exit when it's mid file write.

