How to load one line at a time from a pickle file?
Thanks everyone. I ended up finding a workaround (a machine with more RAM so I could actually load the dataset into memory).
I have a large dataset: 20,000 x 40,000 as a numpy array. I have saved it as a pickle file.
Instead of reading this huge dataset into memory, I'd like to only read a few (say 100) rows of it at a time, for use as a minibatch.
How can I read only a few randomly-chosen (without replacement) lines from a pickle file?