How should I divide a large (~50Gb) dataset into training, test, and validation sets?
You could assign a unique sequential number to each row, then choose a random sample of those numbers, then serially extract each relevant row to a new file.
I have a large dataset. It's currently in the form of uncompressed numpy array files that were created with numpy.array.tofile(). Each file is approximately 100000 rows of 363 floats each. There are 192 files totalling 52 Gb.
I'd like to separate a random fifth of this data into a test set, and a random fifth of that test set into a validation set. 
In addition, I can only train on 1 Gb at a time (limitation of GPU's onboard memory) So  I need to randomize the order of all the data so that I don't introduce a bias by training on the data in the order it was collected.
My main memory is 8 Gb in size. Can any recommend a method of randomizing and partitioning this huge dataset?