Accelerate computing a Gram-like matrix with large files
Try to cache as much as possible. 

I suppose buying more RAM and read all .npy files in advance is out of the question.

But, for instance, if it is possible to keep the data of 20 files in memory, you could read 10 files for vi-data and 10 files for vj-data, now you can fill a block of 10x10 in matrix G at the cost of 20 file reads instead of 10 + 10x10 = 110 file reads. Looping and indexing will get a bit more complicated of course.