Create a very large absence/presence SPARSEE matrix in Python
Computers used in data science sometimes have absurdly high amount of RAM (I think I've seen one with 1tb before...)
If you don't have that much RAM, then I think the only way to resolve this is to utilize the hard drive...
I would say, process the data, write it as a structured data on to a hard drive, and loop through while reading, say 50mb at a time to check if the name has already been added to the file, and modify it.
Is there any magic way to create an huge absence presence matrix in python? pd.crosstab and pd.pivot_table fail because of the memory requirement.
I have an extremely large dataset like this one:




Person
Thing




Peter
bike


Peter
car


Jenny
bike


Jenny
plane




and need this:





Bike
car
plane




Peter
1
1
0


Jenny
1
0
1




Note, the matrix is rather sparse. It contains a lot of zeros.