I have ~200K named properties and ~25K files. I extract whether the properties hold or not for each file using Python as a set of properties that hold, one set for each file.

To do this extraction I might run hundreds of individual python extraction scripts on a compute farm, in parallel. Each leaving behind some representation of the set extraction from each of the files.

Further processing involves reading these 20K sets and working on the accumulated data. to generate a report on this set of files/properties.

One issue I have is that if I store the extracted set as text then the long property name strings and file name strings will get repeated wasting disk space and increasing parse time.

I was thinking of creating a central index of the sorted property names and just saving the index - same for the file names, as probably .py files to import. 

An alternative to using an index into the sorted list of names would be to use the str.**hash**() value as the index which would mean probably faster processing, but I am worried about the possibility of two unequal strings ending up with the same **hash**() value. Could this happen?

I would be using the same Python executable and OS version on all machines.