python does not quit when the input file is too big
You've hit the RAM bottleneck. Its very common in de novo assemblies and resolved using template assemblies. I don't know this program but suspect it is performing a pairwise analysis for every piece of data in a genomics data set.
The solution is to either:


find a bigger machine, e.g. a big multi-core machine and make sure the program distributes across every core.

parallelise the calculation (an excellent way to overcome RAM problems), or,

find an approximation to the calculation that is less computationally demanding, templates are a good way to circumvent this.