Hadoop for grid computing
Your job seems like a map-reduce job and hence might be good for Hadoop. Hadoop has a zoo of an ecosystem though. 

Hadoop is a distributed file system. It distributes data on a cluster and because this data is split up it can be analysed in parallel. Out of the box, Hadoop allows you to write map reduce jobs on the platform and this is why it might help with your problem.

The following technologies work on Hadoop: 


If the data can be represented in a table format you might want to check out technologies like hive and impala. Impala uses all the distributed memory across a cluster and is very performant while it allows you to still work with a table structure. 
A more new, but promising alternative might also be spark which allows for more iterative procedures to be run on the cluster. 


Don't underestimate the amount of time setting up and the amount of time needed to understand Hadoop.