Python for HPC: Parse file, generate statistics, record output
I have a program which is designed to be highly parallelizable.


I can't see this, all I can see is a broken module level program.

As for improvements in your code.
There are quite a few ways you can improve your for-loop to make vals:


Put it in a function,
Use a defaultdict, this should result in something like:

if split_line[1] != nam: 
    continue
vals[split_line[1]].append(float(split_line[0]))

This should make you realise that you'll only ever add to vals[nam].
And so change it to just a list.
slice dchunk.values.

for line in dchunk.values[:49] + dchunk.values[numlines - 50:]:
    split_line = ''.join(str(s) for s in line).split()


Alternately if this doesn't work use a comprehension with enumerate.

for line in (l for i, l in enumerate(dchunk.values) if i &lt; 49 or i &gt; numlines - 50):
    split_line = ''.join(str(s) for s in line).split()

Merge all the if statements into one.

if (len(split_line) != 2 or
    split_line[0] in ('nan', '-nan') or
    split_line[1] != nam):
    continue



You should change some of your variables to 'constants' using the UPPER_SNAKE_CASE convention.

And you can change count_vals into a single line with len and itertools.takewhile.

count_vals = float(len(takewhile(lambda i: i &gt; coe_threshold, coe_vals)))


You can change the way you get numlines to len(list(f)).