Speeding up I/O bound python program I have the following program. I have 90 million domain names split between 87 files. I need to resolve the IP of all. Right now I feel like this is super slow.
import multiprocessing
import fileinput
import socket
from datetime import datetime

socket.setdefaulttimeout(2)

def worker(i):
    filename = (str(i).zfill(len(str(111))))
    buffer_array = []
    for line in fileinput.input(f&quot;./data/{filename}&quot;):
        domain = '.'.join( list( reversed( line.split('\t')[1].split('.') ) ) )
        try:
            ip_list = socket.gethostbyname(domain)
            buffer_array.append(ip_list)
            if(len(buffer_array) == 1000):
                with open(f&quot;./parsed_data/{i}&quot;, &quot;a+&quot;) as save_file:
                    save_file.write(&quot;\n&quot;.join(buffer_array))
                    buffer_array = []
                    print(f&quot;{datetime.now()} -- WRITING: worker {i}&quot;)
                    save_file.close()

        except Exception as e:
            pass

    with open(f&quot;./parsed_data/{i}&quot;, &quot;a+&quot;) as save_file:
        save_file.write(&quot;\n&quot;.join(buffer_array))
        print(f&quot;{datetime.now()} -- WRITING ***FINAL***: worker {i}&quot;)
        save_file.close()

if __name__ == '__main__':
    jobs = []
    for i in range(87):
        p = multiprocessing.Process(target=worker, args=(i,))
        jobs.append(p)
        p.start()

In node, I was able to do 1,000,000 on a 2017 macbook pro in about 12 hours without using workers. Right now this accomplished 39 million in 32 hours. My hope was I could make it do it's thing in 12 hours (1m per worker in in 12 hours) on a AWS T3aXL
Does anyone have a bit faster way of doing this?
Minor Python cleanup
First, a bunch of little stuff:
This -
(str(i).zfill(len(str(111))))

does not need outer parens. Also, why the acrobatics to get &quot;3&quot;? Just declare a MAX_DIGITS = 3.
This -
'.'.join( list( reversed( line.split('\t')[1].split('.') ) ) )

should not be creating an inner list. The output of reversed is iterable.
This -
if(len(buffer_array) == 1000):

does not need outer parens; you're not in (C/Java/etc).
This -
save_file.close()

needs to be deleted both places that it appears. There's an implicit close() from your use of a with.
This -
    except Exception as e:
        pass

is a bad idea. Log or output the exception.
87 deserves a named constant.
Broad performance
gethostbyname is probably slow (as indicated in the comments). Consider running a local caching DNS resolution service to eliminate the first network hop.