HTTP scraper efficiency with multiprocessing I built this scraper for work that will take a csv list of firewalls from our network management system and scan a given list of HTTPS ports to see if the firewalls are accepting web requests on the management ports. I originally built this in powershell, but decided to rebuild it in python for the learning experience. 

I was able to cut down the scan time substantially using multiprocessing, but I'm wondering if I can further optimize my code to get it faster.

Also, I'm very new to python. So if you have any input on better more efficient ways that I could have used to accomplish these steps would be much appreciated.

import urllib.request
import re
import os
import ssl
import multiprocessing

#imports a csv list of firewalls with both private and public IP addresses
f = open(r'\h.csv',"r")
if f.mode =="r":
    cont = f.read()

#regex to remove private ip addresses and then put the remaining public ip addresses in a list
c = re.sub(r"(172)\.(1[6-9]|2[0-9]|3[0-1])(\.(2[0-4][0-9]|25[0-5]|[1][0-9][0-9]|[1-9][0-9]|[0-9])){2}|(192)\.(168)(\.(2[0-4][0-9]|25[0-5]|[1][0-9][0-9]|[1-9][0-9]|[0-9])){2}|(10)(\.(2[0-4][0-4]|25[0-5]|[1][0-9][0-9]|[1-9][0-9]|[0-9])){3}","",cont)
d = re.findall(r"[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}",c)

#uses HTTP requests to check if any of the 8 management ports on the addresses in the list are accepting web requests
def httpScan(list):
    iplen = len(list) 
    ports = [443, 4433, 444, 433, 4343, 4444, 4443, 4434]
    portlen = len(ports)
    for k in range(iplen):
        for i in range(portlen):
            context = ssl._create_unverified_context()
            try:
                fp = urllib.request.urlopen("https://" + str(list[k]) + ":" + str(ports[i]), context=context)
                mybytes = fp.read()
                mystr = mybytes.decode("utf8")
                fp.close()
            except:
                continue
            if "SSLVPN" in mystr:
                print(list[k] + " SSLVPN" + ": " + str(ports[i]) + "  " + str(k) + "  " + str(os.getpid()))
            elif "auth1.html" in mystr:
                print(list[k] + " MGMT" + ": " + str(ports[i]) + "  " + str(k))

#splits the list of IP addresses up based on how many CPU there are and adds each segment to a dictionary
cpu = int(multiprocessing.cpu_count())
sliced = int(len(d)/cpu)
mod = int(len(d))%int(cpu)
num = 1
lists = dict()
for i in range(cpu):
    if i != (cpu - 1):
        lists[i] = d[(num*sliced) - sliced:num*sliced]
        num += 1
    else:
        lists[i] = d[(num*sliced) - sliced:(num*sliced) + mod]

#starts a process for each unique segment created
t = dict()
if __name__ == "__main__": 
    for i in range(cpu):
        t[i] = multiprocessing.Process(target=httpScan, args=(lists[i],))
        t[i].start()

```
Reading File

Here is a tip, while reading, the r is optional

f = open(r'\h.csv',"r")


can be written as

f = open(r'\h.csv')


Your whole reading block can use context managers (blocks using the with keyword).

with open(r'\h.csv', encoding='utf8') as f:
    cont = f.read()


If you are dealing with a huge text file, you might do:

with open(r'\h.csv', encoding='utf8') as f:
    for ip in f:
        ip = ip.rstrip('\n')
        .. verify


String

Using string formatting i.e. .format() can give a better idea of what's going on. It also eliminates the use of str() each time.
We can change this

print(list[k] + " MGMT" + ": " + str(ports[i]) + "  " + str(k))


to that

print("{} MGMT: {}  {}".format(list[k], ports[i], k))


and as from 3.6+, adding an f

print(f"{list[k]} MGMT: {ports[i]}  {k}")


Loop Iteration

In many other languages, you need the index while looping to have the element at this index. Python provides a nice and intuitive way to loop over elements

The current implementation:

ports = [443, 4433, 444, 433, 4343, 4444, 4443, 4434]
portlen = len(ports)
for i in range(portlen):
    print(ports[i])


But the pythonic way is:

ports = [443, 4433, 444, 433, 4343, 4444, 4443, 4434]
for port in ports:
    print(port)


port here gives you the element directly.
If ever you still want the index, you do:

for i, port in enumerate(ports):


where i is the index.

Miscellaneous

Here:

cpu = int(multiprocessing.cpu_count())


No need to cast to int as multiprocessing.cpu_count() already returns an integer. You can verify for int by type(multiprocessing.cpu_count())

Normally with .start(), you must include a .join(), as this allows all child processes to terminate before exiting.

for ...:
   ... .start()

for ...:
   ... .join()