hash image url and if valid store image url I am trying to get all image urls which are valid and unique.To make sure i wont end up using the same image again i calculate hash of the image and store it in a text file and read the text file again.if there is any better way to do this, please let me know.Overall any speed optimisation would be really helpful, thanks in advance.

import multiprocessing as mp
import socket
# Set the default timeout in seconds
timeout = 20
socket.setdefaulttimeout(timeout)

from PIL import Image 
import hashlib
import os
import pickle

def getImages(val):
    # import pdb;pdb.set_trace()
    #Dowload images
    f = open('image_files.txt', 'a')
    k = open('image_hash.txt', 'a')    
    try:
        url=val # preprocess the url from the input val
        local=url.split('/')[-1] #Filename Generation From Global Varables And Rand Stuffs...
        urllib.request.urlretrieve(url,local)
        md5hash = hashlib.md5(Image.open(local).tobytes())
        image_hash = md5hash.hexdigest()
        k.write(image_hash+'\n')
        z = open('image_hash.txt', 'r')            
        a = z.readlines()
        image_hash_list = [c.strip('\n') for c in a]
        if image_hash not in image_hash_list:
            os.remove(local)
            f.write(url+'\n')
            print("DONE - " + url)
            return 1
        else:
            os.remove(local)
    except Exception as e:
        # print(e)
        print("CAN'T DOWNLOAD - " + url )
        return 0

files = "Identity.txt"
lst = list(open(files))
lst = [l.replace("\n", "") for l in lst]
lst = lst[::-1]
pool = mp.Pool(processes=12)
res = pool.map(getImages, lst)
print ("tempw")


Here identity.txt file looks like this,

http://url1.com
http://url2.com
...
here's a few comments in no particular order.

Have a look over PEP8. It's Python's style guide which helps keep Python code consistent and readable. Things like naming functions which snake_case, constants as CAPS, proper spacing around operators, commenting style and placement, spacing between imports, etc.

It's recommended to use a context manager for opening files, this prevents them from being left open and the indentation makes it clear when a file is being used. You can open two files at once like so:

with open('file_one.txt', 'a') as file_one, open('file_two.txt', 'a') as file_two:
    ...


Instead of using return to handle errors, just raise the error itself. Catch the specific exception (ValueError, TypeError, etc (not Exception or worse BaseException)) then raise it with a message like so:

except TypeError:
    raise TypeError('Reason for failure message')


If you want to continue having caught the error, you can print the error message and continue instead:

except TypeError as exc:
    print(f'Reason for failure: {exc}; continuing ...')


A better way of storing lines from a file to a list would be to use readlines() and strip(). readlines() converts a file to a list where each item is a line in the file, and strip() can remove unwanted linebreaks and whitespace.

Wrap the code you're going to run with:

if __name__ == '__main__':
    ...


This stops any old code being executed if importing it as a module.

Why do you take val as an argument then immediately rename it to url? Just take url as the argument, it's more descriptive. You've also used url in your except block where it may not have been defined. It wouldn't fail in this example but it could have done if you wrote code like this elsewhere.

Your variable names could be much improved. No single letter variables! Make them descriptive and avoid putting their type in the name. k -> hash_file; image_hash_list -> image_hashes; etc.

You're importing pickle but not using it. You're also using urllib without importing it.

You're calling os.remove(local) regardless of the if condition. Put this outside the if check.

There are a few other nitpicks and I can't speak to what the code is actually doing as it's not something I'm familiar with, but I hope that helps.