Multithreading in python using queue
Question: I am quite new to Python and I need to implement multithreading in my code.

I have a huge .csv file (million lines) as my input. I read the line, make a
rest request for each line, do some processing on each line and write the
output into another file. The ordering of lines in input/output file does
matter . Right now I am doing this line by line. I want to run the same code,
but in parallel, i.e read 20 lines of input from .csv file and make the rest
call in parallel so that my program is faster.

I have been reading up on <http://docs.python.org/2/library/queue.html>, but I
read about the python GIL issue which says the code will not run faster even
after multithreading. Is there any other way to achieve multithreading in a
simple way?


Answer: Can you break the .csv file into multiple smaller files? If you can, then you
could have another program running multiple versions of your processer.

Say the files were all named _file1_ , _file2_ , etc. and your processer took
the filename as an argument. You could have:

    
    
    import subprocess
    import os
    import signal
    
    for i in range(1,numfiles):
        program = subprocess.Popen(['python'], "processer.py", "file" + str(i))
        pid = program.pid
    
        #if you need to kill the process:
        os.kill(pid, signal.SIGINT)
    

