MultiThreading with a python loop
Question: I an trying to run this Python code on several threads of my processor, but I
can't find how to allocate multiple threads. I am using **python 2.7** in
Jupyter (formerly IPython). The initial code is below (all this part works
perfectly). It is a web parser which takes `x` i.e., a url among my_list i.e.,
a list of url and then write a CSV (where out_string is a line).

# Code without MultiThreading

    
    
    my_list = ['http://stackoverflow.com/', 'http://google.com']
    
    def main():
        with open('Extract.csv'), 'w') as out_file:
            count_loop = 0
            for x in my_list:
                #================  Get title ==================#
                out_string = ""
                campaign = parseCampaign(x)
                out_string += ';' + str(campaign.getTitle())
    
                #================ Get Profile ==================#
                if campaign.getTitle() != 'NA':
                    creator = parseCreator(campaign.getCreatorUrl())
                    out_string += ';' + str(creator.getCreatorProfileLinkUrl())
                else:
                    pass
                #================ Write ==================#
                out_string += '\n'
                out_file.write(out_string) 
                count_loop +=1
                print '---- %s on %s ------- ' %(count_loop, len(my_list))
    

# Code with MultiThreading but not working

    
    
    from threading import Thread
    my_list = ['http://stackoverflow.com/', 'http://google.com']
    
    def main(x):
        with open('Extract.csv'), 'w') as out_file:
            count_loop = 0
            for x in my_list:
                #================  Get title ==================#
                out_string = ""
                campaign = parseCampaign(x)
                out_string += ';' + str(campaign.getTitle())
    
                #================ Get Profile ==================#
                if campaign.getTitle() != 'NA':
                    creator = parseCreator(campaign.getCreatorUrl())
                    out_string += ';' + str(creator.getCreatorProfileLinkUrl())
                else:
                    pass
                #================ Write ==================#
                out_string += '\n'
                out_file.write(out_string) 
                count_loop +=1
                print '---- %s on %s ------- ' %(count_loop, len(my_list))
    
    for x in my_list:
        t = Thread(target=main, args=(x,))
        t.start()
        t2 = Thread(target=main, args=(x,))
        t2.start()
    

I cannot find a good way to implement more than one thread to run this piece
of code, and I am a bit confused because the documentation is not very easy to
understand. With one core, this code takes 2 hours long, multi-threading will
save me lot of time!


Answer: Well... the answer to:

> Why would you assign two threads for the same exact task?

is:

> to run faster the loop

(see at the comments of the original post)

then something is pretty wrong here.

Dear OP, both of the threads will do _exactly_ the same thing! This means that
the first thread will do exactly the same thing as the second.

What you can do is something like the following:

    
    
    import multiprocessing
    
    nb_cores = 2  # Put the correct amount
    
    def do_my_process_for(this_argument):
      # Add the actual code
      pass
    
    def main():
    
      pool = multiprocessing.Pool(processes=nb_cores)
    
      results_of_processes = [pool.applyasync(
          do_my_process, 
          args=(an_argument, ),
          callback=None
      ) for an_argument in arguments_list]
    
      pool.close()
      pool.join()
    

Basically, you can think each process/thread as having its own "mind". This
means that in your code the first thread will do the process defined in
`main()` for the argument `x` (taken from your iteration on your list) and the
second one will do the same task (the one in the `main()`) again for `x`.

What you need is to formulate your process as a procedure having a set of
input parameters and a set of output. Then you can create multiple processes,
to each of them give one of the desired input parameters and then the process
will execute your main routine with the proper parameter.

Hope it helps. See also the code and I think you will understand it.

Also, see:

multiprocessing map and asynchronous map (I don't remember right now the exact
name)

and

functools partial

