Measuring runtime of CUDA through an script in python
Question: I am trying to use some profiling tools on CUDA with so many different inputs
to see the the differences among different inputs. For this purpose I wrote a
python script to compile the code on CUDA, run the profiler with different
inputs and write the results into a CSV file. Some of these runs take so much
time ( about 3 days ! ) and I want to kill that specific run if it takes more
than a threshold ( for example 30 minutes ) but I do no know how to do it.
Here is a part of python script that I wrote :

    
    
    import subprocess ,sys, string, os
    {START TIME}
    p = subprocess.Popen([CUDA PROFILER COMMAND], stdout=subprocess.PIPE)
    s, err = p.communicate()
    {END TIME}
    

I want to measure START TIME and END TIME and kill this run if it takes more
than 30 minutes. The script would run line by line and if the CUDA PROFILER
command takes more than 30 minutes, it never gets to {END TIME} line and I
cannot measure the elapsed time of the CUDA PROFILER command. Any suggestion
would be so much appreciated.


Answer: There are many answers in [subprocess with
timeout](http://stackoverflow.com/q/1191374/4279) question that you've linked
in the comments. I've chosen a couple that are both simple and portable. [In
Python 3.3+](http://stackoverflow.com/a/12698328/4279):

    
    
    from subprocess import check_output
    
    s = check_output(['cuda', 'profiler', 'command'], timeout=30*60)
    

[On older versions](http://stackoverflow.com/a/10012262/4279):

    
    
    from subprocess import Popen, PIPE
    from threading import Timer
    
    p = Popen(['cuda', 'profiler', 'command'], stdout=PIPE)
    t = Timer(30*60, p.kill) # kill process in 30 minutes
    t.start()
    s = p.communicate()[0] 
    t.cancel() # cancel the hit
    

