Is tensorflow session running in parallel to the rest of my code?
It entirely depends upon how you have written your code. This should be trivial to check, by checking out your CPU  and GPU utilization simultanously
I normally make use of tf.data.Dataset API. I use the get_next() method of an iterator to feed data to a network. CPU and GPU work in parallel in this case.
I'm running my session on a GPU and I'm wondering if the 'session.run()' piece of code is running in parallel to my other code in my script. 
I use batch processing on the CPU prior to running 'session.run()' in a loop and would like to pipeline this processing with the execution on the GPU. Is this already satisfied in this setting or do I need to manually start threads?