What does a tensorflow session exactly do?
TensorFlow sessions allocate ~all GPU memory on startup, so they can bypass the cuda allocator.
Do not run more than one cuda-using library in the same process or weird things (like this stream executor error) will happen.
I have tensorflow's gpu version installed, as soon as I create a session, it shows me this log:

I tensorflow/core/common_runtime/gpu/gpu_device.cc:885] Found device 0
  with properties:  name: GeForce GTX TITAN Black major: 3 minor: 5
  memoryClockRate (GHz) 0.98 pciBusID 0000:01:00.0 Total memory: 5.94GiB
  Free memory: 5.31GiB I
  tensorflow/core/common_runtime/gpu/gpu_device.cc:906] DMA: 0  I
  tensorflow/core/common_runtime/gpu/gpu_device.cc:916] 0:   Y  I
  tensorflow/core/common_runtime/gpu/gpu_device.cc:975] Creating
  TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX TITAN
  Black, pci bus id: 0000:01:00.0)

And when I check my GPU memory usage, around 90% of it gets consumed.
Tensorflow documentation does not say anything about this. Does it take control of the gpu ? Why does it consume most of the memory ?