pin and allocate tensorflow on specific NUMA node
no, answer ...
I'm using numactl --cpunodebind=1 --membind=1 - binds execution and memory allocation to NUMA node 1.
My system has two NUMA nodes and two GTX 1080 Ti attached to NUMA node 1 (XEON E5).
The NN models are trained via single-machine multi-GPU data parallelism using Keras' multi_gpu_model.
How can TF be instructed to allocate memory and execute the TF workers (merging weights) only on NUMA node 1? For performance reasons I'd like to prevent accessing memory through the QPI.
tf.device():
1) Does tf.device('/cpu:0') refer to a physical CPU or a physical core or is it simply a 'logical device' (thread|pool?) that is moved between all physical cores that are online?
2) How can the TF scheduler be influenced to map the logical device to a set of physical cores?
3) In the case of memory allocation on NUMA systems - does TF support allocating memory on specific nodes? Or do I have to fall back to set_mempolicy()/numactl (LINUX)?