<issue_start><issue_comment>Title: Nvidia Docker - MPS on all GPUs (Nvidia-DGX1)
username_0: Hi,

We are currently trying to setup the NVIDIA MPS (EXPERIMENTAL) example and get it to run on all 8  GPUs but are running into some issues.
At first we had problems with the IPC in the docker-compose, because the nbody examples were not able to join the ipc of the docker daemon. This was resolve by using ipc: host.

NVIDIA_VISIBLE_DEVICES=all is set and the nvidia-cuda-mps-control daemon is started. When we start the examples wit docker-compose up, the mps-server starts on all GPUs independetly and also the mps-daemon is reporting, that all GPUs are available.
But the nbody only runs on GPU0 and won't scale to other GPUs, as we expected. (CUDA_ACTIVE_THREAD_PERCENTAGE was set to several values but neither made a change)

We also tried running multiple to exceed the memory limit of a sinigle GPU, to get the MPS-Daemon to schedule further containers onto different GPUs, but ended up using only a single GPU.


Output of nvidia-smi:

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0     45865      C   nvidia-cuda-mps-server                       418MiB |
|    0     51818    M+C   ./nbody                                      438MiB |
|    0     51882    M+C   ./nbody                                      438MiB |
|    0     51898    M+C   ./nbody                                      438MiB |
|    1     45865      C   nvidia-cuda-mps-server                       418MiB |
|    2     45865      C   nvidia-cuda-mps-server                       418MiB |
|    3     45865      C   nvidia-cuda-mps-server                       418MiB |
|    4     45865      C   nvidia-cuda-mps-server                       418MiB |
|    5     45865      C   nvidia-cuda-mps-server                       418MiB |
|    6     45865      C   nvidia-cuda-mps-server                       418MiB |
|    7     45865      C   nvidia-cuda-mps-server                       418MiB |
+-----------------------------------------------------------------------------+



Output of /var/log/nvidia-mps/control.log 
 Accepting connection...
NEW UI
UI closed
Accepting connection...
User did not send valid credentials
Accepting connection...
NEW CLIENT 51818 from user 0: Server already exists
Accepting connection...
User did not send valid credentials
Accepting connection...
NEW CLIENT 51882 from user 0: Server already exists
Accepting connection...
User did not send valid credentials
Accepting connection...
NEW CLIENT 51898 from user 0: Server already exists
Accepting connection...
NEW CLIENT 51818 from user 0: Server already exists
Accepting connection...
NEW CLIENT 51882 from user 0: Server already exists
Accepting connection...
NEW CLIENT 51898 from user 0: Server already exists


<details>
 <summary>Click here to see debug information></summary>
 - [ 4.4.0-128-generic #154-Ubuntu SMP Fri May 25 14:15:18 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux] Kernel version from `uname -a`
 - [ doesnt show any useful GPU information besides GPUBoardSerialNumber/PCI_BUS_ID ] Any relevant kernel output lines from `dmesg`
 - [384.145] Driver information from `nvidia-smi -a` See attached for GPU0 => [1]
 - [ 18.03.1-ce ] Docker version from `docker version` [2]
 - [ ] NVIDIA packages version from `dpkg -l '*nvidia*'` _or_ `rpm -qa '*nvidia*'` [3]
 - [ 1.0.0 ] NVIDIA container library version from `nvidia-container-cli -V`
 - [ ] NVIDIA container library logs (see [troubleshooting](https://github.com/NVIDIA/nvidia-docker/wiki/Troubleshooting))
 - [ ] Docker command, image and tag used => using the MPS-nbody Sample delivered with https://gitlab.com/nvidia/samples/tree/master/mps

[1]==============NVSMI LOG==============

Timestamp                           : Thu Nov 22 12:23:04 2018
Driver Version                      : 384.145

Attached GPUs                       : 8
GPU 00000000:06:00.0
    Product Name                    : Tesla V100-SXM2-16GB
    Product Brand                   : Tesla
    Display Mode                    : Enabled
    Display Active                  : Disabled
    Persistence Mode                : Enabled
    Accounting Mode                 : Disabled
[Truncated]
 API version:  1.37
 Go version:   go1.9.5
 Git commit:   9ee9f40
 Built:        Thu Apr 26 07:17:20 2018
 OS/Arch:      linux/amd64
 Experimental: false
 Orchestrator: swarm

Server:
 Engine:
  Version:      18.03.1-ce
  API version:  1.37 (minimum version 1.12)
  Go version:   go1.9.5
  Git commit:   9ee9f40
  Built:        Thu Apr 26 07:15:30 2018
  OS/Arch:      linux/amd64
  Experimental: false

[3]  https://imgur.com/a/A8oo8fb
</details>
<issue_comment>username_0: Can anyone confirm that she/he, at least, read this issue?
<issue_comment>username_1: This is because by default the nbody sample will only run on one GPU, you will need to also set the command line parameters (in mps/cuda-samples) to `CMD ["./nbody", "-benchmark", "-i=10000", "-numdevices=8"]`

Closing for now, feel free to reopen if you still have issues with the IPC :)!<issue_closed>
<issue_comment>username_2: @username_1 do you use AWS ECS to launch your container?

I have a problem running this docker container with ipc: host inside the ECS container.

It works if I run it manually with docker run --ipc=Host