NVidia 1080ti eGPU Ubuntu 16.04.5 LTS - PyTorch / Tensorflow without root permissions
I have some trouble in setting up my system properly. My system consists of:


Intel NUC7i7BNH
ASUS ROG with a NVidia 1080ti


My GPU is correctly detected by lspci:

06:00.0 VGA compatible controller: NVIDIA Corporation Device 1b06 (rev a1) (prog-if 00 [VGA controller])
        Subsystem: ASUSTeK Computer Inc. Device 85ea
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast &gt;TAbort- &lt;TAbort- &lt;MAbort- &gt;SERR- &lt;PERR- INTx-
        Latency: 0, Cache Line Size: 128 bytes
        Interrupt: pin A routed to IRQ 18
        Region 0: Memory at c4000000 (32-bit, non-prefetchable) [size=16M]
        Region 1: Memory at a0000000 (64-bit, prefetchable) [size=256M]
        Region 3: Memory at b0000000 (64-bit, prefetchable) [size=32M]
        Region 5: I/O ports at 2000 [size=128]
        Expansion ROM at c5000000 [disabled] [size=512K]
        Capabilities: &lt;access denied&gt;
        Kernel driver in use: nvidia
        Kernel modules: nvidiafb, nouveau, nvidia_384_drm, nvidia_384


and with the commands below I installed drivers for the GPU, CUDA and cuDNN: 

# Show thunderbolt port / authorize eGPU
$ cat /sys/bus/thunderbolt/devices/0-1/device_name
$ echo 1 | sudo tee -a /sys/bus/thunderbolt/devices/0-1/authorized 

# eGPU - on Ubuntu 16.04 - nvidia-384
$ sudo ubuntu-drivers devices
$ sudo ubuntu-drivers autoinstall
$ sudo apt-get install nvidia-modprobe

# CUDA - Download CUDA from Nvidia - http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/
$ wget https://developer.nvidia.com/compute/cuda/9.0/Prod/local_installers/cuda_9.0.176_384.81_linux-run
$ chmod +x cuda_9.0.176_384.81_linux-run
$ ./cuda_9.0.176_384.81_linux-run --extract=$HOME
$ sudo ./cuda-linux.9.0.176-22781540.run
$ sudo ./cuda-samples.9.0.176-22781540-linux.run
$ sudo bash -c "echo /usr/local/cuda/lib64/ &gt; /etc/ld.so.conf.d/cuda.conf"
$ sudo ldconfig
# add :/usr/local/cuda/bin (including the ":") at the end of the PATH="/blah:/blah/blah" string (inside the quotes).
$ sudo nano /etc/environments

# CUDA samples - Check installation
$ cd /usr/local/cuda-9.0/samples
$ sudo make
$ /usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery

# cuDNN
$ wget http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1604/x86_64/libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb
$ dpkg -i libcudnn7_7.0.5.15-1+cuda9.0_amd64.deb

# Authorization (Security risk!)
$ sudo nano /etc/udev/rules.d/99-local.rules
# Add
ACTION=="add", SUBSYSTEM=="thunderbolt", ATTR{authorized}=="0", ATTR{authorized}="1"


With sudo prime-select nvidia I can switch between the eGPU and the Intel integrated one. After a logout + login the eGPU seems to work (nvidia-smi):

Fri Sep 21 09:25:18 2018       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 384.130                Driver Version: 384.130                   |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 108...  Off  | 00000000:06:00.0 Off |                  N/A |
|  0%   47C    P0    84W / 275W |    214MiB / 11172MiB |      4%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|    0      6734      G   /usr/lib/xorg/Xorg                           138MiB |
|    0      7081      G   kwin_x11                                      23MiB |
|    0      7084      G   /usr/bin/krunner                               2MiB |
|    0      7092      G   /usr/bin/plasmashell                          47MiB |
+-----------------------------------------------------------------------------+


The CUDA samples work also:

$ /usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery
/usr/local/cuda/samples/bin/x86_64/linux/release/deviceQuery Starting...

 CUDA Device Query (Runtime API) version (CUDART static linking)

Detected 1 CUDA Capable device(s)

Device 0: "GeForce GTX 1080 Ti"
  CUDA Driver Version / Runtime Version          9.0 / 9.0
  CUDA Capability Major/Minor version number:    6.1
  Total amount of global memory:                 11172 MBytes (11715084288 bytes)
  (28) Multiprocessors, (128) CUDA Cores/MP:     3584 CUDA Cores
  GPU Max Clock rate:                            1683 MHz (1.68 GHz)
  Memory Clock rate:                             5505 Mhz
  Memory Bus Width:                              352-bit
  L2 Cache Size:                                 2883584 bytes
  Maximum Texture Dimension Size (x,y,z)         1D=(131072), 2D=(131072, 65536), 3D=(16384, 16384, 16384)
  Maximum Layered 1D Texture Size, (num) layers  1D=(32768), 2048 layers
  Maximum Layered 2D Texture Size, (num) layers  2D=(32768, 32768), 2048 layers
  Total amount of constant memory:               65536 bytes                                                                                                                              
  Total amount of shared memory per block:       49152 bytes                                                                                                                              
  Total number of registers available per block: 65536                                                                                                                                    
  Warp size:                                     32                                                                                                                                       
  Maximum number of threads per multiprocessor:  2048                                                                                                                                     
  Maximum number of threads per block:           1024                                                                                                                                     
  Max dimension size of a thread block (x,y,z): (1024, 1024, 64)                                                                                                                          
  Max dimension size of a grid size    (x,y,z): (2147483647, 65535, 65535)                                                                                                                
  Maximum memory pitch:                          2147483647 bytes                                                                                                                         
  Texture alignment:                             512 bytes                                                                                                                                
  Concurrent copy and kernel execution:          Yes with 2 copy engine(s)                                                                                                                
  Run time limit on kernels:                     Yes                                                                                                                                      
  Integrated GPU sharing Host Memory:            No                                                                                                                                       
  Support host page-locked memory mapping:       Yes                                                                                                                                      
  Alignment requirement for Surfaces:            Yes                                                                                                                                      
  Device has ECC support:                        Disabled                                                                                                                                 
  Device supports Unified Addressing (UVA):      Yes                                                                                                                                      
  Supports Cooperative Kernel Launch:            Yes                                                                                                                                      
  Supports MultiDevice Co-op Kernel Launch:      Yes                                                                                                                                      
  Device PCI Domain ID / Bus ID / location ID:   0 / 6 / 0                                                                                                                                
  Compute Mode:                                                                                                                                                                           
     &lt; Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) &gt;                                                                                             

deviceQuery, CUDA Driver = CUDART, CUDA Driver Version = 9.0, CUDA Runtime Version = 9.0, NumDevs = 1                                                                                     
Result = PASS


Let's come to the problem: The following snippet works only, if I execute python with root permissions.

$ python                                                                                                
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)                                                                       
[GCC 7.2.0] on linux                                                                                                                 
Type "help", "copyright", "credits" or "license" for more information.                                                               
&gt;&gt;&gt; import torch                                                                                                                     
&gt;&gt;&gt; torch.cuda.is_available()                                                                                                        
True                                                                                                                                 
&gt;&gt;&gt; torch.Tensor([0]).cuda()                                                                                                         
Traceback (most recent call last):                                                                                                   
  File "&lt;stdin&gt;", line 1, in &lt;module&gt;                                                                                                
RuntimeError: CUDA error: unknown error  


vs.

$ sudo ~/miniconda3/envs/PyTorch/bin/python                                                             
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51)                                                                       
[GCC 7.2.0] on linux                                                                                                                 
Type "help", "copyright", "credits" or "license" for more information.                                                               
&gt;&gt;&gt; import torch                                                                                                                     
&gt;&gt;&gt; torch.cuda.is_available()                                                                                                        
True                                                                                                                                 
&gt;&gt;&gt; torch.Tensor([0]).cuda()                                                                                                         
tensor([0.], device='cuda:0')


TLDR:
My CUDA installation seems to work and my eGPU is correctly recognized by the system. However in PyTorch (and also Tensorflow) an 'Unknown Error' is thrown, when I try to move same data onto the eGPU, although it is recognized by the frameworks. When I execute the same code with root permissions everything works like a charm. Does somebody know, where I have to adopt some permissions, such that the code can be executed as standard user?

After trying out different versions of the nvidia driver, I came to the conclusion, that there is a problem with version 384.

I deleted all nvidia driver related stuff via sudo apt-get purge nvidia. and installed version 390:

$ sudo add-apt-repository ppa:graphics-drivers/ppa
$ sudo ubuntu-drivers devices
$ sudo apt install nvidia-390


And here the result:

(PyTorch) max@MaxKubuntuNUC:~$ python
Python 3.6.6 |Anaconda, Inc.| (default, Jun 28 2018, 17:14:51) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; import torch
&gt;&gt;&gt; torch.cuda.is_available()
True
&gt;&gt;&gt; torch.tensor([1]).cuda()
tensor([1], device='cuda:0')


Now everything works like a charm.
