apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
  name: nvidia-add-unrestricted-permissions-dameon-set
spec:
  selector:
    matchLabels:
      name: nvidia-add-unrestricted-permissions
  template:
    metadata:
      labels:
        name: nvidia-add-unrestricted-permissions
    spec:
      containers:
      - name: nvidia-add-unrestricted-permissions
        image: nvidia/cuda:8.0-devel-ubuntu16.04
        securityContext:
          privileged: true
        resources:
          limits:
            nvidia.com/gpu: 1
        command: [ "/bin/bash", "-c", "nvidia-smi -pm 1 && nvidia-smi --applications-clocks-permission=UNRESTRICTED && nvidia-smi --auto-boost-permission=UNRESTRICTED && tail -f /dev/null" ]


# This file defines a daemonset which runs automatically on all
# kubernetes nodes. It is used like so: kubectl create -f <this file_path>.
# The daemonset does the following:
#   - enables persistence mode on the nvidia driver
#   - allows all users to set the GPU clock speed
# In effect, this allows pods created without a privilaged security context to 
# set the GPU clock speeds
# This daemonset config does not define GPU resources, because otherwise it
# would consume them, leaving them unavaliable to pods. Instead, it runs in 
# privilaged mode (so it can see all GPUs), and manually mounts the CUDA
# lib and bin directories.

apiVersion: apps/v1beta2
kind: DaemonSet
metadata:
  name: nvidia-add-unrestricted-permissions-dameon-set
spec:
  selector:
    matchLabels:
      name: nvidia-add-unrestricted-permissions
  template:
    metadata:
      labels:
        name: nvidia-add-unrestricted-permissions
    spec:
      containers:
      - name: nvidia-add-unrestricted-permissions
        image: nvidia/cuda:8.0-devel-ubuntu16.04
        securityContext:
          privileged: true
        command: [ "/bin/bash", "-c", "export PATH=$PATH:/usr/local/bin/nvidia/ && nvidia-smi -pm 1 && nvidia-smi --applications-clocks-permission=UNRESTRICTED && nvidia-smi --auto-boost-permission=UNRESTRICTED && tail -f /dev/null" ]
        volumeMounts:
          - name: nvidia-debug-tools
            mountPath: /usr/local/bin/nvidia
          - name: nvidia-libraries
            mountPath: /usr/local/nvidia/lib64
      volumes:
        - name: nvidia-debug-tools
          hostPath:
            path: /home/kubernetes/bin/nvidia/bin
        - name: nvidia-libraries
          hostPath:
            path: /home/kubernetes/bin/nvidia/lib

Change the NVIDIA permissions daemonset so that it does not consume GPU resources, but still works as intended. This allows all GPUs on the node to be useable by pods. Furthermore, pods can set the clock speed of the GPUs, as this daemonset grants them permissions to do so.
Change the NVIDIA permissions daemonset so that it does not consume GPU
resources, but still works as intended. This allows all GPUs on the node to be useable by pods. Furthermore, pods can set the clock speed of the GPUs, as this daemonset grants them permissions to do so.

-------------
Created by MOE: https://github.com/google/moe
MOE_MIGRATED_REVID=177654682
