title: Rootless buildkitd throws permission error inside container
body: <p>I decided to use the rootless version of Buildkit to build and push Docker images to a GCR (Google Container Registry) from within a container in Kubernetes.</p>
  <p>I stumbled upon this error:</p>
  <pre><code>/moby.buildkit.v1.Control/Solve returned error: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: failed to mount /home/user/.local/tmp/buildkit-mount859701112: [{Type:bind Source:/home/user/.local/share/buildkit/runc-native/snapshots/snapshots/2 Options:[rbind ro]}]: operation not permitted
  </code></pre>
  <p>I am running <code>buildkitd</code> as a <code>deployment</code> linked to a <code>service</code> as specified by the <a href="https://github.com/moby/buildkit/tree/master/examples/kubernetes#deployment--service" rel="nofollow noreferrer">buildkit documentation</a>
  Those resources are ran inside a Kubernetes Cluster hosted on the Google Kubernetes Engine.</p>
  <p>I am using the following YAML for the Deployment and Service</p>
  <pre class="lang-yaml prettyprint-override"><code>apiVersion: apps/v1
  kind: Deployment
  metadata:
    labels:
      app: buildkitd
    name: buildkitd
  spec:
    replicas: 1
    selector:
      matchLabels:
        app: buildkitd
    template:
      metadata:
        labels:
          app: buildkitd
        annotations:
          container.apparmor.security.beta.kubernetes.io/buildkitd: unconfined
          container.seccomp.security.alpha.kubernetes.io/buildkitd: unconfined
      spec:
        containers:
        - name: buildkitd
          image: moby/buildkit:master-rootless
          args:
          - --addr
          - unix:///run/user/1000/buildkit/buildkitd.sock
          - --addr
          - tcp://0.0.0.0:1234
          - --oci-worker-no-process-sandbox
          readinessProbe:
            exec:
              command:
              - buildctl
              - debug
              - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          livenessProbe:
            exec:
              command:
              - buildctl
              - debug
              - workers
            initialDelaySeconds: 5
            periodSeconds: 30
          securityContext:
            runAsUser: 1000
            runAsGroup: 1000
          ports:
          - containerPort: 1234
  ---
  apiVersion: v1
  kind: Service
  metadata:
    labels:
      app: buildkitd
    name: buildkitd
  spec:
    ports:
    - port: 1234
      protocol: TCP
    selector:
      app: buildkitd
  </code></pre>
  <p>It is the same as <a href="https://github.com/moby/buildkit/blob/master/examples/kubernetes/deployment%2Bservice.rootless.yaml" rel="nofollow noreferrer">buildkit documentation</a>'s without the TLS certificates setup.</p>
  <p>From another Pod, I then contact the Buildkit Daemon using the following command:</p>
  <pre><code>./bin/buildctl \
      --addr tcp://buildkitd:1234 \
      build \
      --frontend=dockerfile.v0 \
      --local context=. \
      --local dockerfile=. \
      --output type=image,name=eu.gcr.io/$PROJECT_ID/test-image,push=true
  </code></pre>
  <p>The <code>buildkitd</code> container successfuly receives the request but throws the error above.</p>
  <p>The output of the <code>buildctl</code> command is the following:</p>
  <pre><code>#1 [internal] load .dockerignore
  #1 transferring context: 2B done
  #1 DONE 0.1s
  
  #2 [internal] load build definition from Dockerfile
  #2 transferring dockerfile: 120B done
  #2 DONE 0.1s
  error: failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to read dockerfile: failed to mount /home/user/.local/tmp/buildkit-mount859701112: [{Type:bind Source:/home/user/.local/share/buildkit/runc-native/snapshots/snapshots/2 Options:[rbind ro]}]: operation not permitted
  </code></pre>
  <p>Which is the error from the daemon.</p>
  <p>What strikes me is that I am able to containerise <code>buildkitd</code> inside a <code>minikube</code> cluster using the exact same YAML file as such:</p>
  <pre><code>NAME                             READY   STATUS    RESTARTS   AGE
  pod/buildkitd-5b46d94f5d-xvnbv   1/1     Running   0          36m
  
  NAME                 TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
  service/buildkitd    ClusterIP   10.100.72.194   &lt;none&gt;        1234/TCP   36m
  service/kubernetes   ClusterIP   10.96.0.1       &lt;none&gt;        443/TCP    36m
  
  NAME                        READY   UP-TO-DATE   AVAILABLE   AGE
  deployment.apps/buildkitd   1/1     1            1           36m
  
  NAME                                   DESIRED   CURRENT   READY   AGE
  replicaset.apps/buildkitd-5b46d94f5d   1         1         1       36m
  </code></pre>
  <p>I deploy the service and deployment inside <code>minikube</code> and forward the service port using the following command to be able to access the deployment outside <code>minikube</code>.</p>
  <pre><code>kubectl port-forward service/buildkitd 2000:1234
  </code></pre>
  <p>And with that setup I am able to execute my <code>buildctl</code> command without any issue (Image building and push to GCR).</p>
  <p>I wish to understand why it works on <code>minikube</code> and not on the Google Kubernetes Engine.</p>
  <p>Here is the container startup log if that is of any help</p>
  <pre><code>auto snapshotter: using native
  NoProcessSandbox is enabled. Note that NoProcessSandbox allows build containers to kill (and potentially ptrace) an arbitrary process in the BuildKit host namespace. NoProcessSandbox should be enabled only when the BuildKit is running in a container as an unprivileged user.
  found worker \&quot;wdukby0uwmjyvf2ngj4e71s4m\&quot;, labels=map[org.mobyproject.buildkit.worker.executor:oci org.mobyproject.buildkit.worker.hostname:buildkitd-5b46d94f5d-xvnbv org.mobyproject.buildkit.worker.snapshotter:native], platforms=[linux/amd64 linux/386]&quot;
  rootless mode is not supported for containerd workers. disabling containerd worker.
  found 1 workers, default=\&quot;wdukby0uwmjyvf2ngj4e71s4m\&quot;
  currently, only the default worker can be used.
  TLS is not enabled for tcp://0.0.0.0:1234. enabling mutual TLS authentication is highly recommended
  running server on /run/user/1000/buildkit/buildkitd.sock
  running server on [::]:1234
  </code></pre>