<p dir="auto">Docker does a credential check when pulling images, but not when using them. It will let you run an image that is already pulled even if you don't present credentials.</p>
<p dir="auto">That is reasonable behavior from the standpoint of the docker tool since it is not trying to be multi-user.</p>
<p dir="auto">However, it makes it hard to make Kubernetes multi-user. If an image is already pulled, (e.g. using <code class="notranslate">imagePullSecrets</code>, then anyone else can use that image by setting pull policy to <code class="notranslate">IfNotPresent</code>, even if they lack proper credentials.</p>
<p dir="auto">We could try to fix this by disallowing <code class="notranslate">IfNotPresent</code> pull policy on all pods, thus forcing a docker pull before each docker run, and failing the pod if the docker pull failed.  We would need to implement and e2e test that.</p>
<h1 dir="auto">Problem statement</h1>
<p dir="auto">Some images require pull secrets in order to be used by the Kubelet. Pull secrets are per namespace. Imagine the following scenario:</p>
<ol dir="auto">
<li>If a pod in namespaceA has valid pull secrets for <code class="notranslate">foo/bar:stable</code>, when it is scheduled to a given node, the Kubelet uses the pull secrets to pull the image to the node</li>
<li>If a pod in namespaceB also refers to <code class="notranslate">foo/bar:stable</code> but does not have valid pull secrets for it, if it happens to be scheduled to the same node as the pod in step 1, the Kubelet respects the image pull policy</li>
<li>Because the image already exists on the node, if the pull policy is Never or IfNotPresent, the Kubelet runs the pod, even though namespaceB shouldn't be able to use that image.</li>
</ol>
<h1 dir="auto">Proposed solutions</h1>
<h2 dir="auto">Option 1 - Kubelet flag to ignore pull policy and always try to pull</h2>
<p dir="auto">Add a flag to the kubelet, <code class="notranslate">--always-pull-images</code> (or something similarly named) that, if true, would force the Kubelet to attempt to pull every image prior to running a pod, regardless of the image's pull policy.</p>
<p dir="auto">Pros:</p>
<ul dir="auto">
<li>Easy to implement</li>
<li>Guarantees that a namespace is allowed to use an image</li>
</ul>
<p dir="auto">Cons:</p>
<ul dir="auto">
<li>Overrides user-specified intent in image pull policy</li>
<li>Requires remote registry to be online to check authorization</li>
<li>Introduces small amount of latency to pod startup time (because of the additional network requests to the registry)</li>
</ul>
<h2 dir="auto">Option 2 - Keep an in-memory cache of which namespaces are allowed to use which images</h2>
<p dir="auto">After an image has been successfully pulled, record the mapping from namespace to image in an in-memory cache in the Kubelet. Subsequent attempts by pods in the same namespace to use the same image would respect the image pull policy. Any attempts by pods in other namespaces to use this image would ignore the image pull policy and instead force a pull attempt.</p>
<p dir="auto">Pros:</p>
<ul dir="auto">
<li>Doesn't override user-specified intent in image pull policy</li>
<li>No added latency to pod startup time</li>
</ul>
<p dir="auto">Cons:</p>
<ul dir="auto">
<li>Doesn't handle authorization revocation; if the credentials to pull an image have changed in the remote registry, there is no way to evict an entry from the cache, possibly allowing access to an image that would otherwise fail with option 1. This could be mitigated if the entries in the cache have a TTL.</li>
</ul>
<h1 dir="auto">Suggested path forward</h1>
<p dir="auto">Looking for feedback from the community on this one. I'd like to implement option 1 in the short term, with the default being false (so image pull policy is respected) to keep the status quo. If we can come to an agreement on option 2, I'd like to move to that eventually.</p>
<p dir="auto">@kubernetes/goog-node @kubernetes/rh-cluster-infra @kubernetes/sig-node</p>