title: GKE docker image pull extremely slow with low RAM node machine types
body: <p>I am deploying onto a cluster on GKE and image pulling seems to be sometimes untenably slow and also wildly inconsistent. Are these times really what is considered acceptable, or might it indicate a problem in my config somewhere?</p>
  <p>The artifact registry is europe-west2 and I have spun up a cluster (1.21.5-gke.1302) in europe-west2-a. Other than making it single node and using EC2 'small' instances instead of medium I have stuck with the defaults, other than trying with Image streaming both enabled and disabled (which seems to make no difference - at least to first deployment.)</p>
  <p>Pulling times:</p>
  <ul>
  <li><p>k8s.gcr.io/ingress-nginx/controller:v1.0.4 takes up to 6.5 minutes</p>
  </li>
  <li><p>redis takes up to 10 minutes</p>
  </li>
  <li><p>my own image (hosted on the artifact repository - around 3.5G) takes up to 30 minutes</p>
  </li>
  <li><p>another of my own images (hosted on the artifact repository - around 1G) has taken only 53 seconds, however.</p>
  </li>
  </ul>
  <p>It seems to get slower the further through the deployment it gets, which takes about an hour in total. Can anyone think of any config issues that may be causing this? I'm thinking possibly some API limits that I'm not aware of, kubelet download speeds, or too many concurrent downloads...</p>
  <p>--- Edit</p>
  <p>As I typed the above, I had an idea, and it turns out to be the answer. The type of instance makes a difference. I tried again with a cluster using 'medium' EC2 (so 4G of RAM instead of 2G) and it deployed in 15 minutes rather than an hour, again getting noticeably slower pull speeds after (I assume?) the 4G mark is somehow met. I then made a cluster with 'e2-standard-2' (now 8G of RAM) and that totalled only 3.5 minutes! (But again with this attenuation in pull speed as the deployment went on.)</p>
  <p>I had assumed that the images would be downloaded onto a 'real' disk, but maybe all space on my cluster is virtualised in some way and so the RAM size makes a difference even for deployment? But if this is the case and there's no underlying filesystem to provide swap space, I'm once again confused since then a 3G image shouldn't download at all on 2G of RAM! So my new question is why does this make a difference? I would like to have a cluster of nodes that serve the purpose of my app, not that are overpowered just so I can deploy in a reasonable time. (Tin-foil hat time - is it possible this is as designed by Google to encourage people to use the more expensive option?)</p>