<p dir="auto">I have a 50 node cluster with 1 rc with 1500 replicas. Each pod list takes few second and saturates 4 core master for that time. This slows down the whole system including scheduler, node/pod status updates, controllers etc.</p>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/wojtek-t/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/wojtek-t">@wojtek-t</a></p>
<h3 dir="auto">Background</h3>
<p dir="auto">Currently, to serve LIST operation in apiserver we do the following:</p>
<ul dir="auto">
<li>read all objects of a given resource from etcd</li>
<li>filter out objects that user is not interested in</li>
<li>return the result</li>
</ul>
<p dir="auto">This means, that complexity of LIST operation is proportional to the number of all objects of a given type, not to the number of objects returned to the user.</p>
<p dir="auto">As an example, consider 1000-node cluster with 30.000 pods running in it. If a user has a ReplicationController with, say 10 replicas, then listing them requires requires reading all 30.000 from etcd, filtering them and returning just 10 which are interesting for the user.</p>
<h3 dir="auto">Proposal</h3>
<p dir="auto">We would like to make the LIST operation proportional to the number of elements it returns as a result.</p>
<p dir="auto">With the "Cacher" layer (used by watch in apiserver: <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="91791454" data-permission-text="Title is private" data-url="https://github.com/kubernetes/kubernetes/issues/10475" data-hovercard-type="issue" data-hovercard-url="/kubernetes/kubernetes/issues/10475/hovercard" href="https://github.com/kubernetes/kubernetes/issues/10475">#10475</a>), we already store a copies of objects with a given type in apiserver so if we add an Indexer to it (which should be simple), we would be able to effectively serve list operations from it.</p>
<p dir="auto">The problem is that a cache in apiserver is delayed (usually by tens to hundreds milliseconds). So if we just start to serve list operations from there we will change the semantics - e.g. it can happen that if you POST an object to apiserver and call LIST immediately after, the returned list may not contain already POSTed object.</p>
<p dir="auto">In many cases, this doesn't really matter, for example if LIST is only done to start watching from that point, we will get eventual consistency anyway (just the starting point changes). However, it can possibly break some other clients.</p>
<p dir="auto">I can see 3 main options:</p>
<ul dir="auto">
<li>just change the semantics</li>
<li>add an option (parameter) to LIST operation to force listing from database; however, in this case we should probably make listing from database by switched off by default to take advantage from it wherever it's possible</li>
<li>change the LIST operation to take ResourceVersion paramater and ensure that what we return in a result is no older than the given ResourceVersion (however, to make it efficient, we don't give any guarantees on how fresh the result is - we only promise to return the result not older than the given ResourceVersion).</li>
</ul>
<p dir="auto">[The third option is the best in my opinion].</p>
<p dir="auto">What do you think about it?</p>
<p dir="auto">@kubernetes/goog-control-plane @kubernetes/goog-csi<br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/lavalamp/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/lavalamp">@lavalamp</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/brendandburns/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/brendandburns">@brendandburns</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/quinton-hoole/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/quinton-hoole">@quinton-hoole</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/bgrant0607/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/bgrant0607">@bgrant0607</a><br>
<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/smarterclayton/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/smarterclayton">@smarterclayton</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/derekwaynecarr/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/derekwaynecarr">@derekwaynecarr</a></p>