<p dir="auto">As <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/yujuhong/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/yujuhong">@yujuhong</a> noted it'd be useful to reduce variance of measurements. Additionally we may want to include fluentd pod in the measurements.</p>
<p dir="auto">Since 4:25PM the e2e test <code class="notranslate">ResourceUsage should not exceed expected amount</code> has been failing.<br>
Looking at the last batch of merged PRs I can't tell if any of them are responsible, either individually or in combination. Please can the PR authors <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/eparis/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/eparis">@eparis</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/BenTheElder/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/BenTheElder">@BenTheElder</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nikhiljindal/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nikhiljindal">@nikhiljindal</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/gmarek/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/gmarek">@gmarek</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/feihujiang/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/feihujiang">@feihujiang</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/yifan-gu/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/yifan-gu">@yifan-gu</a> have a think about whether their change could have caused:</p>
<div class="snippet-clipboard-content notranslate position-relative overflow-auto" data-snippet-clipboard-copy-content="ResourceUsage should not exceed expected amount.

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/monitor_resources.go:131
Expected
    &lt;map[string]e2e.resourceUsagePerContainer | len:3&gt;: {
        &quot;e2e-gce-minion-jt3o&quot;: {
            &quot;/&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.32753680271174745,
                MemoryUsageInBytes: 5536577126,
                MemoryWorkingSetInBytes: 1995254169,
                CPUInterval: 0,
            },
            &quot;/docker-daemon&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.05330590708577182,
                MemoryUsageInBytes: 3765198848,
                MemoryWorkingSetInBytes: 674401484,
                CPUInterval: 0,
            },
            &quot;/kubelet&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.04093180701209144,
                MemoryUsageInBytes: 94748672,
                MemoryWorkingSetInBytes: 94444748,
                CPUInterval: 0,
            },
            &quot;/kube-proxy&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.015246457387254289,
                MemoryUsageInBytes: 5681152,
                MemoryWorkingSetInBytes: 5656576,
                CPUInterval: 0,
            },
            &quot;/system&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.004761631665055137,
                MemoryUsageInBytes: 42254336,
                MemoryWorkingSetInBytes: 25874432,
                CPUInterval: 0,
            },
        },
        &quot;e2e-gce-minion-yb6a&quot;: {
            &quot;/system&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.00338632073811133,
                MemoryUsageInBytes: 43447910,
                MemoryWorkingSetInBytes: 26343014,
                CPUInterval: 0,
            },
            &quot;/&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.3377336907593952,
                MemoryUsageInBytes: 2699154227,
                MemoryWorkingSetInBytes: 936918220,
                CPUInterval: 0,
            },
            &quot;/docker-daemon&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.06698396036521867,
                MemoryUsageInBytes: 1778617548,
                MemoryWorkingSetInBytes: 339885260,
                CPUInterval: 0,
            },
            &quot;/kubelet&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.0644396340620269,
                MemoryUsageInBytes: 99394355,
                MemoryWorkingSetInBytes: 98837299,
                CPUInterval: 0,
            },
            &quot;/kube-proxy&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.010085711242831061,
                MemoryUsageInBytes: 4357324,
                MemoryWorkingSetInBytes: 4330291,
                CPUInterval: 0,
            },
        },
        &quot;e2e-gce-minion-itlb&quot;: {
            &quot;/&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.45461458647632885,
                MemoryUsageInBytes: 4603497676,
                MemoryWorkingSetInBytes: 1894986547,
                CPUInterval: 0,
            },
            &quot;/docker-daemon&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.059021967282394824,
                MemoryUsageInBytes: 2742996172,
                MemoryWorkingSetInBytes: 488832204,
                CPUInterval: 0,
            },
            &quot;/kubelet&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.06799305314744905,
                MemoryUsageInBytes: 104071168,
                MemoryWorkingSetInBytes: 103599308,
                CPUInterval: 0,
            },
            &quot;/kube-proxy&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.012785393033208461,
                MemoryUsageInBytes: 5046272,
                MemoryWorkingSetInBytes: 5021696,
                CPUInterval: 0,
            },
            &quot;/system&quot;: {
                Name: &quot;&quot;,
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.004112326988701547,
                MemoryUsageInBytes: 67950182,
                MemoryWorkingSetInBytes: 51807846,
                CPUInterval: 0,
            },
        },
    }
to be empty"><pre class="notranslate"><code class="notranslate">ResourceUsage should not exceed expected amount.

/go/src/k8s.io/kubernetes/_output/dockerized/go/src/k8s.io/kubernetes/test/e2e/monitor_resources.go:131
Expected
    &lt;map[string]e2e.resourceUsagePerContainer | len:3&gt;: {
        "e2e-gce-minion-jt3o": {
            "/": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.32753680271174745,
                MemoryUsageInBytes: 5536577126,
                MemoryWorkingSetInBytes: 1995254169,
                CPUInterval: 0,
            },
            "/docker-daemon": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.05330590708577182,
                MemoryUsageInBytes: 3765198848,
                MemoryWorkingSetInBytes: 674401484,
                CPUInterval: 0,
            },
            "/kubelet": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.04093180701209144,
                MemoryUsageInBytes: 94748672,
                MemoryWorkingSetInBytes: 94444748,
                CPUInterval: 0,
            },
            "/kube-proxy": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.015246457387254289,
                MemoryUsageInBytes: 5681152,
                MemoryWorkingSetInBytes: 5656576,
                CPUInterval: 0,
            },
            "/system": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.004761631665055137,
                MemoryUsageInBytes: 42254336,
                MemoryWorkingSetInBytes: 25874432,
                CPUInterval: 0,
            },
        },
        "e2e-gce-minion-yb6a": {
            "/system": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.00338632073811133,
                MemoryUsageInBytes: 43447910,
                MemoryWorkingSetInBytes: 26343014,
                CPUInterval: 0,
            },
            "/": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.3377336907593952,
                MemoryUsageInBytes: 2699154227,
                MemoryWorkingSetInBytes: 936918220,
                CPUInterval: 0,
            },
            "/docker-daemon": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.06698396036521867,
                MemoryUsageInBytes: 1778617548,
                MemoryWorkingSetInBytes: 339885260,
                CPUInterval: 0,
            },
            "/kubelet": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.0644396340620269,
                MemoryUsageInBytes: 99394355,
                MemoryWorkingSetInBytes: 98837299,
                CPUInterval: 0,
            },
            "/kube-proxy": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.010085711242831061,
                MemoryUsageInBytes: 4357324,
                MemoryWorkingSetInBytes: 4330291,
                CPUInterval: 0,
            },
        },
        "e2e-gce-minion-itlb": {
            "/": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.45461458647632885,
                MemoryUsageInBytes: 4603497676,
                MemoryWorkingSetInBytes: 1894986547,
                CPUInterval: 0,
            },
            "/docker-daemon": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.059021967282394824,
                MemoryUsageInBytes: 2742996172,
                MemoryWorkingSetInBytes: 488832204,
                CPUInterval: 0,
            },
            "/kubelet": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.06799305314744905,
                MemoryUsageInBytes: 104071168,
                MemoryWorkingSetInBytes: 103599308,
                CPUInterval: 0,
            },
            "/kube-proxy": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.012785393033208461,
                MemoryUsageInBytes: 5046272,
                MemoryWorkingSetInBytes: 5021696,
                CPUInterval: 0,
            },
            "/system": {
                Name: "",
                Timestamp: {sec: 0, nsec: 0, loc: nil},
                CPUUsageInCores: 0.004112326988701547,
                MemoryUsageInBytes: 67950182,
                MemoryWorkingSetInBytes: 51807846,
                CPUInterval: 0,
            },
        },
    }
to be empty
</code></pre></div>
<p dir="auto"><a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/ixdy/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/ixdy">@ixdy</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/quinton-hoole/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/quinton-hoole">@quinton-hoole</a> <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/dchen1107/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/dchen1107">@dchen1107</a></p>