<p dir="auto">With the rename to argmax, and dims now a kwarg, it seems very tempting to me to define a 2-arg flavor, <code class="notranslate">argmax(f, itr)</code>, where <code class="notranslate">itr</code> is an iterable subset of the domain of <code class="notranslate">f</code>. This would return an <code class="notranslate">x</code> in <code class="notranslate">itr</code> that maximizes <code class="notranslate">f(x)</code>, which is quite literally the mathematical definition of argmax.</p>
<p dir="auto">(originally proposed <a href="https://discourse.julialang.org/t/add-a-2-arg-flavor-argmax-f-domain/11720" rel="nofollow">here</a>)</p>
<p dir="auto">Edit: Previous wording may have been confusing, so I have attempted to improve it.</p>
<p dir="auto">As progress moves forward on v0.5 development (especially <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="106669708" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/13157" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/13157/hovercard" href="https://github.com/JuliaLang/julia/issues/13157">#13157</a>), we'll need an automated system for executing benchmarks and identifying performance regressions.</p>
<p dir="auto">The Julia group has recently purchased dedicated performance testing hardware which is hosted at CSAIL, and I've been brought on to facilitate the development of a system that takes advantage of this hardware. I'm hoping we can use this issue to centralize discussion/development efforts.</p>
<h4 dir="auto">Desired features</h4>
<p dir="auto">Any implementation of a performance tracking system should:</p>
<ul dir="auto">
<li>track multiple metrics (e.g. time, % GC time, bytes allocated)</li>
<li>allow collaborators to trigger the following via GitHub's UI:
<ul dir="auto">
<li>benchmark execution on a specific commit</li>
<li>benchmark result comparison between two specific commits</li>
</ul>
</li>
<li>incorporate a tagging system so that groups of benchmarks can be selectively run by topic (e.g. run only <code class="notranslate">parallel</code> benchmarks, or only <code class="notranslate">string</code> benchmarks)</li>
<li>store benchmark result history for future analysis</li>
<li>report benchmark status/results through GitHub's UI</li>
<li>be applicable to normal Julia packages, not just Base (envision a future where PackageEvaluator benchmarks registered packages)</li>
<li>preferably be written in native Julia</li>
</ul>
<p dir="auto">Feel free to chime in with additional feature goals - the ones listed above just outline what I've been focusing on so far.</p>
<h4 dir="auto">Existing work</h4>
<p dir="auto">In order to make progress on this issue, I've been working on <a href="https://github.com/JuliaCI/BenchmarkTrackers.jl">JuliaCI/BenchmarkTrackers.jl</a>, which supplies a unified framework for writing, executing, and tracking benchmarks. It's still very much in development, but currently supports all of the goals listed above. I encourage you to check it out and open up issues/PRs in that repository if you have ideas or concerns. Just don't expect the package to be stable yet - I'm in the process of making some drastic changes (mainly to improve the testability of the code).</p>
<p dir="auto">Here are some other packages that any interested parties will want to be familiar with:</p>
<ul dir="auto">
<li><a href="https://github.com/johnmyleswhite/Benchmarks.jl">johnmyleswhite/Benchmarks.jl</a>: a framework for benchmark execution, with the goal of delivering more rigorous/useful metrics than those provided by simple calls to <code class="notranslate">@time</code>.</li>
<li><a href="https://github.com/JuliaWeb/GitHub.jl">JuliaWeb/GitHub.jl</a>: will soon contain the <a href="https://github.com/JuliaWeb/GitHub.jl/compare/master...jrevels:jr/all">GitHub event-handling infrastructure that I've been working on</a>, which allows stuff like comment-based triggers and GitHub status reporting.</li>
<li><a href="https://github.com/IainNZ/BasePerfTests.jl">IainNZ/BasePerfTests.jl</a>: a collection of performance tests (I'm not sure that this is still being developed).</li>
<li><a href="https://github.com/staticfloat/Perftests.jl">staticfloat/Perftests.jl</a>: another collection of performance tests + a grouping/logging framework for ease of execution. I believe <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/staticfloat/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/staticfloat">@staticfloat</a> has been running these on occasion via <a href="https://github.com/staticfloat/julia-buildbot">the buildbots</a>.</li>
</ul>
<p dir="auto">Eventually we will want to consolidate "blessed" benchmarking/CI packages under an umbrella Julia group on Github (maybe <a href="https://github.com/JuliaCI">JuliaCI</a>)?</p>
<p dir="auto"><a href="http://speed.julialang.org/" rel="nofollow">I saw that Codespeed was used for a while</a>, but that effort was abandoned due to the burden of maintaining the server through volunteer effort. I've also been told that Codespeed didn't integrate well with the Github-centric CI workflow that we've become accustomed to.</p>
<h4 dir="auto">Resolving this issue</h4>
<p dir="auto">Taking into account the capabilities previously mentioned, I imagine that a single CI benchmark cycle for Base would go through these steps:</p>
<ol dir="auto">
<li>Collaborator comments on a commit with the appropriate trigger phrase</li>
<li>A comment-listening server (the meat of which I've implemented <a href="https://github.com/jrevels/GitHub.jl/commit/d44f70a45f2ecd37ab61908476eaf8942572f626">here</a>) triggers <del>the buildbots</del> a CI server to build Julia at that commit</li>
<li><del>The buildbots call out to</del> BenchmarkTrackers is used to process an external package of performance tests (similar to Perftests, but written using BenchmarkTrackers)</li>
<li>BenchmarkTrackers reports the results via GitHub statuses to the commit. Any additional reporting can be done with the <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nanosoldier/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nanosoldier">@nanosoldier</a> bot ("nanosolider" is the hostname of the hardware at CSAIL, and <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/jakebolewski/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/jakebolewski">@jakebolewski</a> managed to grab the GitHub account name).</li>
</ol>
<p dir="auto">If we can deliver the workflow described above, I think that will be sufficient to declare this issue "resolved."</p>
<h4 dir="auto">Next steps</h4>
<p dir="auto">The following still needs to get done:</p>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Configure a webhook on the JuliaLang/julia repo for firing off events so that we can begin testing the necessary CI infrastructure</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Get BenchmarkTrackers.jl to a release-ready state, which means more robust testing and further development on both itself and its dependencies, GitHub.jl and Benchmarks.jl</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> <a href="https://github.com/JuliaCI/BaseBenchmarks.jl">Start organizing/rewriting existing performance tests using BenchmarkTrackers.jl</a></li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> Set up the <a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/users/nanosoldier/hovercard" data-octo-click="hovercard-link-click" data-octo-dimensions="link_type:self" href="https://github.com/nanosoldier">@nanosoldier</a> to do any additional status reporting we want (e.g. <a href="https://github.com/JuliaDiff/ForwardDiff.jl/commit/180f9dfa5e4616989b67ef46047e9682914a6a84#commitcomment-14070341">linking to a commit status page after somebody triggers benchmark execution</a> - GitHub doesn't do a good job of displaying statuses for commits outside of PRs).</li>
</ul>
<h4 dir="auto">Regression Examples</h4>
<p dir="auto">Here are some examples from regression-prone areas that I think could be more easily moderated with an automated performance tracking system:</p>
<ul dir="auto">
<li>parallel (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="102958895" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/12794" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/12794/hovercard" href="https://github.com/JuliaLang/julia/issues/12794">#12794</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="96039339" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/12223" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/12223/hovercard" href="https://github.com/JuliaLang/julia/issues/12223">#12223</a>)</li>
<li>inlining/type inference (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="110864463" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/13551" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/13551/hovercard" href="https://github.com/JuliaLang/julia/issues/13551">#13551</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="108756041" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/13350" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/13350/hovercard" href="https://github.com/JuliaLang/julia/issues/13350">#13350</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="99305292" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/12476" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/12476/hovercard" href="https://github.com/JuliaLang/julia/issues/12476">#12476</a>)</li>
<li>"real-world" use cases (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="90414767" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/11823" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/11823/hovercard" href="https://github.com/JuliaLang/julia/issues/11823">#11823</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="34414213" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/7000" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/7000/hovercard" href="https://github.com/JuliaLang/julia/issues/7000">#7000</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="30885636" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/6415" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/6415/hovercard" href="https://github.com/JuliaLang/julia/issues/6415">#6415</a>)</li>
<li>individual functions (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="88010306" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/11700" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/11700/hovercard" href="https://github.com/JuliaLang/julia/issues/11700">#11700</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="64801464" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/10650" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/10650/hovercard" href="https://github.com/JuliaLang/julia/issues/10650">#10650</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="40983599" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/8100" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/8100/hovercard" href="https://github.com/JuliaLang/julia/issues/8100">#8100</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="29160118" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/6112" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/6112/hovercard" href="https://github.com/JuliaLang/julia/issues/6112">#6112</a>)</li>
<li>codegen/vectorization (<a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="113418940" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/13777" data-hovercard-type="pull_request" data-hovercard-url="/JuliaLang/julia/pull/13777/hovercard" href="https://github.com/JuliaLang/julia/pull/13777">#13777</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="108150984" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/13301" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/13301/hovercard" href="https://github.com/JuliaLang/julia/issues/13301">#13301</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="91518358" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/11899" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/11899/hovercard" href="https://github.com/JuliaLang/julia/issues/11899">#11899</a>, <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="92772689" data-permission-text="Title is private" data-url="https://github.com/JuliaLang/julia/issues/11997" data-hovercard-type="issue" data-hovercard-url="/JuliaLang/julia/issues/11997/hovercard" href="https://github.com/JuliaLang/julia/issues/11997">#11997</a>)</li>
</ul>