<story><title>Advantages of incompetent management</title><url>https://yosefk.com/blog/advantages-of-incompetent-management.html</url></story><parent_chain><item><author>Too</author><text>From what I&amp;#x27;ve seen this is usually caused by CI pipelines being overly cautious, starting from a pristine state and trying to cover every possible edge case.&lt;p&gt;Spin up new node, congrats, now you have to clone the whole repo from scratch. Oh, &amp;quot;npm install&amp;quot; was not cached, let&amp;#x27;s download half of the internet again, pull &amp;quot;:latest&amp;quot; of some multi-GB docker images while you are at it. For good measure, &amp;quot;make clean&amp;quot;, because...you never know and nobody bothered to architect our build system to use a  more sane tool than Makefiles, btw remember to only use make -j3, because rumor has it that the build fails intermittently when using more cores! Finally, run every test that couldn&amp;#x27;t possibly be affected by the commit in question, preferably the big e2e-suite. As someone else mentioned above, all of this running on some dog-slow cloud network attached disk.&lt;p&gt;There you have it. Spinning up more compute is not the solution, it&amp;#x27;s the cause. Keep one machine on and improve on your build tooling. I guarantee it will be magnitudes faster than any ephemeral pipeline.</text></item><item><author>XorNot</author><text>This has been the bane of my entire career. I have never, not once, encountered a CI system that could run the test suite faster then my development machine.&lt;p&gt;Which is &lt;i&gt;absurd&lt;/i&gt;. Cloud resources on servers with more CPUs then I have, and yet the difference in timing is measured in tens of minutes.</text></item><item><author>vbezhenar</author><text>Why does it happen? My CI launches separate kubernetes pod for every incoming task. If there are not enough servers, autoscaler will spin up new and remove it after some time. So there are not much expenses, we&amp;#x27;re paying for what we using. I feel that fast CI iterations are very important for developers.</text></item><item><author>yummypaint</author><text>&lt;i&gt;Things work the same with any resource, not just actual money - it could be team size, or processor cycles and memory bytes. If you free up 200 ms of CPU cycles and 500 megs of RAM, someone else can deploy their functionality using these newly available resources, and then you won’t be able to. In fact, a mature, well-run CI system will measure everyone’s resource footprint after each commit, and will not let you exceed your budget, which was frozen at some point based on how much you were using at the time (hope it was a lot! - always spend like crazy before the baseline is established!) Is it any wonder that people learn to never optimize their code - unless they want to deploy something new themselves, and only after asking for more resources to deploy it and not getting any?&lt;/i&gt;&lt;p&gt;This explains so much. It also sounds like broken water laws in the western US that incentivise farmers to intentionally waste water to preserve their future ability to waste water.</text></item></parent_chain><comment><author>torginus</author><text>This is usually a result of following &amp;#x27;cloud best practices&amp;#x27; instead of being pragmatic.&lt;p&gt;For example, using Kubernetes with Docker where each pod is some virtual ECS 4-core whatever instance, assuming scaling the workload over a 1000 instances will be fast. Which in your case will lead to said pods individually spending 90% of  their time running &amp;#x27;npm install&amp;#x27; , and each pod is way slower than your desktop PC.&lt;p&gt;Different tests have different needs and bottlenecks. For example, if you&amp;#x27;re running unit tests, with no external dependencies, you&amp;#x27;d want to separate out the build process, and distribute the artifacts to multiple test machines that run tests in parallel.&lt;p&gt;The choice of using a few big machines or more small ones is usually up to you, and is a wash cost-wise. However I do need to point out AWS sells you 192 core machines, which I&amp;#x27;d wager are WAY faster than what you&amp;#x27;re sitting in front of.&lt;p&gt;And Amdahl&amp;#x27;s law is a thing as well. If there&amp;#x27;s a 10 minute non-parallizable build process, doing a 10 minute test run on a 192 core machine vs doing an ~0 minute test run on infinite machines ends up being only a 2x speedup.&lt;p&gt;And there&amp;#x27;s such a thing as setup costs. Spinning up an AWS machine from an image, setting up network interfaces, configuring the instance from scratch has a cost associated with it as well. And managing a cluster of 1000 computers also has its set of unique challenges. And assuming that if you ask Amazon for a 1000 machines at a drop of the hat, and plan to use each for a minute or two, AWS will throttle the fuck out of you.&lt;p&gt;All I&amp;#x27;m trying to say with this incoherent rambling is KISS  - know your requirements, and build the simplest infra that can satisfy it. Having a few large stopped instances in a pool might be all the complexity you need, and while it flies in the face of cloud best practices, it&amp;#x27;s probably going to be the fastest to start up.</text></comment>
<story><title>Advantages of incompetent management</title><url>https://yosefk.com/blog/advantages-of-incompetent-management.html</url></story><parent_chain><item><author>Too</author><text>From what I&amp;#x27;ve seen this is usually caused by CI pipelines being overly cautious, starting from a pristine state and trying to cover every possible edge case.&lt;p&gt;Spin up new node, congrats, now you have to clone the whole repo from scratch. Oh, &amp;quot;npm install&amp;quot; was not cached, let&amp;#x27;s download half of the internet again, pull &amp;quot;:latest&amp;quot; of some multi-GB docker images while you are at it. For good measure, &amp;quot;make clean&amp;quot;, because...you never know and nobody bothered to architect our build system to use a  more sane tool than Makefiles, btw remember to only use make -j3, because rumor has it that the build fails intermittently when using more cores! Finally, run every test that couldn&amp;#x27;t possibly be affected by the commit in question, preferably the big e2e-suite. As someone else mentioned above, all of this running on some dog-slow cloud network attached disk.&lt;p&gt;There you have it. Spinning up more compute is not the solution, it&amp;#x27;s the cause. Keep one machine on and improve on your build tooling. I guarantee it will be magnitudes faster than any ephemeral pipeline.</text></item><item><author>XorNot</author><text>This has been the bane of my entire career. I have never, not once, encountered a CI system that could run the test suite faster then my development machine.&lt;p&gt;Which is &lt;i&gt;absurd&lt;/i&gt;. Cloud resources on servers with more CPUs then I have, and yet the difference in timing is measured in tens of minutes.</text></item><item><author>vbezhenar</author><text>Why does it happen? My CI launches separate kubernetes pod for every incoming task. If there are not enough servers, autoscaler will spin up new and remove it after some time. So there are not much expenses, we&amp;#x27;re paying for what we using. I feel that fast CI iterations are very important for developers.</text></item><item><author>yummypaint</author><text>&lt;i&gt;Things work the same with any resource, not just actual money - it could be team size, or processor cycles and memory bytes. If you free up 200 ms of CPU cycles and 500 megs of RAM, someone else can deploy their functionality using these newly available resources, and then you won’t be able to. In fact, a mature, well-run CI system will measure everyone’s resource footprint after each commit, and will not let you exceed your budget, which was frozen at some point based on how much you were using at the time (hope it was a lot! - always spend like crazy before the baseline is established!) Is it any wonder that people learn to never optimize their code - unless they want to deploy something new themselves, and only after asking for more resources to deploy it and not getting any?&lt;/i&gt;&lt;p&gt;This explains so much. It also sounds like broken water laws in the western US that incentivise farmers to intentionally waste water to preserve their future ability to waste water.</text></item></parent_chain><comment><author>TeMPOraL</author><text>Not my experience. The CI systems I dealt with most recently, for example, were very much compute-bound. Way too weak VMs for the runnres, and way too few of them. With CI builds taking each of upward 2h, and capacity to run maybe 10 of them in parallel, this pretty much destroys any ability to work with small commits - you have to squash them before submitting for review anyway, otherwise you destroy anyone else&amp;#x27;s ability to get their changeset through CI for the rest of the day.&lt;p&gt;This is entirely solvable by getting more resources. At least 2x more runners, each at least 2x as beefy. You could go 10x and I doubt it would be anywhere as expensive as the amount of money company wastes on devs waiting for CI. Alas, good luck convincing people managing the infra of this.</text></comment>