What would be the best way to improve this Gitlab CI/CD (Gitlab.com) pipeline's efficiency and consistency? This project has a pipeline that runs integration tests against a spun up instance of Gitlab. Each Job in the integration test stage has its own instance testing different build distributions (es, cjs, original source). Unfortunately, the shared runners have limited resources and inconsistently cause these jobs to fail because the spun up GitLab instances have insufficient resources. This occurs more often when I have more than one instance spin up at a time since the jobs run in parallel.
Here is a test repo showcasing the problem and it's pipeline
Gitlab Support has suggested using a dedicated runner OR refactoring my pipeline. Since I'm trying to maintain a $0 cost for this open-source project, the only feasible suggestion (that I'm aware of) is the latter one.
I've tested doing just this, specifically splitting these integration tests into separate stages to stop creating multiple GitLab instances at the same time. This, however, has not improved the results. The only other idea I can think of is running all the integration tests in the same script, instead of separate stages or jobs, which would only require spinning up one instance of GitLab. In this solution, I would have to add some sort of id's to the data being created in the tests to avoid having clashes (i.e., creating a project must use a unique name)
What I want to figure out is if I'm overlooking something? Are there any better ways to improve the pipeline to avoid the resource bottlenecks causing the crashing?
Can you explain the reason for why the &quot;step_script&quot; installs the dependencies and not use a prebuild images that has all the dependencies installed?
Solution
Based on the information you provided, Your test infrastucture rebuild even the test container from scratch.
You tests take a long time to be done as those lines indicate also a fresh composing of the docker container:

(1/11) Installing ca-certificates (20191127-r2) 
(2/11) Installing nghttp2-libs (1.40.0-r1) 
(3/11) Installing libcurl (7.67.0-r3) 
(4/11) Installing expat (2.2.9-r1) 
(5/11) Installing pcre2 (10.34-r1) 
(6/11) Installing git (2.24.3-r0) 
(7/11) Installing c-ares (1.15.0-r0) 
(8/11) Installing libgcc (9.3.0-r0) 
(9/11) Installing libstdc++ (9.3.0-r0) 
(10/11) Installing nodejs (12.20.1-r0) 
(11/11) Installing yarn (1.19.2-r0)

If you build a docker image with those thing preinstalled, your Tests will run much faster.
This Process creates clean environment where the only change to it will be the code that has changed!!
The new Docker image then need to be only pulled ones and can be reused.
Build/test infra
docker run -v &quot;$(pwd)&quot;/target:/app:ro -name BUILDID jdalrymple/docker-compose-with-node yarn test
run that for each of your tests.