tl;dr
=====

How do I pass data, e.g. the `$BUILD_VERSION` variable, between jobs in different pipelines in Gitlab CI?

So (in my case) this:

```
Pipeline 1 on push ect.            Pipeline 2 after merge

    `building` job ...                `deploying` job
          │                                ▲
          └─────── $BUILD_VERSION ─────────┘

```

---

Background
==========

Consider the following example (full `yml` below):

```yaml
building:
    stage: staging
    # only on merge requests
    rules:
        # execute when a merge request is open
        - if: $CI_PIPELINE_SOURCE == "merge_request_event"
          when: always
        - when: never
    script:
        - echo "BUILD_VERSION=1.2.3" > build.env
    artifacts:
        reports:
            dotenv: build.env

deploying:
    stage: deploy
    # after merge request is merged
    rules:
        # execute when a branch was merged to staging
        - if: $CI_COMMIT_BRANCH == $STAGING_BRANCH
          when: always
        - when: never
    dependencies: 
        - building
    script:
        - echo $BUILD_VERSION

```

I have two stages, *staging* and *deploy*. The `building` job in *staging* builds the app and creates a "Review App" (no separate build stage for simplicity). The `deploying` job in *deploy* then uploads the new app.

The pipeline containing the `building` job runs whenever a merge request is opened. This way the app is built and the developer can click on the "Review App" icon in the merge request. The `deploying` job is run right after the merge request is merged. The idea is the following:

```
                      *staging* stage (pipeline 1)        *deploy* stage (pipeline 2)

<open merge request> -> `building` job (and show)   ...   <merge> -> `deploying` job
                             │                                            ▲
                             └───────────── $BUILD_VERSION ───────────────┘

```

The problem for me is, that the *staging*/`building` creates some data, e.g. a `$BUILD_VERSION`. I want to have this `$BUILD_VERSION` in the *deploy*/`deploying`, e.g. for creating a new release via the Gitlab API.

So my question is: How do I pass the `$BUILD_VERSION` (and other data) from *staging*/`building` to *deploy*/`deploying`?

---

What I've tried so far
======================

`artifacts.reports.dotenv`
--------------------------

The described case is more less handled in the gitlab docs in [Pass an environment variable to another job](https://docs.gitlab.com/ee/ci/variables/#pass-an-environment-variable-to-another-job). Also the `yml` file shown below is heavily inspired by this example. Still, it does not work.

The `build.env` artifact is created in `building`, but whenever the `deploying` job is executed, the `build.env` file gets removed as shown below in line 15: "Removing build.env". I tried to add `build.env` to the `.gitignore` but it still gets removed.

[![Preparing environment - Running on runner- via gitlab-runner... - Getting source from Git repository - Fetching changes with git depth set to 50... - Reinitialized existing Git repository in  - Checking out as staging... - Removing build.env - Skipping Git submodules setup - Executing "step_script" stage of the job script - Using docker image - echo $BUILD_VERSION - Job succeeded](https://i.stack.imgur.com/pYibY.png)](https://i.stack.imgur.com/pYibY.png)

After hours of searching I found in [this gitlab issue comment](https://gitlab.com/gitlab-org/gitlab/-/issues/246777#note_417210007) and [this stackoverflow post](https://stackoverflow.com/a/65878853/5934316) that the `artifacts.reports.dotenv` doesn't work with the `dependencies` or the `needs` keywords.

Removing `dependencies` doesn't work. Using `needs` only doesn't work either. Using both is not allowed.

Does anyone know a way how to get this to work? I feel like this is the way it should work.

Getting the artifacts as a file
-------------------------------

This answer of the stackoverflow post [Gitlab ci cd removes artifact for merge requests](https://stackoverflow.com/a/65878853/5934316) suggests to use the `build.env` as a normal file. I also tried this. The (relevant) `yml` is the following:

```yaml
building:
    # ...
    artifacts:
        paths:
            - build.env

deploying:
    # ...
    before_script:
        - source build.env

```

The result is the same as above. The `build.env` gets removed. Then the `source build.env` command fails because `build.env` does not exist. (Doesn't matter if `build.env` is in the `.gitignore` or not, tested both)

Getting the artifacts from the API
----------------------------------

I also found the answer of the stackoverflow post [Use artifacts from merge request job in GitLab CI](https://stackoverflow.com/a/66344557/5934316) which suggests to use the API together with `$CI_JOB_TOKEN`. But since I need the artifacts in a non-merge-request pipeline, I cannot use the suggested `CI_MERGE_REQUEST_REF_PATH`.

I tried to use `$CI_COMMIT_REF_NAME`. The (important section of the) `yml` is then:

```yaml
deploying:
    # ...
    script:
        - url=$CI_API_V4_URL/projects/jobs/artifacts/$CI_COMMIT_REF_NAME/download?job=building
        - echo "Downloading $url"
        - 'curl --header "JOB-TOKEN: ${CI_JOB_TOKEN}" --output $url'
        # ...

```

But this the API request gets rejected with "404 Not Found". Since [commit SHAs are not supported](https://docs.gitlab.com/ee/api/job_artifacts.html#download-a-single-artifact-file-from-specific-tag-or-branch), `$CI_COMMIT_BEFORE_SHA` or `$CI_COMMIT_SHA` do not work either.

Using `needs`
-------------

*Update*: I found the section [Artifact downloads between pipelines in the same project](https://docs.gitlab.com/ee/ci/yaml/#artifact-downloads-between-pipelines-in-the-same-project) in the gitlab docs which is exactly what I want. But: I can't get it to work.

The `yml` looks like the following after more less copying from the docs:

```yaml
building:
    # ...
    artifacts:
        paths:
            - version
        expire_in: never

deploying:
    # ...
    needs:
        - project: $CI_PROJECT_PATH
          job: building
          ref: staging # building runs on staging branch, main doesn't work either
          artifacts: true

```

Now the `deploying` job instantly fails and I get the following error banner:

[![This job depends on other jobs with expired/erased artifacts:
Please refer to https://docs.gitlab.com/ee/ci/yaml/README.html#dependencies](https://i.stack.imgur.com/LCXJr.jpg)](https://i.stack.imgur.com/LCXJr.jpg)

I tried to set `artifacts.expire_in = never` (as shown) but I still get the same error. Also in *Settings* > *CI/CD* > *Artifacts* "Keep artifacts from most recent successful jobs" is selected. So the artifact should be present. What did I miss here? This should work according to the docs!

---

I hope somebody can help me on getting the `$BUILD_VERSION` to the `deploying` job. If there are other ways than the ones I've tried, I'm very happy to hear them. Thanks in advance.

---

The example `.gitlab-ci.yml`:

```yaml
stages:
    - staging
    - deploy

building:
    tags: 
        - docker
    image: bash
    stage: staging
    rules:
        - if: ($CI_PIPELINE_SOURCE == "merge_request_event") && $CI_MERGE_REQUEST_TARGET_BRANCH_NAME == "staging"
          when: always
        - when: never
    script:
        - echo "BUILD_VERSION=1.2.3" > build.env
    artifacts:
        reports:
            dotenv: build.env
    environment:
        name: Example
        url: https://example.com

deploying:
    tags: 
        - docker
    image: bash
    stage: deploy
    rules:
        - if: $CI_COMMIT_BRANCH == "staging"
          when: always
        - when: never
    dependencies:
        - building
    script:
        echo $BUILD_VERSION

```