# Source: concourse
---

- type: replace
  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties/prometheus/custom_rules?/-
  value:
    name: ConcourseSmoketestsFailures
    rules:
      - record: "concourse_continuous_smoke_tests_failures"
        expr: sum(concourse_builds_finished{exported_job="continuous-smoke-tests",pipeline="create-cloudfoundry",status="failed"} or (absent(concourse_builds_finished{exported_job="continuous-smoke-tests",pipeline="create-cloudfoundry",status="failed"})-1))

      - alert: ConcourseSmoketestsFailures
        expr: increase(concourse_continuous_smoke_tests_failures[1h]) >= 3
        labels:
          severity: critical
        annotations:
          summary: Concourse continuous-smoke-tests failures
          description: The continuous-smoke-tests Concourse job has been failing for a while now.

# Source: concourse
---

- type: replace
  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties/prometheus/custom_rules?/-
  value:
    name: ConcourseSmoketestsFailures
    rules:
      - alert: ConcourseSmoketestsFailuresWarning
        expr: increase(concourse_builds_finished{exported_job="continuous-smoke-tests",pipeline="create-cloudfoundry",status="failed"}[1h]) >= 2
        labels:
          severity: warning
        annotations:
          summary: Concourse continuous-smoke-tests failures
          description: The continuous-smoke-tests Concourse job has failed at least twice in the last hour.

      - alert: ConcourseSmoketestsFailuresCritical
        expr: increase(concourse_builds_finished{exported_job="continuous-smoke-tests",pipeline="create-cloudfoundry",status="failed"}[30m]) >= 3
        labels:
          severity: critical
          notify: pagerduty
        annotations:
          summary: Concourse continuous-smoke-tests failures
          description: The continuous-smoke-tests Concourse job has failed at least three times in the last 30 minutes.

Create separate warning and critical alerts for the smoke tests failures
Create separate warning and critical alerts for the smoke tests failures

If the tests fail at least twice in the last hour, we'll send a warning alert
email.
If the tests fail at least three times in the last hour, we'll send a critical
alert email and a pagerduty notification.
