# Source: firehose-exporter
---

- type: replace
  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties/prometheus/custom_rules?/-
  value:
    name: GorouterLatency
    rules:
      - record: "cc:firehose_value_metric_gorouter_latency:avg10m"
        expr: avg_over_time(firehose_value_metric_gorouter_latency[10m])

      - &alert
        alert: GorouterLatency_Warning
        expr: cc:firehose_value_metric_gorouter_latency:avg10m > 750
        labels:
          severity: warning
        annotations:
          summary: Gorouter latency
          description: "Gorouter latency is too high ({{ $value | printf \"%.0f\" }}ms) on {{ $labels.bosh_job_ip }}"
          url: https://team-manual.cloud.service.gov.uk/incident_management/responding_to_alerts/#gorouter-high-latency-alerts"

      - <<: *alert
        alert: GorouterLatency_Critical
        expr: cc:firehose_value_metric_gorouter_latency:avg10m > 1500
        labels:
          severity: critical

# Source: firehose-exporter
---

- type: replace
  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties/prometheus/custom_rules?/-
  value:
    name: GorouterLatency
    rules:
      - &alert
        alert: GorouterLatency_Warning
        expr: min(avg_over_time(firehose_value_metric_gorouter_latency[5m])) > 750
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: Gorouter latency
          description: "Gorouter latency is too high ({{ $value | printf \"%.0f\" }}ms) on {{ $labels.bosh_job_ip }}"
          url: https://team-manual.cloud.service.gov.uk/incident_management/responding_to_alerts/#gorouter-high-latency-alerts"

      - <<: *alert
        alert: GorouterLatency_Critical
        expr: min(avg_over_time(firehose_value_metric_gorouter_latency[5m])) > 1500
        labels:
          severity: critical

Make the GorouterLatency* alerts less noisy.
Make the GorouterLatency* alerts less noisy.

These alerts are not really useful as any long running requests can skew the
results easily. In the future we have to try to get a more useful metrics, e.g.
avg response time for only 95% of the requests and base this alert on that.

Until then we'll look at the minimum latency and we'll only alert if all
routers report a high average latency for at least 15 minutes.
