# Source: blackbox-exporter
---

- type: replace
  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties/prometheus/custom_rules?/-
  value:
    name: DNSResolutionWorking
    rules:
    - &alert
      alert: DNSResolutionWorking_Warning
      expr: (count_over_time(dns_resolution_probe_success[5m]) - sum_over_time(dns_resolution_probe_success[5m])) / count_over_time(dns_resolution_probe_success[5m]) > 0.25
      for: 10m
      labels:
        severity: warning
      annotations:
        summary: "DNS Resolution is failing on {{ $labels.address }}"
        description: "If this persisists, consider contacting AWS Support."

    - <<: *alert
      alert: DNSResolutionWorking_Critical
      expr: (count_over_time(dns_resolution_probe_success[5m]) - sum_over_time(dns_resolution_probe_success[5m])) / count_over_time(dns_resolution_probe_success[5m]) > 0.75
      labels:
        severity: critical

# Source: blackbox-exporter
---

- type: replace
  path: /instance_groups/name=prometheus2/jobs/name=prometheus2/properties/prometheus/custom_rules?/-
  value:
    name: DNSResolutionWorking
    rules:
    - alert: DNSResolutionWorking
      expr: sum_over_time(dns_resolution_probe_success[15m]) / count_over_time(dns_resolution_probe_success[15m]) < 0.9
      labels:
        severity: warning
      annotations:
        summary: "DNS Resolution is failing on {{ $labels.address }}"
        description: "If this persisists, consider contacting AWS Support."

Improve DNS resolution working alert.
Improve DNS resolution working alert.

During the recent issues in staging where DNS resolution wasn't working
on one of the API VMs, this alert didn't fire quickly enough / at all.

This updates it alert to fire if more than 10% of the probes failes in
the last 15mins.

As part of this I've also removed the separate Warning and Critical
alert as they haven't proved helpful.
