title: How to config alertmanager which installed by helm on kubernetes?
body: <p>Use <code>Helm</code> installed <code>Prometheus</code> and <code>Grafana</code> in a kubernetes cluster:</p>
  
  <pre><code>helm install stable/prometheus
  helm install stable/grafana
  </code></pre>
  
  <p>It has an <code>alertmanage</code> service.</p>
  
  <p><a href="https://i.stack.imgur.com/ePQ9w.png" rel="noreferrer"><img src="https://i.stack.imgur.com/ePQ9w.png" alt="enter image description here"></a></p>
  
  <p>But I saw a blog introduced how to setup alertmanager config with yaml files:</p>
  
  <blockquote>
    <p><a href="http://blog.wercker.com/how-to-setup-alerts-on-prometheus" rel="noreferrer">http://blog.wercker.com/how-to-setup-alerts-on-prometheus</a></p>
  </blockquote>
  
  <p>Is it possible to use the current way(installed by helm) to set some <code>alert rules</code> and config for <code>CPU</code>, <code>memory</code> and send Email without create other yaml files?</p>
  
  <p>I saw a introduction for k8s <code>configmap</code> to <code>alertmanager</code>:</p>
  
  <blockquote>
    <p><a href="https://github.com/kubernetes/charts/tree/master/stable/prometheus#configmap-files" rel="noreferrer">https://github.com/kubernetes/charts/tree/master/stable/prometheus#configmap-files</a></p>
  </blockquote>
  
  <p>But not clear how to use and how to do.</p>
  
  <hr>
  
  <h1>Edit</h1>
  
  <p>I downloaded source code of <code>stable/prometheus</code> to see what it do. From the <code>values.yaml</code> file I found:</p>
  
  <pre><code>serverFiles:
    alerts: ""
    rules: ""
  
    prometheus.yml: |-
      rule_files:
        - /etc/config/rules
        - /etc/config/alerts
      scrape_configs:
        - job_name: prometheus
          static_configs:
            - targets:
              - localhost:9090
  </code></pre>
  
  <blockquote>
    <p><a href="https://github.com/kubernetes/charts/blob/master/stable/prometheus/values.yaml#L600" rel="noreferrer">https://github.com/kubernetes/charts/blob/master/stable/prometheus/values.yaml#L600</a></p>
  </blockquote>
  
  <p>So I think should write to this config file by myself to define alert <code>rules</code> and <code>alertmanager</code> here. But don't clear about this block:</p>
  
  <pre><code>    rule_files:
        - /etc/config/rules
        - /etc/config/alerts
  </code></pre>
  
  <p>Maybe it's meaning the path in the container. But there isn't any file now. Should add here:</p>
  
  <pre><code>serverFiles:
    alert: ""
    rules: ""
  </code></pre>
  
  <h1>Edit 2</h1>
  
  <p>After set <code>alert rules</code> and <code>alertmanager</code> configuration in <code>values.yaml</code>:</p>
  
  <pre><code>## Prometheus server ConfigMap entries
  ##
  serverFiles:
    alerts: ""
    rules: |-
      #
      # CPU Alerts
      #
      ALERT HighCPU
        IF ((sum(node_cpu{mode=~"user|nice|system|irq|softirq|steal|idle|iowait"}) by (instance, job)) - ( sum(node_cpu{mode=~"idle|iowait"}) by (instance,job) )   )   /  (sum(node_cpu{mode=~"user|nice|system|irq|softirq|steal|idle|iowait"}) by (instance, job)) * 100 &gt; 95
        FOR 10m
        LABELS { service = "backend" }
        ANNOTATIONS {
          summary = "High CPU Usage",
          description = "This machine  has really high CPU usage for over 10m",
        }
  
      # TEST
      ALERT APIHighRequestLatency
        IF api_http_request_latencies_second{quantile="0.5"} &gt;1
        FOR 1m
        ANNOTATIONS {
          summary = "High request latency on {{$labels.instance }}",
          description = "{{ $labels.instance }} has amedian request latency above 1s (current value: {{ $value }}s)",
      }
  </code></pre>
  
  <p>Ran <code>helm install prometheus/</code> to install it.</p>
  
  <p>Start <code>port-forward</code> for <code>alertmanager</code> component:</p>
  
  <pre><code>export POD_NAME=$(kubectl get pods --namespace default -l "app=prometheus,component=alertmanager" -o jsonpath="{.items[0].metadata.name}")
  kubectl --namespace default port-forward $POD_NAME 9093
  </code></pre>
  
  <p>Then access browser to <code>http://127.0.0.1:9003</code>, got these messages:</p>
  
  <pre><code>Forwarding from 127.0.0.1:9093 -&gt; 9093
  Handling connection for 9093
  Handling connection for 9093
  E0122 17:41:53.229084    7159 portforward.go:331] an error occurred forwarding 9093 -&gt; 9093: error forwarding port 9093 to pod 6614ee96df545c266e5fff18023f8f7c87981f3340ee8913acf3d8da0e39e906, uid : exit status 1: 2018/01/22 08:37:54 socat[31237.140275133073152] E connect(5, AF=2 127.0.0.1:9093, 16): Connection refused
  Handling connection for 9093
  E0122 17:41:53.243511    7159 portforward.go:331] an error occurred forwarding 9093 -&gt; 9093: error forwarding port 9093 to pod 6614ee96df545c266e5fff18023f8f7c87981f3340ee8913acf3d8da0e39e906, uid : exit status 1: 2018/01/22 08:37:54 socat[31238.140565602109184] E connect(5, AF=2 127.0.0.1:9093, 16): Connection refused
  E0122 17:41:53.246011    7159 portforward.go:331] an error occurred forwarding 9093 -&gt; 9093: error forwarding port 9093 to pod 6614ee96df545c266e5fff18023f8f7c87981f3340ee8913acf3d8da0e39e906, uid : exit status 1: 2018/01/22 08:37:54 socat[31239.140184300869376] E connect(5, AF=2 127.0.0.1:9093, 16): Connection refused
  Handling connection for 9093
  Handling connection for 9093
  E0122 17:41:53.846399    7159 portforward.go:331] an error occurred forwarding 9093 -&gt; 9093: error forwarding port 9093 to pod 6614ee96df545c266e5fff18023f8f7c87981f3340ee8913acf3d8da0e39e906, uid : exit status 1: 2018/01/22 08:37:55 socat[31250.140004515874560] E connect(5, AF=2 127.0.0.1:9093, 16): Connection refused
  E0122 17:41:53.847821    7159 portforward.go:331] an error occurred forwarding 9093 -&gt; 9093: error forwarding port 9093 to pod 6614ee96df545c266e5fff18023f8f7c87981f3340ee8913acf3d8da0e39e906, uid : exit status 1: 2018/01/22 08:37:55 socat[31251.140355466835712] E connect(5, AF=2 127.0.0.1:9093, 16): Connection refused
  Handling connection for 9093
  E0122 17:41:53.858521    7159 portforward.go:331] an error occurred forwarding 9093 -&gt; 9093: error forwarding port 9093 to pod 6614ee96df545c266e5fff18023f8f7c87981f3340ee8913acf3d8da0e39e906, uid : exit status 1: 2018/01/22 08:37:55 socat[31252.140268300003072] E connect(5, AF=2 127.0.0.1:9093, 16): Connection refused
  </code></pre>
  
  <p>Why?</p>
  
  <p>When I check <code>kubectl describe po illocutionary-heron-prometheus-alertmanager-587d747b9c-qwmm6</code>, got:</p>
  
  <pre><code>Name:           illocutionary-heron-prometheus-alertmanager-587d747b9c-qwmm6
  Namespace:      default
  Node:           minikube/192.168.99.100
  Start Time:     Mon, 22 Jan 2018 17:33:54 +0900
  Labels:         app=prometheus
                  component=alertmanager
                  pod-template-hash=1438303657
                  release=illocutionary-heron
  Annotations:    kubernetes.io/created-by={"kind":"SerializedReference","apiVersion":"v1","reference":{"kind":"ReplicaSet","namespace":"default","name":"illocutionary-heron-prometheus-alertmanager-587d747b9c","uid":"f...
  Status:         Running
  IP:             172.17.0.10
  Created By:     ReplicaSet/illocutionary-heron-prometheus-alertmanager-587d747b9c
  Controlled By:  ReplicaSet/illocutionary-heron-prometheus-alertmanager-587d747b9c
  Containers:
    prometheus-alertmanager:
      Container ID:  docker://0808a3ecdf1fa94b36a1bf4b8f0d9d2933bc38afa8b25e09d0d86f036ac3165b
      Image:         prom/alertmanager:v0.9.1
      Image ID:      docker-pullable://prom/alertmanager@sha256:ed926b227327eecfa61a9703702c9b16fc7fe95b69e22baa656d93cfbe098320
      Port:          9093/TCP
      Args:
        --config.file=/etc/config/alertmanager.yml
        --storage.path=/data
      State:          Waiting
        Reason:       CrashLoopBackOff
      Last State:     Terminated
        Reason:       Error
        Exit Code:    1
        Started:      Mon, 22 Jan 2018 17:55:24 +0900
        Finished:     Mon, 22 Jan 2018 17:55:24 +0900
      Ready:          False
      Restart Count:  9
      Readiness:      http-get http://:9093/%23/status delay=30s timeout=30s period=10s #success=1 #failure=3
      Environment:    &lt;none&gt;
      Mounts:
        /data from storage-volume (rw)
        /etc/config from config-volume (rw)
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-h5b8l (ro)
    prometheus-alertmanager-configmap-reload:
      Container ID:  docker://b4a349bf7be4ea78abe6899ad0173147f0d3f6ff1005bc513b2c0ac726385f0b
      Image:         jimmidyson/configmap-reload:v0.1
      Image ID:      docker-pullable://jimmidyson/configmap-reload@sha256:2d40c2eaa6f435b2511d0cfc5f6c0a681eeb2eaa455a5d5ac25f88ce5139986e
      Port:          &lt;none&gt;
      Args:
        --volume-dir=/etc/config
        --webhook-url=http://localhost:9093/-/reload
      State:          Running
        Started:      Mon, 22 Jan 2018 17:33:56 +0900
      Ready:          True
      Restart Count:  0
      Environment:    &lt;none&gt;
      Mounts:
        /etc/config from config-volume (ro)
        /var/run/secrets/kubernetes.io/serviceaccount from default-token-h5b8l (ro)
  Conditions:
    Type           Status
    Initialized    True 
    Ready          False 
    PodScheduled   True 
  Volumes:
    config-volume:
      Type:      ConfigMap (a volume populated by a ConfigMap)
      Name:      illocutionary-heron-prometheus-alertmanager
      Optional:  false
    storage-volume:
      Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
      ClaimName:  illocutionary-heron-prometheus-alertmanager
      ReadOnly:   false
    default-token-h5b8l:
      Type:        Secret (a volume populated by a Secret)
      SecretName:  default-token-h5b8l
      Optional:    false
  QoS Class:       BestEffort
  Node-Selectors:  &lt;none&gt;
  Tolerations:     &lt;none&gt;
  Events:
    Type     Reason                 Age                 From               Message
    ----     ------                 ----                ----               -------
    Warning  FailedScheduling       29m (x2 over 29m)   default-scheduler  PersistentVolumeClaim is not bound: "illocutionary-heron-prometheus-alertmanager"
    Normal   Scheduled              29m                 default-scheduler  Successfully assigned illocutionary-heron-prometheus-alertmanager-587d747b9c-qwmm6 to minikube
    Normal   SuccessfulMountVolume  29m                 kubelet, minikube  MountVolume.SetUp succeeded for volume "config-volume"
    Normal   SuccessfulMountVolume  29m                 kubelet, minikube  MountVolume.SetUp succeeded for volume "pvc-fa84b197-ff4e-11e7-a584-0800270fb7fc"
    Normal   SuccessfulMountVolume  29m                 kubelet, minikube  MountVolume.SetUp succeeded for volume "default-token-h5b8l"
    Normal   Started                29m                 kubelet, minikube  Started container
    Normal   Created                29m                 kubelet, minikube  Created container
    Normal   Pulled                 29m                 kubelet, minikube  Container image "jimmidyson/configmap-reload:v0.1" already present on machine
    Normal   Started                29m (x3 over 29m)   kubelet, minikube  Started container
    Normal   Created                29m (x4 over 29m)   kubelet, minikube  Created container
    Normal   Pulled                 29m (x4 over 29m)   kubelet, minikube  Container image "prom/alertmanager:v0.9.1" already present on machine
    Warning  BackOff                9m (x91 over 29m)   kubelet, minikube  Back-off restarting failed container
    Warning  FailedSync             4m (x113 over 29m)  kubelet, minikube  Error syncing pod
  </code></pre>
  
  <h1>Edit 3</h1>
  
  <p><code>alertmanager</code> config in <code>values.yaml</code> file:</p>
  
  <pre><code>## alertmanager ConfigMap entries
  ##
  alertmanagerFiles:
    alertmanager.yml: |-
      global:
        resolve_timeout: 5m
        smtp_smarthost: smtp.gmail.com:587
        smtp_from: sender@gmail.com
        smtp_auth_username: sender@gmail.com
        smtp_auth_password: sender_password
  
      receivers:
        - name: default-receiver
        email_configs:
        - to: target_email@gmail.com
  
      route:
        group_wait: 10s
        group_interval: 5m
        receiver: default-receiver
        repeat_interval: 3h
  </code></pre>
  
  <p>Not work. Got errors above.</p>
  
  <pre><code>alertmanagerFiles:
    alertmanager.yml: |-
      global:
        # slack_api_url: ''
  
      receivers:
        - name: default-receiver
          # slack_configs:
          #  - channel: '@you'
          #    send_resolved: true
  
      route:
        group_wait: 10s
        group_interval: 5m
        receiver: default-receiver
        repeat_interval
  </code></pre>
  
  <p>Works without any error.</p>
  
  <p>So, the problem was the <code>email_configs</code> config method.</p>