anchor.diff:  
   image::monitoring-pods-after.png[]
   
  -== Prometheus
  +== Prometheus, Node exporter and Grafana
   
   http://prometheus.io/[Prometheus] is an open-source systems monitoring and alerting toolkit. Prometheus collects metrics from monitored targets by scraping metrics from HTTP endpoints on these targets.
   
   Different targets to scrape are defined in the Prometheus configuration file. Targets may be statically configured via the `static_configs` parameter in the configuration fle or dynamically discovered using one of the supported service-discovery mechanisms (Consul, DNS, Etcd, etc.).
   
  -First in the file templates/prometheus-configmap.yaml modify the etcd section by replacing the IP address with the IP of your etcd server:
  -
  -    - job_name: 'etcd'
  -    target_groups:
  -    - targets:
  -    - 172.17.4.51:4001
  -
  -To determine the IP address that your etcd cluster is listening on (if deployed into Kubernetes cluster), execute this command:
  +Prometheus gathers these metrics from https://github.com/kubernetes/kube-state-metrics[kube-state-metrics]. A wide variety of metrics are available and documented at https://github.com/kubernetes/kube-state-metrics/tree/master/Documentation.
  +
  +=== Installation
  +
  +Prometheus configuration file is defined as a ConfigMap in the file `prometheus/templates/prometheus-configmap.yaml`. 
  +
  +We need to provide the location of etcd server in our cluster in this configuration file. In our case, etcd is running inside the Kubernetes cluster. Find IP address of the etcd using this command:
   
        kubectl get pods --namespace=kube-system
   
  -etcd clusters deployed with the most recent version of kops use port 4001, if you have a newer version of etcd it will be listening on port 2379
  -
  -=== Installation
  +It shows an output as:
  +
  +```
  +NAME                                                                  READY     STATUS    RESTARTS   AGE
  +etcd-server-events-ip-172-20-48-121.us-west-2.compute.internal        1/1       Running   0          10h
  +etcd-server-ip-172-20-48-121.us-west-2.compute.internal               1/1       Running   0          9h
  +```
  +
  +Other pods are shown as well, but we are only showing the etcd pods.
  +
  +Note down the name of the pod that starts with `etcd-server-ip`.
  +
  +Get more details about this pod:
  +
  +  kubectl describe pod/etcd-server-ip-172-20-48-121.us-west-2.compute.internal --namespace=kube-system 
  +
  +The output is like:
  +
  +```
  +Name:         etcd-server-ip-172-20-48-121.us-west-2.compute.internal
  +Namespace:    kube-system
  +Node:         ip-172-20-48-121.us-west-2.compute.internal/172.20.48.121
  +Start Time:   Wed, 18 Oct 2017 03:09:53 +0200
  +Labels:       k8s-app=etcd-server
  +Annotations:  kubernetes.io/config.hash=29e2e50ee6b1cd5c2f410da0f8c0f295
  +              kubernetes.io/config.mirror=29e2e50ee6b1cd5c2f410da0f8c0f295
  +              kubernetes.io/config.seen=2017-10-18T01:09:48.793098897Z
  +              kubernetes.io/config.source=file
  +Status:       Running
  +IP:           172.20.48.121
  +
  +. . .
  +
  +QoS Class:       Burstable
  +Node-Selectors:  <none>
  +Tolerations:     :NoExecute
  +Events:          <none>
  +```
  +
  +Note down the IP address from this output.
  +
  +Update the file `prometheus/templates/prometheus-configmap.yaml`, and replace `<IP>` with the IP address of the `etcd` server in your cluster. The updated fragment will look like as shown:
  +
  +```
  +- job_name: 'etcd'
  +  target_groups:
  +  - targets:
  +    - 172.20.48.121:4001
  +```
  +
  +TODO: Is this required? Check config map again. etcd is already running in the cluster. etcd clusters deployed with the most recent version of kops use port 4001, if you have a newer version of etcd it will be listening on port 2379.
  +
  +Node exporter is defined as a DaemonSet, and so there is a single instance running on each node of the cluster.
   
   Once you have saved the etcd information into the file you can deploy the config map:
   
  -    kubectl create -f templates/prometheus-configmap.yaml
  +    kubectl create -f prometheus/templates/prometheus-configmap.yaml
   
   Next deploy Prometheus into your cluster:
   
  -    kubectl create -f templates/prometheus-deployment.yaml
  -
  -You should now be able to open a browser to the public IP address of your master node at port 30900 to display the Prometheus dashboard.
  +    kubectl create -f prometheus/templates/prometheus-deployment.yaml
   
   Finally we will deploy the node exporter DaemonSet which will read system level metrics from each node and export them to Prometheus:
   
  -    kubectl create -f templates/node-exporter.yaml
  -
  +    kubectl create -f prometheus/templates/node-exporter.yaml
  +
  +Finally, deploy the Grafana dashboard:
  +
  +    kubectl create -f prometheus/templates/grafana.yml
  +
  +=== Prometheus Dashboard
  +
  +Prometheus is now scraping metrics from the etcd server, the Kubernetes API server and the node exporter. Metrics exported by different sources are listed below:
  +
  +- etcd: https://coreos.com/etcd/docs/latest/metrics
  +- Kubernetes API server: https://github.com/kubernetes/kube-state-metrics
  +- Node exporter: https://github.com/prometheus/node_exporter
  +
  +Let's take a look at how we can see these metrics in the Prometheus dashboard.
  +
  +. Find the master node in the EC2 console:
  ++
  +image::prometheus-ec2-nodes.png[]
  ++
  +. Copy public IP address of the master node.
  +. Edit the security group and add a rule to allow inbound traffic on port 30900 as shown:
  ++
  +image::prometheus-ec2-security-group.png[]
  ++
  +. Prometheus dashboard is now available at http://<master-ip>:30900. In our case, this will be http://ec2-34-215-44-44.us-west-2.compute.amazonaws.com:30900 and looks like:
  ++
  +image::prometheus-dashboard-initial.png[]
  ++
  +. A wide set of metrics are available and can be seen in the dashboard:
  ++
  +image::prometheus-dashboard-metrics.png[]
  +
  +=== Grafana dashboard
  +
  +Grafana dashboard, accessible at http://localhost:8001/api/v1/namespaces/kube-system/services/monitoring-grafana/proxy/?orgId=1, now shows the information about the cluster. 
  +
  +NOTE: Grafana dashboard will not be available if Kubernetes proxy is not running. If not running, then it can be started with the command `kubectl proxy`.
  +
  +
  
positive.code: 
  This will change after Heapster, InfluxDB and Grafana are installed.
  
  == Heapster
  
  https://github.com/kubernetes/heapster[Heapster] is a metrics aggregator and processor. It is installed as a cluster-wide pod. It gathers monitoring and events data for all containers on each node by talking to the Kubelet. Kubelet itself fetches this data from https://github.com/google/cadvisor[cAdvisor]. This data is persisted in a time series database https://github.com/influxdata/influxdb[InfluxDB] for storage. The data is then visualized using a using http://grafana.org/[Grafana] dashboard or can be viewed in Kubernetes Dashboard.
  
  
negative.code: 
  image::monitoring-grafana-dashboards.png[]
  
  The "`Cluster`" dashboard shows all worker nodes, and their CPU and memory metrics. Type in a node name to see its collected metrics during a chosen period of time.
  
  The cluster dashboard looks like:
  
  image::monitoring-grafana-dashboards-cluster.png[]
  
  The "`Pods`"" dashboard allows you to see resource utilization of every pod in the cluster. As with nodes, you can select the pod by typing its name in the top filter box.
  
  image::monitoring-grafana-dashboards-pods.png[]
  
  