title: How to implement logging with Fluentd/Elasticsearch/Ikebana on Kubernetes 1.1?
body: <p>I am trying to implement logging on my AWS/CoreOS/Kubernetes (1.1) setup, which I've set up without kube-up. So far I've installed fluentd as a static <a href="https://github.com/kubernetes/kubernetes/blob/release-1.1/cluster/saltbase/salt/fluentd-es/fluentd-es.yaml" rel="nofollow">pod</a> on all nodes and the <a href="https://github.com/kubernetes/kubernetes/tree/release-1.1/cluster/addons/fluentd-elasticsearch" rel="nofollow">Fluentd-Elasticsearch addon</a> replication controllers and services. However, it doesn't yet work. Specifically, Kibana crashes like this:</p>
  
  <pre><code>ELASTICSEARCH_URL=http://elasticsearch-logging.kube-system:9200
  {"@timestamp":"2016-03-14T22:54:04.906Z","level":"error","message":"Service Unavailable","node_env":"production","error":{"message":"Service Unavailable","name":"Error","stack":"Error: Service Unavailable\n  at respond (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:235:15)\n  at checkRespForFailure (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:203:7)\n  at HttpConnector.&lt;anonymous&gt; (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/connectors/http.js:156:7)\n  at IncomingMessage.bound (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)\n  at IncomingMessage.emit (events.js:117:20)\n  at _stream_readable.js:944:16\n  at process._tickCallback (node.js:442:13)\n"}}
  {"@timestamp":"2016-03-14T22:54:04.908Z","level":"fatal","message":"Service Unavailable","node_env":"production","error":{"message":"Service Unavailable","name":"Error","stack":"Error: Service Unavailable\n  at respond (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:235:15)\n  at checkRespForFailure (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/transport.js:203:7)\n  at HttpConnector.&lt;anonymous&gt; (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/src/lib/connectors/http.js:156:7)\n  at IncomingMessage.bound (/kibana-4.0.2-linux-x64/src/node_modules/elasticsearch/node_modules/lodash-node/modern/internals/baseBind.js:56:17)\n  at IncomingMessage.emit (events.js:117:20)\n  at _stream_readable.js:944:16\n  at process._tickCallback (node.js:442:13)\n"}}
  </code></pre>
  
  <p>What should I do?</p>
  
  <p>FWIW, Elasticsearch is connectable at <a href="http://elasticsearch-logging.kube-system:9200/" rel="nofollow">http://elasticsearch-logging.kube-system:9200/</a>, although it does return status 503. That could be the issue for all I know.</p>
  
  <pre><code># curl http://elasticsearch-logging.kube-system:9200/
  {
    "status" : 503,
    "name" : "Puppet Master",
    "cluster_name" : "kubernetes-logging",
    "version" : {
      "number" : "1.5.2",
      "build_hash" : "62ff9868b4c8a0c45860bebb259e21980778ab1c",
      "build_timestamp" : "2015-04-27T09:21:06Z",
      "build_snapshot" : false,
      "lucene_version" : "4.10.4"
    },
    "tagline" : "You Know, for Search"
  }
  </code></pre>
  
  <h2>fluentd-es.yaml</h2>
  
  <pre><code>apiVersion: v1
  kind: Pod
  metadata:
    name: fluentd-elasticsearch
    namespace: kube-system
  spec:
    containers:
    - name: fluentd-elasticsearch
      image: gcr.io/google_containers/fluentd-elasticsearch:1.13
      resources:
        limits:
          cpu: 100m
      args:
      - -q
      volumeMounts:
      - name: varlog
        mountPath: /var/log
      - name: varlibdockercontainers
        mountPath: /var/lib/docker/containers
        readOnly: true
    terminationGracePeriodSeconds: 30
    volumes:
    - name: varlog
      hostPath:
        path: /var/log
    - name: varlibdockercontainers
      hostPath:
        path: /var/lib/docker/containers
  </code></pre>
  
  <h2>es-controller.yaml</h2>
  
  <pre><code>apiVersion: v1
  kind: ReplicationController
  metadata:
    name: elasticsearch-logging-v1
    namespace: kube-system
    labels:
      k8s-app: elasticsearch-logging
      version: v1
      kubernetes.io/cluster-service: "true"
  spec:
    replicas: 2
    selector:
      k8s-app: elasticsearch-logging
      version: v1
    template:
      metadata:
        labels:
          k8s-app: elasticsearch-logging
          version: v1
          kubernetes.io/cluster-service: "true"
      spec:
        containers:
        - image: gcr.io/google_containers/elasticsearch:1.7
          name: elasticsearch-logging         
          resources:
            limits:
              cpu: 100m
          ports:
          - containerPort: 9200
            name: db
            protocol: TCP
          - containerPort: 9300
            name: transport
            protocol: TCP
          volumeMounts:
          - name: es-persistent-storage
            mountPath: /data
        volumes:
        - name: es-persistent-storage
          emptyDir: {}
  </code></pre>
  
  <h2>es-service.yaml</h2>
  
  <pre><code>apiVersion: v1
  kind: Service
  metadata:
    name: elasticsearch-logging
    namespace: kube-system
    labels:
      k8s-app: elasticsearch-logging
      kubernetes.io/cluster-service: "true"
      kubernetes.io/name: "Elasticsearch"
  spec:
    ports:
    - port: 9200
      protocol: TCP
      targetPort: db
    selector:
      k8s-app: elasticsearch-logging
  </code></pre>
  
  <h2>kibana-controller.yaml</h2>
  
  <pre><code>apiVersion: v1
  kind: ReplicationController
  metadata:
    name: kibana-logging-v1
    namespace: kube-system
    labels:
      k8s-app: kibana-logging
      version: v1
      kubernetes.io/cluster-service: "true"
  spec:
    replicas: 1
    selector:
      k8s-app: kibana-logging
      version: v1
    template:
      metadata:
        labels:
          k8s-app: kibana-logging
          version: v1
          kubernetes.io/cluster-service: "true"
      spec:
        containers:
        - name: kibana-logging
          image: gcr.io/google_containers/kibana:1.3
          resources:
            limits:
              cpu: 100m
          env:
            - name: "ELASTICSEARCH_URL"
              value: "http://elasticsearch-logging.kube-system:9200"
          ports:
          - containerPort: 5601
            name: ui
            protocol: TCP
  </code></pre>
  
  <h2>kibana-service.yaml</h2>
  
  <pre><code>apiVersion: v1
  kind: Service
  metadata:
    name: kibana-logging
    namespace: kube-system
    labels:
      k8s-app: kibana-logging
      kubernetes.io/cluster-service: "true"
      kubernetes.io/name: "Kibana"
  spec:
    ports:
    - port: 5601
      protocol: TCP
      targetPort: ui
    selector:
      k8s-app: kibana-logging
  </code></pre>