anchor.diff:          utc
       </match>
   ----
  -
  -*Enabling fluentd*
  -
  -[shell,sh]
  -****
  -    # systemctl enable td-agent
  -    # systemctl start td-agent
  -****
  -
  -TIP: Any errors will be logged in the *_/var/log/td-agent/td-agent.log_* file.
  +====
  +
  +. Enable *fluentd*:
  ++
  +====
  +----
  +# systemctl enable td-agent
  +# systemctl start td-agent
  +----
  +====
  ++
  +[TIP]
  +====
  +Any errors are logged in the *_/var/log/td-agent/td-agent.log_* file.
  +====
   
   You should now find all the containers' logs available on the master in the
   *_/var/log/td-agent/containers.log_* file.
   
  ----
  -
  -= Using Elasticsearch
  -
  -https://www.elastic.co/products/elasticsearch[Elasticsearch] is an open source
  -distributed document database that indexes documents & provides full-text
  -search capabilities. By storing container logs in Elasticsearch, users will be
  -able to search all content & filter appropriately. This documentation shows how
  -to run https://www.elastic.co/products/kibana[Kibana].
  -
  -This option requires more configuration & more resources than the
  -<<Using a Centralized Filesystem,centralised filesystem>> option, but makes logs more useful
  -for troubleshooting & fault finding.
  -
  -Enabling aggregated logging to Elasticsearch involves:
  -
  -. <<Creating an Elasticsearch cluster>>
  -. <<Creating logging pods>>
  -. <<Creating Kibana service>>
  -
  -== Creating an Elasticsearch cluster
  -
  -Logs are stored in an Elasticsearch cluster running on OpenShift. This cluster
  -is scalable via a replication controller so you can scale the Elasticsearch
  -cluster up & down as required.
  -
  -The following is the manifest for the Elasticsearch cluster:
  +== Using Elasticsearch
  +
  +https://www.elastic.co/products/elasticsearch[*Elasticsearch*] is an open source
  +distributed document database that indexes documents and provides full-text
  +search capabilities. By storing container logs in *Elasticsearch*, users are
  +able to search all content and filter appropriately. This documentation shows
  +how to run https://www.elastic.co/products/kibana[*Kibana*].
  +
  +This option requires more configuration and more resources than the
  +link:#using-a-centralized-file-system[centralized file system option], but makes
  +logs more useful for troubleshooting and fault finding.
  +
  +Enabling aggregated logging to *Elasticsearch* involves:
  +
  +. link:#creating-an-elasticsearch-cluster[Creating an *Elasticsearch* cluster]
  +. link:#creating-logging-pods[Creating logging pods]
  +. link:#creating-the-kibana-service[Creating the *Kibana* service]
  +
  +=== Creating an Elasticsearch cluster
  +
  +Logs are stored in an *Elasticsearch* cluster running on OpenShift. This cluster
  +is scalable using a
  +link:../architecture/core_objects/kubernetes_model.html#replication-controller[replication
  +controller], so you can link:../dev_guide/deployments.html#scaling[scale] the
  +*Elasticsearch* cluster up and down as required.
  +
  +The following is the manifest for the *Elasticsearch* cluster:
  +
  +====
   
   [source,yaml]
   ----
  
positive.code: ----
  <1> The name for the master that will be used during logging.
  <2> The IP or a DNS resolvable name used to access the master.
  
  === Enabling fluentd
  
  [source,sh]
  ----
  # systemctl enable td-agent
  # systemctl start td-agent
  ----
  
  TIP: Any errors will be logged in the *_/var/log/td-agent/td-agent.log_* file.
  
  === Optional Method to Verify Working Nodes
  
  You can optionally set up the master to be the aggregator to test and verify
  that the nodes are working properly.
  
  *Installing fluentd (td-agent) on the Master*
  
  [source,sh]
  ----
  # export RPM=td-agent-2.2.0-0.x86_64.rpm
  # curl http://packages.treasuredata.com/2/redhat/7/x86_64/$RPM -o /tmp/$RPM
  # yum localinstall $RPM
  # mkdir -p /etc/td-agent/config.d
  # chown td-agent:td-agent /etc/td-agent/config.d
  ----
  
  Ensure port 24224 is open on the master's firewall to allow the nodes access.
  
  *Configuring fluentd to Aggregate Container Logs*
  
  Add the following line to the *_/etc/td-agent/td-agent.conf_* file:
  
  ----
    @include config.d/*.conf
  ----
  
  Create the *_/etc/td-agent/config.d/kubernetes.conf_* file with the following
  contents:
  
  [source,html]
  ----
      <match kubernetes.**>
          type file
  
negative.code: :experimental:
  :toc: macro
  :toc-title:
  :prewrap!:
  
  toc::[]
  
  == Overview
  
  As an OpenShift administrator, you may want to view the logs from all containers
  in one user interface. There are two options for aggregating container logs,
  depending on user requirements:
  
  . link:#using-a-centralized-file-system[Using a centralized file system]
  . link:#using-elasticsearch[Using *Elasticsearch*]
  
  [IMPORTANT]
  ====
  These solutions are a work in progress. As packaging improvements are made these
  instructions will be simplified.
  ====
  
  == Using a Centralized File System
  
  This option reads all container logs and forwards them to a central server for
  storage on the file system. This solution requires less resources and requires
  less management than the link:#using-elasticsearch[*Elasticsearch* option], but
  the logs are not indexed and searchable.
  
  [[installing-fluentd-td-agent-on-nodes]]
  === Installing fluentd (td-agent) on Nodes
  
  Perform the following steps on each node to install and configure *fluentd*
  (*td-agent*):
  
  . Run the following commands:
  +
  ====
  ----
  # export RPM=td-agent-2.2.0-0.x86_64.rpm
  # curl http://packages.treasuredata.com/2/redhat/7/x86_64/$RPM \
      -o /tmp/$RPM
  # yum localinstall $RPM
  # /opt/td-agent/embedded/bin/gem install fluent-plugin-kubernetes
  # mkdir -p /etc/td-agent/config.d
  # chown td-agent:td-agent /etc/td-agent/config.d
  ----
  ====
  
  . To allow *td-agent* access to the containers logs, create the
  *_/etc/sysconfig/td-agent_* file with the following contents:
  +
  ====
  ----
  DAEMON_ARGS=
  TD_AGENT_ARGS="/usr/sbin/td-agent --log /var/log/td-agent/td-agent.log --use-v1-config"
  ----
  ====
  
  . Add the following line to the *_/etc/td-agent/td-agent.conf_* file:
  +
  ====
  ----
    @include config.d/*.conf
  ----
  ====
  
  . Create the *_/etc/td-agent/config.d/kubernetes.conf_* file with the following
  contents:
  +
  ====
  
  [source,xml]
  ----
  