<!doctype html>
<html lang="en">

	<head>
		<meta charset="utf-8">

		<title>ELK-Stack bei Frontline</title>

		<meta name="description" content="Search Technology Meetup Hamburg">
		<meta name="author" content="Jens Fischer, Timo Zingel">

		<meta name="apple-mobile-web-app-capable" content="yes" />
		<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />

		<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, minimal-ui">

		<link rel="stylesheet" href="css/reveal.css">
		<link rel="stylesheet" href="css/theme/white.css" id="theme">

		<!-- Code syntax highlighting -->
		<link rel="stylesheet" href="lib/css/zenburn.css">

		<!-- inline css sux -->
		<style type="text/css">
			.reveal section img {
				border: 0;
				box-shadow: 0 0 0;
			}
		</style>

		<!-- Printing and PDF exports -->
		<script>
			var link = document.createElement( 'link' );
			link.rel = 'stylesheet';
			link.type = 'text/css';
			link.href = window.location.search.match( /print-pdf/gi ) ? 'css/print/pdf.css' : 'css/print/paper.css';
			document.getElementsByTagName( 'head' )[0].appendChild( link );
		</script>

		<!--[if lt IE 9]>
		<script src="lib/js/html5shiv.js"></script>
		<![endif]-->
	</head>

	<body>

		<div class="reveal">

			<!-- Any section element inside of this container is displayed as a slide -->
			<div class="slides">
				<section data-markdown>
					<script type="text/template">
						# ELK-Stack bei Frontline
						### Erfahrungen aus zwei Jahren produktivem Einsatz
						
						Timo Zingel / [@pyspam](http://twitter.com/pyspam)

						Jens Fischer / [@jensfischerhh](http://twitter.com/jensfischerhh)
					</script>
				</section>

				<section data-markdown>
					<script type="text/template">
						<img width="50%" src="frontline.svg" />

						* Onlineshop für Streetfashion
							* 1986 als Musik-Mailorder gegründet
							* 1996 Streetfashion: *www.ziehdichan.de*
							* 2004 *www.frontlineshop.com*
						* ca. 80 Mitarbeiter
						* Scrum Team mit 6 Entwicklern

						Note:
							test
					</script>
				</section>

				<section>
					<h2>Motivation</h2>

					<img src="motivation.svg" />
					<br>

					<div class="fragment">Ops @Novomind: Shell Zugriff</div>
					<div class="fragment">Devs @Frontline: <strong>keinen</strong> Shell Zugriff</div>

					<aside class="notes" data-markdown>
						test
					</aside>
				</section>
				<section data-markdown>
					<script type="text/template">
						<h2>Bug Fixing</h2>
						<h3>aber bitte mit Logs!</h3>
						<ul>
							<li class="fragment">Shell Zugriff?</li>
							<li class="fragment">ad hoc anfordern?</li>
							<li class="fragment">cron rsync?</li>
						</ul>

						<img class="fragment" src="meh.png" height="300"/>
					</script>

					<aside class="notes" data-markdown>
					</aside>
				</section>

				<section>
					<h2>ELK</h2>
					<ul class="fragment">
						<li><strong>E</strong>lasticsearch</li>
						<li><strong>L</strong>ogstash</li>
						<li><strong>K</strong>ibana</li>
					</ul>
					<p class="fragment fade-in">
						<img src="ELK-without-broker2.svg" width="60%">
					</p>
					<aside class="notes" data-markdown>
						* einfacher ELK Stack
					</aside>
				</section>

				<section>
					<h2>Erste Schritte</h2>

					<ul>
						<li class="fragment">Q1 2013
							<ul>
								<li>Elasticsearch 0.90.x</li>
								<li>Logstash 1.1.x</li>
								<li>Kibana 2.x</li>
							</ul>
						</li>
						<li class="fragment">Analyse ERP Latenz
							<ul>
								<li>Logfile Historie</li>
								<li>CSV Export</li>
							</ul>
						</li>
					</ul>

					<aside class="notes">
						Current versions:
						<ul>
							<li>Elasticsearch 1.4.4</li>
							<li>Logstash 1.4.2</li>
							<li>Kibana 4.0.1</li>
						</ul> 
					</aside>
				</section>

				<section>
					<h2>Schnelle Ergebnisse</h2>

					<ul>
						<li class="fragment">Wenig Aufwand</li>
						<li class="fragment">tolle Visualisierung</li>
						<li class="fragment">Naive Herangehensweise</li>
					</ul>

					<aside class="notes">
						Probleme:
						<ul>
							<li>komplexer Logstash Filter</li>
							<li>Full Re-Index zu jeder Stunde</li>
						</ul>
            <p>next: <strong>Timo</strong></p>
					</aside>
				</section>

				<section>
					<h2>Logstash</h2>
					<img src="logstash-logo.png" style="height: 20vh" />
					<br>
					<img class="fragment" src="Logstash.svg" style="flex:1;" />
					<aside class="notes" data-markdown>
						- logstash pipe
						- Log-Event
					</aside>
				</section>
				<section>
					<h2>Logstash plugins</h2>
					<div style="display: flex;">
						<div class="fragment" style="flex:1;">
							<h3 style="">input</h3>
							<ul >
								<li>file</li>
								<li>redis</li>
								<li>stdin</li>
								<li>syslog</li>
								<li>heroku</li>
								<li>...</li>
							</ul>
						</div>
						<div class="fragment" style="flex:1;">
							<h3 style="">filter</h3>
							<ul >
								<li>grok</li>
								<li>multiline</li>
								<li>geoip</li>
								<li>useragent</li>
								<li>csv</li>
								<li>...</li>
							</ul>
						</div>
						<div class="fragment" style="flex:1;">
							<h3 style="">output</h3>
							<ul >
								<li>elasticsearch</li>
								<li>redis</li>
								<li>statsd</li>
								<li>graphite</li>
								<li>irc</li>
								<li>...</li>
							</ul>
						</div>
					</div>
					<br>
					<div class="fragment">
						<a href="http://logstash.net/docs/1.4.2/">logstash plugin documentation</a> <small>(logstash.net/docs/1.4.2)</small>
						<br>
						<a href="https://github.com/logstash-plugins">community plugins</a> <small>(github.com/logstash-plugins)</small>
					</div>
					<aside class="notes" data-markdown>
						- etliche Plugins für jeden Anwendungsfall
					</aside>
				</section>

					
				<section>
					<h3>logstash config</h3>
					<pre><code data-trim>
# simple.conf
input { 
    stdin {
    } 
}
output { 
    stdout { 
        codec => rubydebug 
    } 
}
					</code></pre>
					<pre class="fragment"><code data-trim>
echo "Hello Logstash" | ./bin/logstash -f simple.conf
					</code></pre>
					<pre class="fragment"><code data-trim>
{
       "message" => "Hello Logstash",
      "@version" => "1",
    "@timestamp" => "2015-03-15T10:00:42.149Z",
          "host" => "jfischer-mac"
}
					</code></pre>
				</section>

				<section>
					<h2>Live Demo:</h2>

					<h3>Hello Logstash</h3>

					<aside class="notes">
echo "Hello Logstash" | ./bin/logstash -e 'input { stdin { } } output { stdout { codec => rubydebug } }
					</aside>
				</section> 

			<section data-markdown>
				<script type="text/template">
					## Live Logs
						
					* viele Log-Dateien
					* unstrukturiert

					 Note:
					  - nächste Schritte Live Logs
					  - in Echtzeit?
					  - Redis Broker
				</script>
			</section> 
				<section>
				<h3>ELK mit broker</h3>
				<img src="ELK-with-broker.svg" width="75%">
			</section>

			<section data-markdown>
				<script type="text/template">
					## Filter

					* file-input liefert ein Event pro Zeile
					* Log-Event mit Regex parsen
						* grok
						* multiline

					* strukturiert in ES speichern
				</script>
				
				<aside class="notes" data-markdown>
          <p>next: <strong>Jens</strong></p>
				</aside>
			</section>
			<section data-markdown>
				<script type="text/template">
					## apache
					
					```
					127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] "GET /xampp/status.php HTTP/1.1" 200 3891 "http://cadenza/xampp/navi.php" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0"
					```
					```
					input { stdin { } }

					filter {
						grok {
							match => { "message" => "%{COMBINEDAPACHELOG}" }
						}
						date {
							match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
						}
					}

					output {
						elasticsearch { host => localhost }
						stdout { codec => rubydebug }
					}
					```
				</script>
				<aside class="notes" data-markdown>
					one line -> one Log-Event
					maintained patterns
				</aside>
			</section>
			<section data-markdown>
				<script type="text/template">
					```
					{
						"message" => "127.0.0.1 - - [11/Dec/2013:00:01:45 -0800] \"GET /xampp/status.php HTTP/1.1\" 200 3891 \"http://cadenza/xampp/navi.php\" \"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\"",
						"@timestamp" => "2013-12-11T08:01:45.000Z",
						"@version" => "1",
						"host" => "cadenza",
						"clientip" => "127.0.0.1",
						"ident" => "-",
						"auth" => "-",
						"timestamp" => "11/Dec/2013:00:01:45 -0800",
						"verb" => "GET",
						"request" => "/xampp/status.php",
						"httpversion" => "1.1",
						"response" => "200",
						"bytes" => "3891",
						"referrer" => "\"http://cadenza/xampp/navi.php\"",
						"agent" => "\"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.9; rv:25.0) Gecko/20100101 Firefox/25.0\""
					}
					```
					 [patterns](https://github.com/elastic/logstash/tree/v1.4.2/patterns) <small>(github.com/elastic/logstash/tree/v1.4.2/patterns)</small>
				</script>

				<aside class="notes" data-markdown>
				LIVE DEMO
					* wir kennen App Log format
					* -> parsen
					* aber was ist hiermit? -> next slide
				</aside>
			</section>

			<section>
					<h2>Live Demo:</h2>

					<h3>NASA <a href="http://localhost:5600/index.html#/dashboard/elasticsearch/NASA" target="_blank">Access Logs</a></h3>
				
				<aside class="notes">
					Juli 1995
				</aside>
			</section>

			<section>
				<img src="kibana-apache.png" />
			</section> 


			<section data-markdown>
				<!-- https://gist.github.com/edalquist/5467723#file-gistfile1-txt -->
				<script type="text/template">
					## STACKTRACE
					
					```
09:26:36.538 [catalina-exec-423] [#:dDdGKiLmEKeOwBJgTxN2HNgEckB] [c4178934f02c4102a4239ab5748cf0da1217cc01.live1@85.31.11.202] [540443758/guest@579852812] ERROR  com.frontline.shop.web.taglib.IncludeTag - Could not render jsp /WEB-INF/jsp/frontline/pages/checkout/checkout_shipper.jsp
org.apache.jasper.JasperException: An exception occurred processing JSP page /WEB-INF/jsp/frontline/pages/checkout/checkout_shipper.jsp at line 9

6:   <h2 class="headline"><i:message key="checkout.step2.heading" /></h2>
7:   <div class="shipping"></div>
8:   <div>
9:     <p><i:out value="${model.currentShipper.displayName}, ${model.currentShipper.deliveryTime}" /></p>
10:     <i:url var="shippingFormUrl" of="${urlObject}" destination="ShippingMethod" />
11:     <a href="${shippingFormUrl}" class="button light"><i:message key="checkout.edit" /></a>
12:   </div>


Stacktrace:
        at org.apache.jasper.servlet.JspServletWrapper.handleJspException(JspServletWrapper.java:521) ~[jasper.jar:6.0.35]
        at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:430) ~[jasper.jar:6.0.35]
        at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:313) ~[jasper.jar:6.0.35]
        at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:260) ~[jasper.jar:6.0.35]
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) ~[servlet-api.jar:na]
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) ~[catalina.jar:6.0.35]
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) ~[catalina.jar:6.0.35]
        at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:646) ~[catalina.jar:6.0.35]
        at org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:551) ~[catalina.jar:6.0.35]
        at org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:488) ~[catalina.jar:6.0.35]
        at org.apache.jasper.runtime.JspRuntimeLibrary.include(JspRuntimeLibrary.java:968) ~[jasper.jar:6.0.35]
        at org.apache.jasper.runtime.PageContextImpl.include(PageContextImpl.java:624) ~[jasper.jar:6.0.35]
        at com.frontline.shop.web.taglib.IncludeTag.render(IncludeTag.java:136) ~[web-17.0.0.jar:na]
        at com.frontline.shop.web.taglib.IncludeTag.endTag(IncludeTag.java:95) ~[web-17.0.0.jar:na]
        at com.frontline.shop.web.taglib.AbstractReusableTag.doEndTag(AbstractReusableTag.java:33) ~[web-17.0.0.jar:na]
        at org.apache.jsp.WEB_002dINF.jsp.frontline.pages.page_005fcheckout_jsp._jspx_meth_i_005finclude_005f5(page_005fcheckout_jsp.java:580) ~[na:na]
        at org.apache.jsp.WEB_002dINF.jsp.frontline.pages.page_005fcheckout_jsp._jspx_meth_c_005fwhen_005f3(page_005fcheckout_jsp.java:545) ~[na:na]
        at org.apache.jsp.WEB_002dINF.jsp.frontline.pages.page_005fcheckout_jsp._jspx_meth_c_005fchoose_005f1(page_005fcheckout_jsp.java:468) ~[na:na]
        at org.apache.jsp.WEB_002dINF.jsp.frontline.pages.page_005fcheckout_jsp._jspService(page_005fcheckout_jsp.java:97) ~[na:na]
        at com.commons.lang.Exceptions.shorten(Unknown Source) ~[ishop-commons-1.27.jar:na]
Caused by: javax.el.PropertyNotFoundException: Property 'currentShipper' not found on type com.frontline.shop.core.checkout.shipping.CheckoutShippingModel
        at javax.el.BeanELResolver$BeanProperties.get(BeanELResolver.java:214) ~[el-api.jar:2.1.FR]
        at javax.el.BeanELResolver$BeanProperties.access$400(BeanELResolver.java:191) ~[el-api.jar:2.1.FR]
        at javax.el.BeanELResolver.property(BeanELResolver.java:300) ~[el-api.jar:2.1.FR]
        at javax.el.BeanELResolver.getValue(BeanELResolver.java:81) ~[el-api.jar:2.1.FR]
        at javax.el.CompositeELResolver.getValue(CompositeELResolver.java:54) ~[el-api.jar:2.1.FR]
        at org.apache.el.parser.AstValue.getValue(AstValue.java:123) ~[jasper-el.jar:6.0.35]
        at org.apache.el.parser.AstDynamicExpression.getValue(AstDynamicExpression.java:42) ~[jasper-el.jar:6.0.35]
        at org.apache.el.parser.AstCompositeExpression.getValue(AstCompositeExpression.java:48) ~[jasper-el.jar:6.0.35]
        at org.apache.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:186) ~[jasper-el.jar:6.0.35]
        at org.apache.jasper.runtime.PageContextImpl.proprietaryEvaluate(PageContextImpl.java:938) ~[jasper.jar:6.0.35]
        at org.apache.jsp.WEB_002dINF.jsp.frontline.pages.checkout.checkout_005fshipper_jsp._jspx_meth_i_005fout_005f0(checkout_005fshipper_jsp.java:126) ~[na:na]
        at org.apache.jsp.WEB_002dINF.jsp.frontline.pages.checkout.checkout_005fshipper_005feditable_jsp._jspService(checkout_005fshipperjsp.java:75) ~[na:na]
        ... 1 common frames omitted
					```
					**multiline & grok filter!** <!-- .element: class="fragment" -->

					Note:
					- wie parse ich das denn jetzt?
					- multiline filter
				</script>
				<aside class="notes" data-markdown>
					
				</aside>
			</section>
			<section data-markdown>
				<script type="text/template">
					```
					filter {
						multiline {
							pattern => "^${TIME}"
							negate => true
							what => "previous"
						}
						grok {
							match => { "message" => "%{TIME:timestamp} \[%{DATA:thread}\] \[\#:(?:%{DATA:exHash})\] \[(?:%{DATA:sessionid})@(?:%{IPV4:ip}?)\] \[(?:%{DATA:rnd})\/(?:%{DATA:username})\] %{LOGLEVEL:level}\s*%{JAVACLASS:marker} - (?<logmessage>(?:.|\n\t)*)" }
						}
					}
					```

					Note:
						- funktionierte erstmal ganz gut
				</script>
			</section>

			<section>
				<h2>probleme</h2>
				<div style="display: flex;">
					<div class="fragment" style="flex:1;">
						<h4 style="">grok</h4>
						<ul>
							<li>komplexe Regex</li>
							<li>performance</li>
						</ul>
					</div>
					<div class="fragment" style="flex:1;">
						<h4 style="">multiline</h4>
						<ul>
							<li>multithreading</li>
							<li>performance</li>
						</ul>
					</div>
				</div>
				<br>
				<div class="fragment">
					<ul>
						<li>logstash-indexer stirbt</li>
						<li>Echtzeit geht verloren</li>
						<li>Logs gehen verloren</li>
					</ul>
				</div>
				<!-- [grok debugger](https://grokdebug.herokuapp.com/) <small>(grokdebug.herokuapp.com)</small> -->
				<aside class="notes" data-markdown>
				* shipper:
					* keine Filter
					* pro Zeile ein Event
					* kennt kein Format
				* skalierte aber überhaupt nicht mehr
				* Montag: logstash-indexer tot
					* logs hängen mehrere Tage zurück
					* del redis-key
        * next: <strong>Timo</strong>
				</aside>
			</section>
			<section data-markdown>
				<script type="text/template">
					# codecs
					## JSON?  <!-- .element: class="fragment" -->

					Note:
					- Logge für Logstash?
					- ES output? JSON?
				</script>
			</section>
			<section data-markdown>
				<script type="text/template">
					# JSON
					* strukturiert <!-- .element: class="fragment" -->
					* typisiert <!-- .element: class="fragment" -->

					Note:
					- JSON Format
					- keine Filter
					- eine Zeile pro Event
					- JSON Codec im Shipper & Indexer
				</script>
			</section>
			<section data-markdown>
				<script type="text/template">
					## [Apache LogFormat](http://httpd.apache.org/docs/2.4/mod/mod_log_config.html#logformat)


					```
					LogFormat '{"@timestamp":"%{%Y-%m-%dT%H:%M:%S%z}t","@version":"1","message":"access","clientip":"%a","ip":"%a","auth":"%u","ident":"%l","verb":"%m","url":{"original":"%<U","final":"%>U"},"query":"%q","request":{"original":"%<U%<q","final":"%>U%>q"},"response":%>s,"referrer":"%{Referer}i","agent":"%{User-Agent}i","xforwardedfor":"%{X-Forwarded-For}i","location":"%{Location}o","sessionid":"%{JSESSIONID}C","ishopuserid":"%{i.u.v2}C","microseconds":%D,"bytes":%B,"protocol":"%H","hostname":"%{Host}i","localip":"%A","servername":"%V","port":%p}' logstash
					```

					<img src="slow_requests.png" width="90%" />

					Note:
					- JSON Format
					- keine Filter
					- eine Zeile pro Event
					- JSON Codec im Shipper & Indexer
				</script>
			</section>

			<section data-markdown>
				<script type="text/template">
					## [logstash-logback-encoder](https://github.com/logstash/logstash-logback-encoder)

					```
					<configuration>
					  <appender name="ISHOP" class="FileAppender">
					    <file>${catalina.base}/logs/ishop.log</file>
					    <layout class="PatternLayout">
					      <pattern>
					      	%-28(%d{"yyyy-MM-dd'T'HH:mm:ss,SSS"} [%thread]) 
					      	[#:%exHash] [%X{sessionid}@%X{ipaddr}] 
					      	[%X{rnd}/%X{username}] %-5level 
					      	%marker %c - %m%n
					      </pattern>
					    </layout>
					  </appender>

					  <appender name="ISHOP_LOGSTASH" class="FileAppender">
					    <file>${catalina.base}/logs/ishop.json</file>
					    <encoder class="LogstashEncoder" />
					  </appender>

					  <root>
					    <level value="INFO" />
					    <appender-ref ref="ISHOP" />
					    <appender-ref ref="ISHOP_LOGSTASH" />
					  </root>
					</configuration>
					```
				</script>

				<aside class="notes" data-markdown>

				</aside>
			</section>
					
			<section>
				<h3>Logstash Shipper @ Frontline</h3>
				<pre><code data-trim>
input {
  file {
    path => "/var/logs/ishop_logstash.log"
    type => "ishop_logstash"
    codec => "json"
  }
}
				</code></pre>
				<pre class="fragment"><code data-trim>
output {
  if [type] == "ishop_logstash" {
    redis { 
      host => "redis" 
      data_type => "list" 
      key => "ftl-app-ishop-logstash" 
    }
  }
}
				</code></pre>
				<p class="fragment">keine Filter im Shipper</p>

				<aside class="notes" data-markdown>
					- IST Zustand
					- file input
					- redis output
					- **keine** filter
				</aside>
			</section>


			<section>
				<h3>Logstash Indexer @ Frontline</h3>
				
				<pre><code data-trim>
input {							
  redis {
    host => "redis"
    type => "ishop_logstash"
    key => "ftl-app-ishop-logstash"
    data_type => "list"
    codec => json
    add_field => { "source_host" => "live" }
  }
}
				</code></pre>
				<pre class="fragment"><code data-trim>
filter {
  if [source_host] == "live" and [type] == "ishop_logstash" {
    mutate {
      rename => [ "ipaddr", "ip" ]
    }
  }
}
				</code></pre>
				<pre class="fragment"><code data-trim>
output {
  if [source_host] == "live" {
    elasticsearch_http {
      host => "elasticsearch"
    }
  }
}
				</code></pre>
				<aside class="notes" data-markdown>
					- einheitliche Feldnamen
					- unique-request-id
					- MDC fields (class, method, ...)

					- redis input
					- es output
					- kleine filter

          - next: <strong>Jens</strong>
				</aside>
			</section>

			<section>
				<h2>Akzeptanz im Unternehmen</h2>

				<ul>
					<li class="fragment"><strong>Logs</strong> interessiert nur IT</li>
					<li class="fragment"><strong>Bestellungen</strong> interessieren ALLE</li>
					<li class="fragment">Realtime Analytics fasziniert</li>
				</ul>

				<aside class="notes" data-markdown>

				</aside>
			</section>
			<section>
				<h3>Dashboards auf TFT</h3>

				<img src="tft.jpg" width="75%" />
				<aside class="notes" data-markdown>

				</aside>
			</section>

			<section data-markdown>
				<script type="text/template">
					### inhaltliches logging
					```
					class OrderData {
						List<LineItem> lineItems;
						Money totalAmount;
						PaymentType paymentType;
						ShipperType shipperType; 
					}
					 ```
					```
					Logger.info(Markers.append("order", orderData), "order success");
					```
				</script>

				<aside class="notes" data-markdown>
					- Wie kann man ELK für nicht tech. Daten nutzen?
					- Beim Entwickeln an den Log-Payload denken
				</aside>
			</section>


			<section data-markdown>
				<script type="text/template">
					#### logstash output
					
					```
					{
						"message": "order success",
						"@version": "1",
						"@timestamp": "2015-03-10T13:53:25.775Z",
						"host": "work",
						"order": {
							"lineItems": [{}, {}, ...],
							"totalAmount": 23.55,
							"paymentType": "PAYPAL",
							"shipperType": "DHL"
						}
					}
					```
				</script>

				<aside class="notes" data-markdown>

				</aside>
			</section>

			<section>
				<h4>kibana dashboard</h4>
				<img src="orders_dashboard.png">

				<aside class="notes" data-markdown>
				</aside>
			</section>
			<section>
				<h4>kibana dashboard</h4>
				<img src="dashboard_ratings2.png">

				<aside class="notes" data-markdown>
				</aside>
			</section>

			<section data-markdown>
				<script type="text/template">
					<h2>Ausblick: Logfiles</h2>

					<ul>
						<li class="fragment">Nur noch JSON</li>
						<li class="fragment">kürzere RollingPolicy</li>
						<li class="fragment">Weniger Logfiles
							<ul>
								<li><code>application.log</code> vs.</li>
								<li><code>[order|paypal|billpay|diva|epoq|js|...].log</code></li>
							</ul>
						</li>
						<li class="fragment">[SiftingAppender](http://logback.qos.ch/manual/appenders.html#SiftingAppender) <small>(http://logback.qos.ch/manual/appenders.html#SiftingAppender)</small>
							<ul>
								<li>ein Logfile pro Thread</li>
							</ul>
						</li>
						<li class="fragment">LMAX Disruptor RingBuffer mit [*AsyncDisruptorAppender]( https://github.com/logstash/logstash-logback-encoder#usage) <small>( https://github.com/logstash/logstash-logback-encoder#usage)</small></li>
					</ul>
				</script>
				
				<aside class="notes" data-markdown>

				</aside>
			</section>

			<section data-markdown>
				<script type="text/template">
					<h2>Ausblick: Elasticsearch</h2>

					<ul>
						<li class="fragment">Mehr Nodes</li>						
						<li class="fragment">[Index Template](http://www.elastic.co/guide/en/elasticsearch/guide/current/index-templates.html) <small>(http://www.elastic.co/guide/en/elasticsearch/guide/current/index-templates.html)</small>
							<ul>
								<li><Kein code>_all</code> Feld</li>
								<li>Kompression</li>
								<li><code>not_analyzed</code> indizieren <br><small>(oder <code>*.raw</code> nutzen)</small></li>
							</ul>
						</li>
						<li class="fragment">[Doc Values](http://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html) <small>(http://www.elastic.co/guide/en/elasticsearch/guide/current/doc-values.html)</smalll> 
							<ul>
								<li><code>disk-based</code> statt <code>in-memory fielddata</code></li>
							</ul>
						</li>
					</ul>
				</script>
				
				<aside class="notes" data-markdown>

				</aside>
			</section>

			<section data-markdown>
				<script type="text/template">
					## Logvolumen
					* 45 Tage im ES
						* daily index
						* daily snapshots
						* daily cleanup
					* [Curator](https://github.com/elastic/curator) <small>(github.com/elastic/curator)</small>

					Note:

						ePages: 
						4x ES Nodes, (je 96GB RAM, 1.8TB HDD),
						3.7Mrd Docs, 94 Indices,
						5.0TB Index Size, 32GB/Tag Log Events
						3 Monate Retention Time
				</script>
			</section>

			<section data-markdown>
				<script type="text/template">
					## Staging ELK
					* neue Konfigurationen testen
					* Jede Komponente leicht skalierbar
						* docker?
				</script>
			</section>

			<section data-markdown>
				<script type="text/template">
					# Vielen Dank

					## Fragen?

				</script>

				<aside class="notes" data-markdown>

				</aside>
			</section>

			</div>
		</div>

		<script src="lib/js/head.min.js"></script>
		<script src="js/reveal.js"></script>

		<script>

			// Full list of configuration options available at:
			// https://github.com/hakimel/reveal.js#configuration
			Reveal.initialize({
				controls: true,
				progress: true,
				history: true,
				center: true,

				transition: 'slide', // none/fade/slide/convex/concave/zoom

				// Optional reveal.js plugins
				dependencies: [
					{ src: 'lib/js/classList.js', condition: function() { return !document.body.classList; } },
					{ src: 'plugin/markdown/marked.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } },
					{ src: 'plugin/markdown/markdown.js', condition: function() { return !!document.querySelector( '[data-markdown]' ); } },
					{ src: 'plugin/highlight/highlight.js', async: true, condition: function() { return !!document.querySelector( 'pre code' ); }, callback: function() { hljs.initHighlightingOnLoad(); } },
					{ src: 'plugin/zoom-js/zoom.js', async: true },
					{ src: 'plugin/notes/notes.js', async: true }
				]
			});

		</script>

	</body>
</html>
