anchor.diff:    7. Support for distributed (but not YARN ready) applications using Apache Slider
   <<<<<<< HEAD
   <<<<<<< HEAD
  +<<<<<<< HEAD
   >>>>>>> 3b7285c... PERI-2 Transferring blueprint from apiary.io
   =======
     
   #Group Clusters 
   =======
  +=======
  +  8. Attach priorities to SLAs
  +
  +>>>>>>> a0f5794... PERI-2 apiary changes
   
   #Group Clusters
   >>>>>>> 2a42905... PERI-2 update apiary
   Clusters related resources of the **Periscope API**.
  +A Hadoop cluster is a set of components and services launched in order to store, analyze and process unstructured data. 
  +Periscope can work with any Hadoop 2/ YARN cluster provisioned with Apache Ambari, and supports any YARN application.
   
   ##Clusters [/clusters/{id}]
  -###Add a cluster [POST]
  -Add a cluster to be monitored by Periscope.
  +###Add a cluster to be monitored by Periscope[POST]
  +Add a Hadoop cluster to be monitored by Periscope. Note that Periscope does not require any pre-installation of components on cluster nodes. To link a Hadoop cluster with Periscope the only required this is the Apache Ambari REST API endpoint.
   
   + Parameters
       + id (required String `id`) ... The id of the cluster to be monitored.
  
positive.code: 
  
  ##  Clusters [/clusters/{id}]
  ###Retrieved the monitored cluster information [GET]
  
  + Parameters
      + id (required String `id`) ... The id of the cluster.
  
negative.code: 
  ![SequenceIQ](https://raw.githubusercontent.com/sequenceiq/sequenceiq.github.io/master/img/logo.png)
  
  
  *Periscope is a powerful, fast, thick and top-to-bottom right-hander, eastward from Sumbawa's famous west-coast. Timing is critical, as needs a number of elements to align before it shows its true colors.*
  
  <<<<<<< HEAD
  *Periscope is a heuristic Hadoop scheduler you associate with a QoS profile. Built on YARN schedulers, cloud and VM resource management API's it allows you to associate SLA's to applications and customers.*
  <<<<<<< HEAD
  >>>>>>> f4bf1b1... PERI-2 apiary template added
  =======
  =======
  *Periscope is a heuristic Hadoop scheduler you associate with a QoS profile. Built on YARN schedulers, cloud and VM resource management API's it allows you to associate SLAs to applications and customers.*
  >>>>>>> a0f5794... PERI-2 apiary changes
  
  ##Overview
  
  The purpose of Periscope is to bring QoS to a multi-tenant Hadoop cluster, while allowing to apply SLAs to individual applications and customers.
  At [SequenceIQ](http://sequenceiq.com) working with multi-tenant Hadoop clusters for quite a while we have always seen the same frustration and fight for resource between users.
  The **FairScheduler** was partially solving this problem - bringing in fairness based on the notion of [Dominant Resource Fairness](http://static.usenix.org/event/nsdi11/tech/full_papers/Ghodsi.pdf).
  With the emergence of Hadoop 2 YARN and the **CapacityScheduler** we had the option to maximize throughput and utilization for a multi-tenant cluster in an operator-friendly manner.
  The scheduler works around the concept of queues. These queues are typically setup by administrators to reflect the economics of the shared cluster.
  While this is a pretty good abstraction and brings some level of SLA for `predictable` workloads, it often needs proper `design ahead`.
  The queue hierarchy and resource allocation needs to be changed when new tenants and workloads are moved to the cluster.
  
  Periscope was designed around the idea of `dynamic` clusters - without any need to preconfigure queues, cluster nodes or apply capacity planning ahead.
  
  ##How it works
  
  Periscope monitors the application progress, the number of YARN containers/resources and their allocation, queue depths, and the number of available cluster nodes and their health.
  Since we have switched to YARN a while ago (been among the first adopters) we have run an open source [monitoring project](https://github.com/sequenceiq/yarn-monitoring), based on R.
  We have been collecting metrics from the YARN Timeline server, Hadoop Metrics2 and Ambari's Nagios/Ganglia - and profiling the applications and correlating with these metrics.
  One of the key findings was that while low level metrics are good to understand the cluster health - they might not necessarily help on making decisions when applying different SLAs on a multi-tenant cluster. 
  Focusing on higher level building blocks as queue depth, YARN containers, etc actually brings in the same quality of service, while not being lost in low level details.
  
  _Example: Applying SLA based on `resource` load might not be the best solution - each application tasks generates different loads, and a CPU heavy step might be followed by an I/O heavy one - making a decision based on a low `snapshot` might not be the right option (the FairScheduler does something similar for the dominant resource).
  Also note that a YARN cluster can run different applications - MR2, HBase, Spark, etc - and they all generate different load across different timeframes.
  When YARN allocates containers it associates `resources` - it's actually more predictable to let YARN to deal with the resource allocation, and have Periscope orchestrate the process._
  
  Periscope works with two types of Hadoop clusters: `static` and `dynamic`. Periscope does not require any pre-installation - the only thing it requires is to be `attached` to an Ambari server's REST API.
  
  ##Clusters
  
  ### Static clusters
  From Periscope point of view we consider a cluster `static` when the cluster capacity can't be increased horizontally.
  This means that the hardware resources are already given - and the throughput can't be increased by adding new nodes.
  Periscope introspects the job submission process, monitors the applications and applies the following SLAs:
  
    1. Application ordering - can guaranty that a higher priority application finishes before another one (supporting parallel or sequential execution)
    2. Moves running applications between priority queues
    3. *Attempts* to enforce time based SLA (execution time, finish by, finish between, recurring)
    4. *Attempts* to enforce guaranteed cluster capacity requests ( x % of the resources)
    5. Support for distributed (but not YARN ready) applications using Apache Slider
    6. Attach priorities to SLAs
  
  ### Dynamic clusters
  From Periscope point of view we consider a cluster `dynamic` when the cluster capacity can be increased horizontally.
  This means that nodes can be added or removed dynamically - thus the throughput can be increased or decreased based on the cluster load, and scheduled applications.
  In order to do that Periscope instructs [Cloudbreak](http://sequenceiq.com/cloudbreak/) to add or remove nodes from the cluster based on the SLAs and thus continuously provide a high *quality of service* for the multi-tenand Hadoop cluster.
  Just to refresh memories - [Cloudbreak](http://sequenceiq.com/products.html) is [SequenceIQ's](http://sequenceiq.com) open source, cloud agnostic Hadoop as a Service API.
  Given the option of provisioning or decommissioning cluster nodes on the fly, Periscope allows you to use the following set of SLAs:
  
    1. Application ordering - can guaranty that a higher priority application finishes before another one (supporting parallel or sequential execution)
    2. Moves running applications between priority queues
  