anchor.diff:  
   ZaggClient checks are simple gathering of Zabbix items and sending them to the ZaggServer (which eventually sends them to a Zabbix Server).
   
  +*Note: This document assumes that that the item has already been configured in Zabbix.  These checks will update the items, but will not add them.  Please configure Zabbix appropriately.*
  +
  +==== Dynamic Zagg/Zabbix Objects
  +
  +Zagg supports Zabbix dynamic object support.  Explaining the Low Level Discovery rules is out of scope of this document.  Please refer to https://www.zabbix.com/documentation/2.4/manual/discovery/low_level_discovery[Zabbix's documentation] for more information.
  +
  +This document will show how to add dynamic objects and items, then show how to send data to update those items.
  +
  +Dynamic objects in Zabbix require 3 things:
  +
  +* discovery key (defined in Zabbix Template -> Discovery Rule)
  +* Macro Identifier - string to identify the Macro (we call this the macro string)
  +* Macro Name(s) - string(s) to name an invidual object.
  +
  +Dynamic objects need to continually be updated in Zabbix.  If they are not, Zabbix will expunge the objects after a defined set of time.
  +
  +After these are created, we can start creating dynamic items (defined in Zabbix) to monitor.
  +
  +A real world example.  Let's say that we want to monitor all file systems on the system and how much space they are using.  All systems are not going to have the same file systems, so this is a perfect use case for dynamic objects.
  +
  +In this example, the discovery key would be "filesystems"
  +The Macro Identifier would be "#FILESYSTEMS".  The capatilization and the '#' are Zabbix conventions
  +The Macro Names would be the file systems we want to monitor "/", "/var", "/home"
  +
  +More examples will be shown below on how to write and collect dynamic data.
   
   === Zagg Clients Overview
   
  
positive.code: 
  ops-zagg-client -k web.google.status -o ${GOOGLE_STATUS}
  ----
  
negative.code: zs.add_zabbix_keys({ 'web.google.status' : google_response })
  zs.send_metrics()
  
  ----
  
  ===== Python Dynamic Object/item example
  
  Here is an example on how to write a dynamic object that uses the macros strings and macro names.
  
  In this example we will assume that we want to monitor a file system that has this output of 'df -h'
  ----
  $ df -h
  Filesystem      Size  Used Avail Use% Mounted on
  /dev/sda2       219G   62G  158G  29% /
  devtmpfs        9.7G     0  9.7G   0% /dev
  tmpfs           9.8G  8.0K  9.8G   1% /dev/shm
  tmpfs           9.8G  281M  9.5G   3% /run
  tmpfs           9.8G     0  9.8G   0% /sys/fs/cgroup
  /dev/sda1       497M  291M  207M  59% /boot
  /dev/md0         11T  60.1G 40.8G 56% /home
  ----
  In this example we want to report on /, /boot, and /home.  The tmpfs drives can be ignored at this time.
  
  * First, in Zabbix we would add a discovery rule under a template.  We would call the key in that discover rule "filesystems"
  * Next, we would add a dynamic item under this newly created discovery rule called "filesystem.percent_full"
  * We will use the Macro string of "#FILESYSTEM"
  
  
  .ZaggSender python dynamic item script example
  ----
  #!/usr/bin/evn python
  import subprocess
  from openshift_tools.monitoring.zagg_sender import ZaggSender
  
  df_cmd = "df"
  
  # get the output of df
  output = subprocess.check_output(df_cmd, shell=True)
  
  # this will only collect the filesystems we care about, and strip out the other lines
  filesystem_list = [f for f in output.split('\n') if f.startswith("/dev")]
  
  # we need a dict in the format of {"filesystem1_name" : "% free"}
  filesystem_dict = {}
  for f in filesystem_list:
      line = f.split()
      filesystem_dict[line[5]] = line[4].replace("%","")
  
  # we now have all the data we want.  Let's send it to Zagg
  
  zs = ZaggSender()
  
  # This will bundle up the dynamic object
  zs.add_zabbix_dynamic_item('filesystem', '#FILESYSTEM', filesystem_dict.keys() )
  
  # This will bundle up the dynamic items
  for filesys_name, filesys_percent in filesytem_dict.iteritems():
      zs.add_zabbix_keys({'%s[%s]' % (#FILESYSTEM, filesys_name): filesys_percent})
  
  # Finally, sent them to zabbix
  zs.send_metrics()
  ----
  
  ==== ops-zagg-client
  