anchor.diff:  
   1. Run `nats-sub -s "nats://${NATS_USERNAME}:${NATS_PASSWORD}@${NATS_HOST}:${NATS_PORT}" ">" | grep APPLICATION-GUID` to subscribe to NATS and show messages related to the application.
   
  -##<a id='htop'></a>Tracking Down an Application by CPU Usage ##
  -
  -Applications do not always functions as designed.
  -If you notice a sharp decrease in available staging VMs or an unanticipated
  -increase in CPU, disk utilization, or memory utilization across the VMs in your
  -deployment, use the following steps to determine if a specific application is at
  -fault.
  -
  -1. Run `bosh ssh` to open a secure shell into the DEA node that is experiencing the unexpected high resource usage.
  -
  -1. Install the [htop](http://hisham.hm/htop/) interactive process viewer.
  -In a terminal window, run `htop` to launch the application.
  -
  -    <%= image_tag("../images/htop/htop-open.png") %>
  -
  -1. Press the `F6` key and use the arrow keys to sort the htop output by the "TIME+" column.
  -Processes using excessive resources display a TIME+ value significantly greater than other processes.
  -Note the names of each of these processes.
  -
  -    <%= image_tag("../images/htop/htop-name.png") %>
  -
  -1. Press the `F5` key to view all the processes as a tree.
  -
  -
  -1. Press the `/` key to search by the name of the processes. Locate the names of any processes found to be using excessive resources. Note the `wshd` command above each of these processes. Each of these `wshd` commands is a Warden handle for an application.
  -
  -    In this example, the `wshd` command is `17vg99ooo1n`:
  -
  -    <%= image_tag("../images/htop/htop-find.png") %>
  -
  -1. Press the `F10` key to exit `htop`.
  -
  -1.  Run `sudo su -` to enter the root environment with root privileges.
  -
  -1. Search `/var/vcap/data/dea_next/db/instances.json` to find the Warden handles
  -identified using `htop`.
  -The JSON object containing a Warden handle also contains the application GUID
  -and name.
  -
  -    In this example, the JSON object containing the Warden handle `17vg99ooo1n` also contains the application GUID `a8a5fb5e-f41d-49b5-af18-576a28f80e7f` and name `console-workers`.
  -
  -    <%= image_tag("../images/htop/htop-log-handle.png") %>
  -
  -    <%= image_tag("../images/htop/htop-log-name.png") %>
  -
   ##<a id='hm9000'></a>Viewing the HM9000 Data Store ##
   
   The HM9000 data store contains information about every VM and application in a
  
positive.code: 
  1. Run `bosh ssh hm9000_z1/0` to open a secure shell into the HM9000 VM.
  1. Run `/var/vcap/packages/hm9000/hm9000 dump --config=/var/vcap/jobs/hm9000/config.json > /tmp/hm9000ds`.
  This command outputs the data store into a text file, `/tmp/hm9000ds`.
  
  
negative.code: ---
  title: Troubleshooting Applications
  ---
  
  ##<a id='htop'></a>Tracking Down an Application by CPU Usage ##
  
  Applications do not always functions as designed.
  If you notice a sharp decrease in available staging VMs or an unanticipated
  increase in CPU, disk utilization, or memory utilization across the VMs in your
  deployment, use the following steps to determine if a specific application is at
  fault.
  
  1. Run `bosh ssh` to open a secure shell into the DEA node that is experiencing
  the unexpected high resource usage.
  
  1. Install the [htop](http://hisham.hm/htop/) interactive process viewer.
  In a terminal window, run `htop` to launch the application.
  
      <%= image_tag("../images/htop/htop-open.png") %>
  
  1. Press the `F6` key and use the arrow keys to sort the htop output by the
  "TIME+" column.
  Processes using excessive resources display a TIME+ value significantly greater
  than other processes.
  Note the names of each of these processes.
  
      <%= image_tag("../images/htop/htop-name.png") %>
  
  1. Press the `F5` key to view all the processes as a tree.
  
  1. Press the `/` key to search by the name of the processes.
  Locate the names of any processes found to be using excessive resources.
  Note the `wshd` command above each of these processes.
  Each of these `wshd` commands is a **Warden handle** for an application.
  
      In this example, the `wshd` command is `17vg99ooo1n`:
  
      <%= image_tag("../images/htop/htop-find.png") %>
  
  1. Press the `F10` key to exit `htop`.
  
  1.  Run `sudo su -` to enter the root environment with root privileges.
  
  1. Search `/var/vcap/data/dea_next/db/instances.json` to find the Warden handles
  identified using `htop`.
  The JSON object containing a Warden handle also contains the application GUID
  and name.
  
      In this example, the JSON object containing the Warden handle `17vg99ooo1n` also contains the application GUID `a8a5fb5e-f41d-49b5-af18-576a28f80e7f` and name `console-workers`.
  
      <%= image_tag("../images/htop/htop-log-handle.png") %>
  
      <%= image_tag("../images/htop/htop-log-name.png") %>
  
  ##<a id='access-warden'></a>Accessing a Warden Container ##
  
  Cloud Foundry runs each application in an isolated environment called a **Warden container**.
  When troubleshooting an issue with an application, you might want direct access
  to the Warden container running the application.
  
  To access a Warden container:
  
  1. Run `bosh ssh` to open a secure shell into the DEA node containing the
  application you want to troubleshoot.
  
  1. Use the process described in [Tracking Down an Application by CPU Usage](#htop) to find the Warden handle for the application.
  
  1. On the DEA node, use the following command to open an interactive shell in
  the Warden container running your application:
  
  `/var/vcap/packages/warden/warden/src/wsh/wsh --socket /var/vcap/data/warden/depot/WARDEN-HANDLE/run/wshd.sock --user vcap`
  
  <pre class="terminal">
  $ /var/vcap/packages/warden/warden/src/wsh/wsh --socket /var/vcap/data/warden/depot/17vg99ooo1n/run/wshd.sock --user vcap
  
  password for vcap: ********
  Logged into vcap@17vg99ooo1n
  $
  </pre>
  
  ##<a id='keep-container'></a>Preserving a Warden Container ##
  
  By default, a Warden container exists only as long as the application it runs
  exists.
  If an application consistently crashes while starting, you might not be able to
  access the Warden container running the application without changing this
  default behavior.
  
  To keep a Warden container from being deleted even when the application it runs
  has stopped, set the `container_grace_time` parameter in the [Warden linux.yml](https://github.com/cloudfoundry/warden/blob/master/warden/config/linux.yml) file to `~`.
  
  ```
  ---
  server:
    container_klass: Warden::Container::Linux
  
    container_grace_time: ~
  
    unix_domain_permissions: 0777
    . . .
  ```
  
  ##<a id='monitor-nats'></a>Monitoring NATS Message Bus Traffic ##
  
  