anchor.diff:  
   * Routers are under heavy load from incoming client requests.
   
  -* Apps are taking a long time to process requests. This increases the number of concurrent threads held open by Gorouter, reducing capacity to handle requests for other apps.
  +* Apps are taking a long time to process requests. This increases the number of concurrent threads
  +held open by Gorouter, reducing capacity to handle requests for other apps.
   
   ## <a id="ops-recommendations"></a> Operations Recommendations
   
  -* Monitor CPU load for Gorouters. At high CPU (70%+), latency increases. If the Gorouter CPU reaches this threshold, consider adding another Gorouter instance.
  -
  -* Monitor latency of all routers using metrics from the Firehose. Do not monitor the average latency across all routers. Instead, monitor them individually on the same graph.
  -
  -* Consider using Pingdom against an app on your <%= vars.app_runtime_abbr %> deployment to monitor latency and uptime. For more information, see the [Pingdom](http://www.pingdom.com) website.
  -
  -* Consider enabling access logs on your load balancer. To enable access logs, see your load balancer documentation. Just as you use Gorouter access log messages above to determine latency from Gorouter, you can compare load balancer logs to identify latency between the load balancer and Gorouter. You can also compare load balancer response times with the client response times to identify latency between client and load balancer.
  -
  -* Deploy a nozzle to the Loggregator Firehose to track metrics for Gorouter. For more information, see [Deploying a Nozzle to the Loggregator Firehose](../loggregator/nozzle-tutorial.html). Available metrics include:
  -	* CPU utilization
  -	* Latency
  -	* Requests per second
  +* Monitor CPU load for Gorouters. At high CPU (70%+), latency increases. If the Gorouter CPU
  +reaches this threshold, consider adding another Gorouter instance.
  +
  +* Monitor latency of all routers using metrics from the Firehose. Do not monitor the average
  +latency across all routers. Instead, monitor them individually on the same graph.
  +
  +* Consider using Pingdom against an app on your <%= vars.app_runtime_abbr %> deployment to monitor
  +latency and uptime. For more information, see the [Pingdom](http://www.pingdom.com) website.
  +
  +* Consider enabling access logs on your load balancer. To enable access logs, see your load
  +balancer documentation. Just as you use Gorouter access log messages above to determine latency
  +from Gorouter, you can compare load balancer logs to identify latency between the load balancer and
  +Gorouter. You can also compare load balancer response times with the client response times to
  +identify latency between client and load balancer.
  +
  +* Deploy a nozzle to the Loggregator Firehose to track metrics for Gorouter. For more information,
  +see [Deploying a Nozzle to the Loggregator Firehose](../loggregator/nozzle-tutorial.html).
  +Available metrics include:
  +
  +    * CPU utilization
  +    * Latency
  +    * Requests per second
  
positive.code: ---
  title: Troubleshooting Slow Requests in Cloud Foundry
  owner: Routing
  ---
  
  <% current_page.data.title = "Troubleshooting Slow Requests in " + vars.app_runtime_abbr %>
  
  This topic provides experiments you can run to help determine what part of the <%= vars.app_runtime_full %> (<%= vars.app_runtime_abbr %>) request flow is adding latency to your requests. The procedures in this topic are intended to be run in order.
  
  ## <a id="overview"></a>Overview
  
  
negative.code: 
  <%= image_tag("request_lifecycle.png", :alt => "There are four boxes labeled, from left to right, 'Client', 'Load Balancer', 'Router', and 'Backend'. A black box frames the 'Router' and 'Backend' boxes and is labeled 'CF'. A horizontal black arrow points left to right from the 'Client' box to the 'Load Balancer' box. A horizontal black arrow points left to right from the 'Load Balancer' box to the 'Router' box. A horizontal black arrow points left to right from the 'Router' box to the 'Backend' box. A horizontal black arrow points right to left from the 'Backend' box back to the 'Router' box. A horizontal black arrow points right to left from the 'Router' box back to the 'Load Balancer' box. A horizontal black arrow points right to left from the 'Load Balancer' box back to the 'Client' box.") %>
  
  Any of the components in the diagram above can cause latency. Delays can also come from the network
  itself.
  
  To troubleshoot slow requests and diagnose what might be causing latency to your app requests:
  
  * [Experiment 1: Measure Total Round-Trip App Requests](#total-latency)
  * [Experiment 2: View Request Time in Access Logs](#within-cf)
  * [Experiment 3: Duplicate Latency on Another Endpoint](#duplicate-latency)
  * [Experiment 4: Remove the Load Balancer from the Request Path](#remove-the-load-balancer)
  * [Experiment 5: Remove Gorouter from the Request Path](#remove-the-gorouter)
  * [Experiment 6: Test the Network between the Router and the App](#test-internal-network)
  
  After you determine the cause of latency, see the following sections for more information:
  
  * [Use App Logs to Locate Delays in <%= vars.app_runtime_abbr %>](#app-logs)
  * [Causes for Gorouter Latency](#gorouter-latency)
  * [Operations Recommendations](#ops-recommendations)
  
  ## <a id="total-latency"></a> Experiment 1: Measure Total Round-Trip App Requests
  
  To measure the total round-trip time for your deployed app that is experiencing latency, run:
  
  ```
  time curl -v APP-ENDPOINT
  ```
  Where `APP-ENDPOINT` is the URL endpoint for the deployed app.
  
  For example:
  
  