title: Prometheus OR when using rate()
body: <p><strong>Summary</strong></p>
  
  <p>I'm trying to figure out how to properly use the OR <code>|</code> operator in a Prometheus query because my imported Grafana dashboard is not working.</p>
  
  <p><strong>Long version</strong></p>
  
  <p>I'm trying to debug a Grafana dashboard based on some data scraped from my Kubernetes pods running <a href="https://github.com/AppMetrics/Prometheus" rel="nofollow noreferrer">AppMetrics/Prometheus</a>; the dashboard is <a href="https://grafana.com/dashboards/2204" rel="nofollow noreferrer">here</a>. Basically what happens is that when the value "All" for the <code>server</code> is selected on the Grafana dashboard (<code>server</code> is an individual pod in this case), no data appears. However, when I select an individual pod, then data does appear.</p>
  
  <p>Here's an example of the same metric scraped from the two pods:</p>
  
  <pre><code># HELP application_httprequests_transactions 
  # TYPE application_httprequests_transactions summary
  application_httprequests_transactions_sum{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test"} 5.006965628
  application_httprequests_transactions_count{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test"} 1367
  application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.5"} 0.000202825
  application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.75"} 0.000279318
  application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.95"} 0.000329862
  application_httprequests_transactions{server="myapp-test-58d94bf78d-jdq78",app="MyApp",env="test",quantile="0.99"} 0.055584233
  
  # HELP application_httprequests_transactions 
  # TYPE application_httprequests_transactions summary
  application_httprequests_transactions_sum{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test"} 6.10214788
  application_httprequests_transactions_count{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test"} 1363
  application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.5"} 0.000218548
  application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.75"} 0.000277483
  application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.95"} 0.033821094
  application_httprequests_transactions{server="myapp-test-58d94bf78d-l9tdv",app="MyApp",env="test",quantile="0.99"} 0.097113234
  </code></pre>
  
  <p>I ran the Query inspector in Grafana to find out which query it is calling, and then ran the PromQL query in Prometheus itself. Basically, when I execute the following PromQL queries individually, they return data:</p>
  
  <pre><code>rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-l9tdv"}[15m])*60
  
  rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-jdq78"}[15m])*60
  </code></pre>
  
  <p>However, when I try to use PromQL's <code>|</code> operator to combine them, I don't get data back:</p>
  
  <pre><code>rate(application_httprequests_transactions_count{env="test",app="MyApp",server="myapp-test-58d94bf78d-l9tdv|myapp-test-58d94bf78d-jdq78"}[15m])*60
  </code></pre>
  
  <p>Here's the raw output from Grafana's query inspector:</p>
  
  <pre><code>xhrStatus:"complete"
  
  request:Object
  
  method:"GET"
  url:"api/datasources/proxy/56/api/v1/query_range?query=rate(application_httprequests_transactions_count%7Benv%3D%22test%22%2Capp%3D%22MyApp%22%2Cserver%3D%22myapp-test-58d94bf78d-jdq78%7Cmyapp-test-58d94bf78d-l9tdv%7Cmyapp-test-5b8c9845fb-7lklm%7Cmyapp-test-5b8c9845fb-8jf7n%7Cmyapp-test-5b8c9845fb-d9x5c%7Cmyapp-test-5b8c9845fb-fw4gj%7Cmyapp-test-5b8c9845fb-vtl9z%7Cmyapp-test-5b8c9845fb-vv7xv%7Cmyapp-test-5b8c9845fb-wq9bs%7Cmyapp-test-5b8c9845fb-xqfrt%7Cmyapp-test-69999d58b5-549vd%7Cmyapp-test-69999d58b5-lmp8x%7Cmyapp-test-69999d58b5-nbvt9%7Cmyapp-test-69999d58b5-qphj2%7Cmyapp-test-6b8dcc5ffb-gjjvj%7Cmyapp-test-6b8dcc5ffb-rxfk2%7Cmyapp-test-7fdf446767-bzhm2%7Cmyapp-test-7fdf446767-hp46w%7Cmyapp-test-7fdf446767-rhqhq%7Cmyapp-test-7fdf446767-wxmm2%22%7D%5B1m%5D)*60&amp;start=1540574190&amp;end=1540574505&amp;step=15"
  
  response:Object
  
  status:"success"
  
  data:Object
  
  resultType:"matrix"
  
  result:Array[0] =&gt; []
  </code></pre>
  
  <p>I opened a GitHub issue for this as well; it has a quick GIF screen recording showing what I mean: <a href="https://github.com/AppMetrics/Prometheus/issues/43" rel="nofollow noreferrer">AppMetrics/Prometheus#43</a></p>