title: Query most recent values for active & stale Prometheus timeseries over range query
body: <p>I am recording and monitoring SLOs (server-side request duration) of Kubernetes Pods via Prometheus using a <a href="https://godoc.org/github.com/prometheus/client_golang/prometheus#HistogramVec" rel="nofollow noreferrer">HistogramVec</a> within a Golang HTTP server. Every request’s duration is timed and persisted as described in the <a href="https://prometheus.io/docs/practices/histograms/" rel="nofollow noreferrer">Prometheus practices</a> and partitioned by status code, method and HTTP path. </p>
  
  <p>I am running autoscaling experiments therefore Pods are created &amp; terminated. After each experiment I fetch the metrics for all pods (including the ones already deleted) and plot a cumulative distribution, e.g.:
  <a href="https://i.stack.imgur.com/KoQkS.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/KoQkS.png" alt="CDF"></a>
  In order to make these plots more “accurate”, I opted for many, smaller histogram buckets and aggregate &amp; analyze the data locally and do not use the built-in <a href="https://prometheus.io/docs/practices/histograms/#quantiles" rel="nofollow noreferrer">Histogram Quantiles</a>. <strong>The ideal query would therefore return only the most recent value for all time series that have existed over a specified time range (green + red circles).</strong>
  <a href="https://i.stack.imgur.com/Wfs6H.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/Wfs6H.png" alt="Timeseries"></a>
  Currently, I am using a <a href="https://prometheus.io/docs/prometheus/latest/querying/api/#range-queries" rel="nofollow noreferrer">range query</a> within the script generating all the plots, e.g.:</p>
  
  <pre><code>http://localhost:9090/api/v1/query_range?query=http_request_duration_milliseconds_bucket{path="/service/login"}&amp;start=1591803898&amp;end=1591804801&amp;step=5s
  </code></pre>
  
  <p>However, I am aware that this is highly inefficient and costly as it retrieves a huge amount of surplus data even though I am only interested in the very last value for each individual time series. On the other hand, if I use an instant query, I only get the values for a specified moment, thus I’d need to shoot multiple queries &amp; first find out when some time series (red circles) were marked stale - which doesn’t seem great either. </p>
  
  <p>So, basically I'm looking for a way to work around the <a href="https://prometheus.io/docs/prometheus/latest/querying/basics/#staleness" rel="nofollow noreferrer">Prometheus basics on staleness</a>, and stop stale time series from "disappearing":</p>
  
  <blockquote>
    <p>If no sample is found (by default) 5 minutes before a sampling timestamp, no value is returned for that time series at this point in time. This effectively means that time series "disappear" from graphs at times where their latest collected sample is older than 5 minutes or after they are marked stale.</p>
  </blockquote>
  
  <p>I am almost certain that there is a way to do this (e.g. an option to simply include stale time series), but I haven’t been able to put it together so far.</p>