title: Prometheus metrics do not disapper after being deleted
body: <p>I need to delete some Prometheus metrics originating from one host. For simplicity let's assume all metrics from the host need to be deleted.</p>
  
  <p>Executed POST to a <code>delete_series</code> endpoint on both VMs. Got 204 as expected. The metrics did not disappear (still visible in web / browser interface, even after forcing cache refresh via <code>CTRL+R</code>, <code>CTRL+F5</code>).</p>
  
  <p>Also tried forcing <code>clean_tombstones</code>, got 204, but not change.</p>
  
  <p>Restarted both instances, then also waited a day in a hope that "something" will reload itself.</p>
  
  <p>The commands used:
  <code># curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={instance="my-hostname"}' -v</code></p>
  
  <p><code># curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones' -v</code></p>
  
  <p>The output:</p>
  
  <pre><code>*   Trying 127.0.0.1...
  * Connected to localhost (127.0.0.1) port 9090 (#0)
  &gt; POST /api/v1/admin/tsdb/delete_series?match[]={instance="my-hostname"} HTTP/1.1
  &gt; Host: localhost:9090
  &gt; User-Agent: curl/7.47.0
  &gt; Accept: */*
  &gt;
  &lt; HTTP/1.1 204 No Content
  &lt; Access-Control-Allow-Headers: Accept, Authorization, Content-Type, Origin
  &lt; Access-Control-Allow-Methods: GET, OPTIONS
  &lt; Access-Control-Allow-Origin: *
  &lt; Access-Control-Expose-Headers: Date
  &lt; Date: Tue, 23 Jul 2019 07:54:30 GMT
  &lt;
  * Connection #0 to host localhost left intact
  </code></pre>
  
  <p>My Prometheus cluster consists of two VMs running Prometheus 2.6.1.</p>
  
  <p><code>admin-api</code> is enabled: </p>
  
  <pre><code>           └─28557 /opt/prometheus/prometheus-server/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/data/prometheus/databases --storage.tsdb.retention=15d --web.enable-admin-api --web.external-url=http://redacted --log.level=debug
  </code></pre>