title: Scale-able plotting of data stream from Elasticsearch
body: <p>Assume that I have a set of documents stored in an index of Elasticsearch, such that each document has the following (simplified) form:</p>
  
  <pre><code>{
    "timestamp": N,
    "val": X
  }
  </code></pre>
  
  <p>where <code>N</code> is a long integer representing a unix-timestamp and <code>X</code> is some float.</p>
  
  <p>My goal is to plot the behavior of <code>val</code> over time; in other words, obtain a graph where the x-axis is the time(stamp) and the y-axis is the <code>val</code>.</p>
  
  <h2>Medium-Small number of document</h2>
  
  <p>If the number of documents stored in the index is medium-small, then using <code>python</code> I could do the following. Scan the documents, using for example the <a href="https://elasticsearch-py.readthedocs.org/en/master/helpers.html#elasticsearch.helpers.scan" rel="nofollow noreferrer">scan-helper</a>, and create a list of the JSON documents. Next, convert the list into a <code>pandas.DataFrame</code> and sort its line according to the <code>timestamp</code>. Finally, I can, now easily, plot the data as I described above. Here is a minimal example:</p>
  
  <pre><code>docs = scan(
              es, # instance of es-client
              index = 'myIndex',
              doc_type = 'myDocType')
  docsList = []
  for doc in docs:
      docsList.append(doc)
  dfDocs = pandas.DataFrame(docsList)
  dfDocsSorted = dfDocs.sort(columns='timestamp')
  dfDocsSorted.plot(x='timestamp', y='val')
  </code></pre>
  
  <p>Here is how the output looks like for some sample data set:</p>
  
  <p><img src="https://i.stack.imgur.com/l5RCM.png" alt="enter image description here"></p>
  
  <p>I find it a rather clean and fine solution, given the number of documents is limited.</p>
  
  <h2>Large number of documents</h2>
  
  <p>What is the "right" way to do the same as above in the case where the number of documents is <em>large</em>? Note that the sorting step above is rather mandatory, as far as I can tell, since the <code>scan</code> returns documents in a "random" order. Therefore, if the number of documents is large, this step (and the storing of the data) will become an issue.</p>
  
  <p>Is there a canonical way to achieve this using Elasticsearch? Or am I bound to carry out a pre-processing locally before being able to plot the data?</p>