title: Require advice on Time Series Database solution that fits with Apache Spark
body: <p>For example, lets say I wish to analyze a months worth of company data for trends. I plan on doing regression analysis and classification using an MLP.</p>
  
  <p>A months worth of data has  ~10 billion data points (rows).
  There are 30 dimensions to the data.
  12 features are numeric (integer or float; continuous).
  The rest are categoric (integer or string).</p>
  
  <p>Currently the data is stored in flat files (CSV) and is processed and delivered in batches. Data analysis is carried out in R.</p>
  
  <p>I want to:</p>
  
  <ol>
  <li>change this to <strong>stream</strong> processed (rather than batch process).</li>
  <li>offload the computation to a <strong>Spark cluster</strong></li>
  <li>house the data in a <strong>time-series database</strong> to facilitate easy read/write and query. <br />In addition, I want the cluster to be able to query data from the database when loading the data into memory.</li>
  </ol>
  
  <p>I have an <em>Apache Kafka</em> system that can publish the feed for the processed input data. I can write a Go module to interface this into the database (via CURL, or a Go API if it exists). <br />
  There is already a development Spark cluster available to work with (assume that it can be scaled as necessary, if and when required).</p>
  
  <p>But I'm stuck on the choice of database. There are many solutions (here is a non-exhaustive <a href="http://www.erol.si/2015/01/the-complete-list-of-all-timeseries-databases-for-your-iot-project/" rel="nofollow noreferrer">list</a>) but I'm looking at <em>OpenTSDB</em>, <em>Druid</em> and <em>Axibase Time Series Database</em>. <br />
  Other time-series databases which I have looked at briefly, seem more as if they were optimised for handling metric data. (I have looked at InfluxDB, RiakTS and Prometheus)</p>
  
  <blockquote>
    <p>Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can
    access diverse data sources including HDFS, Cassandra, HBase, and S3. - <em>Apache Spark Website</em></p>
  </blockquote>
  
  <p>In addition, the time-series database should store that data in a fashion that <strong>exposes it directly to Spark</strong> (as this is time-series data, it should be <em>immutable and therefore satisfies the requirements of an RDD</em> - therefore, it can be loaded natively by Spark into the cluster). <br />
  Once the data (or data with dimensionality reduced by dropping categoric elements) is loaded, using sparklyr (a R interface for Spark) and Spark's machine learning library (MLib, <a href="http://spark.rstudio.com/images/sparklyr-cheatsheet.pdf" rel="nofollow noreferrer">this cheatsheet</a> provides a quick overview of the functionality), regression and classification model can be developed and experimented with.</p>
  
  <p>So, my questions:</p>
  
  <ol>
  <li>Does this seem like a reasonable approach to working with big data?</li>
  <li>Is my choice of database solutions correct? (I am set on working with columnar store and time-series database, please do not recommend SQL/Relational DBMS)</li>
  <li>If you have prior experience working with data analysis on clusters, from both an analytics and systems point of view (as I am doing both), do you have any advice/tips/tricks?</li>
  </ol>
  
  <p>Any help would be greatly appreciated.</p>