title: How to use RocksDB for HDFS?
body: <p>I'm writing a Spark application running on HDFS, the output is an RDD, which I have to save to RocksDB. But I don't know how to get RocksDB to work with HDFS and Spark. Please give me some instructions on how to set up RocksDB on HDFS and integrate it with Spark</p>
  
  <p>We have gone to the website 
  <a href="https://github.com/facebook/rocksdb/tree/master/hdfs" rel="nofollow noreferrer">https://github.com/facebook/rocksdb/tree/master/hdfs</a> but we cannot find hdfs.h and libhdfs so we don't know how to proceed. </p>
  
  <pre class="lang-scala prettyprint-override"><code>var sq = Seq[RDD[(String, Array[String])]]()
  for (file &lt;- files) {
       val filename = file.getPath.toString()
  
       val lines = sc.textFile(filename)
       val doc_id = filename.split("/").last.dropRight(4) 
       val key_value = lines.flatMap(_.split(" ")).map(word =&gt; (word, Array((doc_id)))).reduceByKey((a,b) =&gt; a)
       sq = sq :+ key_value
  } 
       val output = sc.union(sq)
       output.reduceByKey((a,b) =&gt; a ++ b) // this is the output RDD that we need to save to RocksDB
  
  </code></pre>