title: Getting Spark, Python, and MongoDB to work together
body: <p>I'm having difficulty getting these components to knit together properly. I have Spark installed and working successfully, I can run jobs locally, standalone, and also via YARN. I have followed the steps advised (to the best of my knowledge) <a href="https://github.com/mongodb/mongo-hadoop/wiki/Spark-Usage" rel="nofollow noreferrer">here</a> and <a href="https://github.com/mongodb/mongo-hadoop/blob/master/spark/src/main/python/README.rst" rel="nofollow noreferrer">here</a></p>
  <p>I'm working on Ubuntu and the various component versions I have are</p>
  <ul>
  <li><strong>Spark</strong> spark-1.5.1-bin-hadoop2.6</li>
  <li><strong>Hadoop</strong> hadoop-2.6.1</li>
  <li><strong>Mongo</strong> 2.6.10</li>
  <li><strong>Mongo-Hadoop connector</strong> cloned from <a href="https://github.com/mongodb/mongo-hadoop.git" rel="nofollow noreferrer">https://github.com/mongodb/mongo-hadoop.git</a></li>
  <li><strong>Python</strong> 2.7.10</li>
  </ul>
  <p>I had some difficulty following the various steps such as which jars to add to which path, so what I have added are</p>
  <ul>
  <li>in <code>/usr/local/share/hadoop-2.6.1/share/hadoop/mapreduce</code> <strong>I have added</strong> <code>mongo-hadoop-core-1.5.0-SNAPSHOT.jar</code></li>
  <li>the following <strong>environment variables</strong>
  <ul>
  <li><code>export HADOOP_HOME=&quot;/usr/local/share/hadoop-2.6.1&quot;</code></li>
  <li><code>export PATH=$PATH:$HADOOP_HOME/bin</code></li>
  <li><code>export SPARK_HOME=&quot;/usr/local/share/spark-1.5.1-bin-hadoop2.6&quot;</code></li>
  <li><code>export PYTHONPATH=&quot;/usr/local/share/mongo-hadoop/spark/src/main/python&quot;</code></li>
  <li><code>export PATH=$PATH:$SPARK_HOME/bin</code></li>
  </ul>
  </li>
  </ul>
  <p>My Python program is basic</p>
  <pre class="lang-python prettyprint-override"><code>from pyspark import SparkContext, SparkConf
  import pymongo_spark
  pymongo_spark.activate()
  
  def main():
      conf = SparkConf().setAppName(&quot;pyspark test&quot;)
      sc = SparkContext(conf=conf)
      rdd = sc.mongoRDD(
          'mongodb://username:password@localhost:27017/mydb.mycollection')
  
  if __name__ == '__main__':
      main()
  </code></pre>
  <p>I am running it using the command</p>
  <pre><code>$SPARK_HOME/bin/spark-submit --driver-class-path /usr/local/share/mongo-hadoop/spark/build/libs/ --master local[4] ~/sparkPythonExample/SparkPythonExample.py
  </code></pre>
  <p>and I am getting the following output as a result</p>
  <pre><code>Traceback (most recent call last):
    File &quot;/home/me/sparkPythonExample/SparkPythonExample.py&quot;, line 24, in &lt;module&gt;
      main()
    File &quot;/home/me/sparkPythonExample/SparkPythonExample.py&quot;, line 17, in main
      rdd = sc.mongoRDD('mongodb://username:password@localhost:27017/mydb.mycollection')
    File &quot;/usr/local/share/mongo-hadoop/spark/src/main/python/pymongo_spark.py&quot;, line 161, in mongoRDD
      return self.mongoPairRDD(connection_string, config).values()
    File &quot;/usr/local/share/mongo-hadoop/spark/src/main/python/pymongo_spark.py&quot;, line 143, in mongoPairRDD
      _ensure_pickles(self)
    File &quot;/usr/local/share/mongo-hadoop/spark/src/main/python/pymongo_spark.py&quot;, line 80, in _ensure_pickles
      orig_tb)
  py4j.protocol.Py4JError
  </code></pre>
  <p>According to <a href="https://github.com/bartdag/py4j/blob/master/py4j-web/advanced_topics.rst#id19" rel="nofollow noreferrer">here</a></p>
  <blockquote>
  <p>This exception is raised when an exception occurs in the Java client
  code. For example, if you try to pop an element from an empty stack.
  The instance of the Java exception thrown is stored in the
  java_exception member.</p>
  </blockquote>
  <p>Looking at the source code for <code>pymongo_spark.py</code> and the line throwing the error, it says</p>
  <blockquote>
  <p>&quot;Error while communicating with the JVM. Is the MongoDB Spark jar on
  Spark's CLASSPATH? : &quot;</p>
  </blockquote>
  <p>So in response, I have tried to be sure the right jars are being passed, but I might be doing this all wrong, see below</p>
  <pre><code>$SPARK_HOME/bin/spark-submit --jars /usr/local/share/spark-1.5.1-bin-hadoop2.6/lib/mongo-hadoop-spark-1.5.0-SNAPSHOT.jar,/usr/local/share/spark-1.5.1-bin-hadoop2.6/lib/mongo-java-driver-3.0.4.jar --driver-class-path /usr/local/share/spark-1.5.1-bin-hadoop2.6/lib/mongo-java-driver-3.0.4.jar,/usr/local/share/spark-1.5.1-bin-hadoop2.6/lib/mongo-hadoop-spark-1.5.0-SNAPSHOT.jar --master local[4] ~/sparkPythonExample/SparkPythonExample.py
  </code></pre>
  <p>I have imported <code>pymongo</code> to the same python program to verify that I can at least access MongoDB using that, and I can.</p>
  <p>I know there are quite a few moving parts here so if I can provide any more useful information please let me know.</p>