title: pyspark: ImportError: No module named numpy
body: <p>I am using pyspark, and got the result rdd from the following code:</p>
  
  <pre><code>import numpy
  model = PrefixSpan.train(input_rdd,minSupport=0.1)
  result = model.freqSequences().filter(lambda x: (x.freq &gt;= 50)).filter(lambda x: (len(x.sequence) &gt;=2) ).cache()
  </code></pre>
  
  <p>The <code>input_rdd</code> looks fine when I checked with <code>input_rdd.take(5)</code>. The above code created an rdd called 'result', which is of the following format:</p>
  
  <pre><code>PythonRDD[103] at RDD at PythonRDD.scala:48
  </code></pre>
  
  <p>I did have numpy installed, but when I tried to do <code>result.take(5)</code> or <code>result.count()</code>, I kept getting the following error below. </p>
  
  <pre><code>Py4JJavaErrorTraceback (most recent call last)
  &lt;ipython-input-32-7e589dce550c&gt; in &lt;module&gt;()
  ----&gt; 1 result.take(5)
  
  /usr/local/spark-latest/python/pyspark/rdd.py in take(self, num)
     1308 
     1309             p = range(partsScanned, min(partsScanned + numPartsToTry, totalParts))
  -&gt; 1310             res = self.context.runJob(self, takeUpToNumLeft, p)
     1311 
     1312             items += res
  
  /usr/local/spark-latest/python/pyspark/context.py in runJob(self, rdd, partitionFunc, partitions, allowLocal)
      939         # SparkContext#runJob.
      940         mappedRDD = rdd.mapPartitions(partitionFunc)
  --&gt; 941         port = self._jvm.PythonRDD.runJob(self._jsc.sc(), mappedRDD._jrdd, partitions)
      942         return list(_load_from_socket(port, mappedRDD._jrdd_deserializer))
      943 
  
  /usr/local/spark-latest/python/lib/py4j-0.10.1-src.zip/py4j/java_gateway.py in __call__(self, *args)
      931         answer = self.gateway_client.send_command(command)
      932         return_value = get_return_value(
  --&gt; 933             answer, self.gateway_client, self.target_id, self.name)
      934 
      935         for temp_arg in temp_args:
  
  /usr/local/spark-latest/python/pyspark/sql/utils.py in deco(*a, **kw)
       61     def deco(*a, **kw):
       62         try:
  ---&gt; 63             return f(*a, **kw)
       64         except py4j.protocol.Py4JJavaError as e:
       65             s = e.java_exception.toString()
  
  /usr/local/spark-latest/python/lib/py4j-0.10.1-src.zip/py4j/protocol.py in get_return_value(answer, gateway_client, target_id, name)
      310                 raise Py4JJavaError(
      311                     "An error occurred while calling {0}{1}{2}.\n".
  --&gt; 312                     format(target_id, ".", name), value)
      313             else:
      314                 raise Py4JError(
  
  Py4JJavaError: An error occurred while calling z:org.apache.spark.api.python.PythonRDD.runJob.
  : org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 42.0 failed 4 times, most recent failure: Lost task 0.3 in stage 42.0 (TID 85, ph-hdp-abc-dn07): org.apache.spark.api.python.PythonException: Traceback (most recent call last):
    File "/data/0/yarn/nm/usercache/abc-test/appcache/application_1482412711394_0011/container_e16_1482412711394_0011_01_000002/pyspark.zip/pyspark/worker.py", line 161, in main
      func, profiler, deserializer, serializer = read_command(pickleSer, infile)
    File "/data/0/yarn/nm/usercache/abc-test/appcache/application_1482412711394_0011/container_e16_1482412711394_0011_01_000002/pyspark.zip/pyspark/worker.py", line 54, in read_command
      command = serializer._read_with_length(file)
    File "/data/0/yarn/nm/usercache/abc-test/appcache/application_1482412711394_0011/container_e16_1482412711394_0011_01_000002/pyspark.zip/pyspark/serializers.py", line 164, in _read_with_length
      return self.loads(obj)
    File "/data/0/yarn/nm/usercache/abc-test/appcache/application_1482412711394_0011/container_e16_1482412711394_0011_01_000002/pyspark.zip/pyspark/serializers.py", line 422, in loads
      return pickle.loads(obj)
    File "/data/0/yarn/nm/usercache/abc-test/appcache/application_1482412711394_0011/container_e16_1482412711394_0011_01_000002/pyspark.zip/pyspark/mllib/__init__.py", line 28, in &lt;module&gt;
  ImportError: No module named numpy
  
      at org.apache.spark.api.python.PythonRunner$$anon$1.read(PythonRDD.scala:193)
      at org.apache.spark.api.python.PythonRunner$$anon$1.&lt;init&gt;(PythonRDD.scala:234)
      at org.apache.spark.api.python.PythonRunner.compute(PythonRDD.scala:152)
      at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
      at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:332)
      at org.apache.spark.rdd.RDD$$anonfun$8.apply(RDD.scala:330)
      at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:935)
      at org.apache.spark.storage.BlockManager$$anonfun$doPutIterator$1.apply(BlockManager.scala:910)
      at org.apache.spark.storage.BlockManager.doPut(BlockManager.scala:866)
      at org.apache.spark.storage.BlockManager.doPutIterator(BlockManager.scala:910)
      at org.apache.spark.storage.BlockManager.getOrElseUpdate(BlockManager.scala:668)
      at org.apache.spark.rdd.RDD.getOrCompute(RDD.scala:330)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:281)
      at org.apache.spark.api.python.PythonRDD.compute(PythonRDD.scala:63)
      at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:319)
      at org.apache.spark.rdd.RDD.iterator(RDD.scala:283)
      at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:70)
      at org.apache.spark.scheduler.Task.run(Task.scala:85)
      at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:274)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
  </code></pre>
  
  <hr>
  
  <p>Does anyone know what I missed? Thanks!</p>