title: Could not find valid SPARK_HOME on dataproc
body: <p>Spark job executed by Dataproc cluster on Google Cloud gets stuck on a task <code>PythonRDD.scala:446</code></p>
  
  <p>The error log says <code>Could not find valid SPARK_HOME while searching</code> ... paths under <code>/hadoop/yarn/nm-local-dir/usercache/root/</code></p>
  
  <p>The thing is, SPARK_HOME should be set by default on a dataproc cluster.
  Other spark jobs that don't use RDDs work just fine.</p>
  
  <p>During the initialization of the cluster I do not reinstall spark (but I have tried to, which I previously thought caused the issue).</p>
  
  <p>I also found out that all my executors were removed after a minute of running the task.</p>
  
  <p>And yes, I have tried to run the following initialization action and it didn't help:</p>
  
  <pre><code>#!/bin/bash
  
  cat &lt;&lt; EOF | tee -a /etc/profile.d/custom_env.sh /etc/*bashrc &gt;/dev/null
  export SPARK_HOME=/usr/lib/spark/
  EOF
  </code></pre>
  
  <p>Any help?</p>