title: How to get PySpark working on Google Cloud Dataproc cluster
body: <p>I have a series of questions (sorry Google documentation is awful and not-user friendly):</p>
  
  <ol>
  <li>What is the equivalent of Amazon EMR on Google Cloud, Dataproc? I'm using this documentation to run a Spark job: <a href="https://cloud.google.com/dataproc/docs/tutorials/gcs-connector-spark-tutorial" rel="nofollow noreferrer">https://cloud.google.com/dataproc/docs/tutorials/gcs-connector-spark-tutorial</a></li>
  <li>Can you ssh into the head machine and run a Spark in the entire cluster or you have use Google's <code>gcloud dataproc jobs submit ...</code> command?</li>
  <li>When I run a Spark job locally and try to access Google Cloud Storage I do so without a problem. When I try to use Dataproc it crashes.</li>
  </ol>
  
  <p>I have read:</p>
  
  <ul>
  <li><a href="https://cloud.google.com/dataproc/docs/concepts/connectors/install-storage-connector" rel="nofollow noreferrer">https://cloud.google.com/dataproc/docs/concepts/connectors/install-storage-connector</a></li>
  <li><a href="https://stackoverflow.com/questions/46176907/reading-google-bucket-data-in-spark">reading google bucket data in spark</a></li>
  <li><a href="https://stackoverflow.com/questions/27782844/no-filesystem-for-scheme-gs-when-running-spark-job-locally/27814118#27814118">&quot;No Filesystem for Scheme: gs&quot; when running spark job locally</a></li>
  </ul>
  
  <p>I have tried so far:</p>
  
  <ul>
  <li>I have placed <code>gcs-connector-hadoop2-latest.jar</code> and <code>my_project.json</code> on my master and worker nodes in <code>/etc/hadoop/conf</code></li>
  <li><p>I have added the following, on my master and worker nodes, to <code>/etc/hadoop/conf/core-site.xml</code>:
  </p>
  
  <pre><code>&lt;property&gt;
    &lt;name&gt;google.cloud.auth.service.account.enable&lt;/name&gt;
    &lt;value&gt;true&lt;/value&gt;
  &lt;/property&gt;
  &lt;property&gt;
    &lt;name&gt;my_project.json&lt;/name&gt;
    &lt;value&gt;full path to JSON keyfile downloaded for service account&lt;/value&gt;
  &lt;/property&gt;
  </code></pre></li>
  <li><p>I tried running the following commands:</p>
  
  <pre><code>sudo gcloud dataproc jobs submit pyspark  spark.py --cluster=${CLUSTER}
  </code></pre>
  
  <p>and</p>
  
  <pre><code>sudo gcloud dataproc jobs submit pyspark \
      --jars  /etc/hadoop/conf/gcs-connector-hadoop2-latest.jar \
      spark.py --cluster=${CLUSTER}
  </code></pre></li>
  <li>I keep getting the following error:
  
  <blockquote>
    <p>No FileSystem for scheme: gs</p>
  </blockquote></li>
  </ul>
  
  <p>I do not know what to do next.</p>