title: How to run python3 on google's dataproc pyspark
body: <p>I want to run a pyspark job through Google Cloud Platform dataproc, but I can't figure out how to setup pyspark to run python3 instead of 2.7 by default.</p>
  
  <p>The best I've been able to find is adding these <a href="https://github.com/GoogleCloudPlatform/dataproc-initialization-actions/issues/25" rel="noreferrer">initialization commands</a></p>
  
  <p>However, when I ssh into the cluster then<br>
  (a) <code>python</code> command is still python2,<br>
  (b) my job fails due to a python 2 incompatibility. </p>
  
  <p>I've tried uninstalling python2 and also aliasing <code>alias python='python3'</code> in my init.sh script, but alas, no success.  The alias doesn't seem to stick.</p>
  
  <p>I create the cluster like this</p>
  
  <pre><code>cluster_config = {
      "projectId": self.project_id,
      "clusterName": cluster_name,
      "config": {
          "gceClusterConfig": gce_cluster_config,
          "masterConfig": master_config,
          "workerConfig": worker_config,
          "initializationActions": [
              [{
              "executableFile": executable_file_uri,
              "executionTimeout": execution_timeout,
          }]
          ],
      }
  }
  
  credentials = GoogleCredentials.get_application_default()
  api = build('dataproc', 'v1', credentials=credentials)
  
  response = api.projects().regions().clusters().create(
      projectId=self.project_id,
      region=self.region, body=cluster_config
  ).execute()
  </code></pre>
  
  <p>My <code>executable_file_uri</code> is sits on google storage; <code>init.sh</code>:</p>
  
  <pre><code>apt-get -y update
  apt-get install -y python-dev
  wget -O /root/get-pip.py https://bootstrap.pypa.io/get-pip.py
  python /root/get-pip.py
  apt-get install -y python-pip
  pip install --upgrade pip
  pip install --upgrade six
  pip install --upgrade gcloud
  pip install --upgrade requests
  pip install numpy
  </code></pre>