title: Pycharm: Java gateway process exited before sending its port number
body: <p>I'm trying to executing (with Pycharm) some <a href="https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html" rel="nofollow noreferrer">examples</a> in python using self contained sparks applications.</p>
  
  <p>I installed pyspark using:</p>
  
  <pre><code>pip install pyspark 
  </code></pre>
  
  <p>According to the web of the example, it should be just enough to execute it with:</p>
  
  <pre><code>python nameofthefile.py
  </code></pre>
  
  <p>But I have this error:</p>
  
  <pre><code>Exception in thread "main" java.lang.ExceptionInInitializerError
      at org.apache.hadoop.util.StringUtils.&lt;clinit&gt;(StringUtils.java:80)
      at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
      at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
      at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
      at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791)
      at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
      at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
      at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
      at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422)
      at org.apache.spark.SecurityManager.&lt;init&gt;(SecurityManager.scala:79)
      at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:359)
      at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$secMgr$1(SparkSubmit.scala:359)
      at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:367)
      at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:367)
      at scala.Option.map(Option.scala:146)
      at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:366)
      at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
      at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
      at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  Caused by: java.lang.StringIndexOutOfBoundsException: begin 0, end 3, length 2
      at java.base/java.lang.String.checkBoundsBeginEnd(String.java:3319)
      at java.base/java.lang.String.substring(String.java:1874)
      at org.apache.hadoop.util.Shell.&lt;clinit&gt;(Shell.java:52)
      ... 23 more
  Traceback (most recent call last):
    File "C:/Users/.../PycharmProjects/PoC/Databricks.py", line 4, in &lt;module&gt;
      spark = SparkSession.builder.appName("Databricks").getOrCreate()
    File "C:\Users\...\Desktop\env\lib\site-packages\pyspark\sql\session.py", line 173, in getOrCreate
      sc = SparkContext.getOrCreate(sparkConf)
    File "C:\Users\...\Desktop\env\lib\site-packages\pyspark\context.py", line 349, in getOrCreate
      SparkContext(conf=conf or SparkConf())
    File "C:\Users\...\Desktop\env\lib\site-packages\pyspark\context.py", line 115, in __init__
      SparkContext._ensure_initialized(self, gateway=gateway, conf=conf)
    File "C:\Users\...\Desktop\env\lib\site-packages\pyspark\context.py", line 298, in _ensure_initialized
      SparkContext._gateway = gateway or launch_gateway(conf)
    File "C:\Users\...\Desktop\env\lib\site-packages\pyspark\java_gateway.py", line 94, in launch_gateway
      raise Exception("Java gateway process exited before sending its port number")
  Exception: Java gateway process exited before sending its port number
  </code></pre>
  
  <p>What could be wrong?</p>
  
  <hr>
  
  <blockquote>
    <p>EXTRA</p>
  </blockquote>
  
  <p>According to the post where you can find the solution,for my case I had to change from jdk-11 to jdk1.8.</p>
  
  <p>Now I can run the example code, but with an error (which does not prevent it from running)</p>
  
  <pre><code>java.io.IOException: Could not locate executable null\bin\winutils.exe in the Hadoop binaries.
      at org.apache.hadoop.util.Shell.getQualifiedBinPath(Shell.java:379)
      at org.apache.hadoop.util.Shell.getWinUtilsPath(Shell.java:394)
      at org.apache.hadoop.util.Shell.&lt;clinit&gt;(Shell.java:387)
      at org.apache.hadoop.util.StringUtils.&lt;clinit&gt;(StringUtils.java:80)
      at org.apache.hadoop.security.SecurityUtil.getAuthenticationMethod(SecurityUtil.java:611)
      at org.apache.hadoop.security.UserGroupInformation.initialize(UserGroupInformation.java:273)
      at org.apache.hadoop.security.UserGroupInformation.ensureInitialized(UserGroupInformation.java:261)
      at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:791)
      at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
      at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
      at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
      at org.apache.spark.util.Utils$$anonfun$getCurrentUserName$1.apply(Utils.scala:2422)
      at scala.Option.getOrElse(Option.scala:121)
      at org.apache.spark.util.Utils$.getCurrentUserName(Utils.scala:2422)
      at org.apache.spark.SecurityManager.&lt;init&gt;(SecurityManager.scala:79)
      at org.apache.spark.deploy.SparkSubmit.secMgr$lzycompute$1(SparkSubmit.scala:359)
      at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$secMgr$1(SparkSubmit.scala:359)
      at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:367)
      at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:367)
      at scala.Option.map(Option.scala:146)
      at org.apache.spark.deploy.SparkSubmit.prepareSubmitEnvironment(SparkSubmit.scala:366)
      at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:143)
      at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
      at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924)
      at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933)
      at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
  2019-01-24 08:46:16 WARN  NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
  Setting default log level to "WARN".
  To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
  </code></pre>
  
  <p><a href="https://jaceklaskowski.gitbooks.io/mastering-apache-spark/spark-tips-and-tricks-running-spark-windows.html" rel="nofollow noreferrer">Here</a> is the solution of this <code>Could not locate executable null\bin\winutils.exe</code></p>
  
  <p>Resuming, to solve this second problem, you just need to define HADOOP_HOME and PATH environment variables in Control Panel so any Windows program would use them. </p>