title: databricks-connect not working on windows
body: <p>I've installed <code>databricks-connect</code> on Windows 10 with the instructions here: <a href="https://docs.databricks.com/dev-tools/databricks-connect.html" rel="nofollow noreferrer">https://docs.databricks.com/dev-tools/databricks-connect.html</a></p>
  <p>After running <code>databricks-connect configure</code> and entering all values, i'm running <code>databricks-connect test</code>. This is the output I'm getting, and it hangs:</p>
  <pre><code>* PySpark is installed at c:\users\user\.conda\envs\myenv\lib\site-packages\pyspark
  * Checking SPARK_HOME
  * Checking java version
  java version &quot;1.8.0_251&quot;
  Java(TM) SE Runtime Environment (build 1.8.0_251-b08)
  Java HotSpot(TM) 64-Bit Server VM (build 25.251-b08, mixed mode)
  * Skipping scala command test on Windows
  * Testing python command
  The system cannot find the path specified.
  </code></pre>
  <p>Digging a bit deeper, it seems that the underlying <code>pyspark</code> package fails to initialize. It fails on this line:</p>
  <pre class="lang-py prettyprint-override"><code>from pyspark.sql import SparkSession
  spark = SparkSession.builder.getOrCreate()
  </code></pre>
  <p>When I try to run this manually, it hangs. I guess this is a problem is either the local Spark or the required Hadoop (and winutils.exe) installations, but <code>databricks-connect</code> requires a fresh pyspark installation (doc says to uninstall pyspark prior to installation).</p>
  <p>Would be happy for any references for:</p>
  <ol>
  <li>Fixing the databricks-connect issue</li>
  <li>Fixing the underlying pyspark installation issue</li>
  </ol>