title: Create Spark DataFrame from Pandas DataFrame
body: <p>I'm trying to build a Spark DataFrame from a simple Pandas DataFrame. This are the steps I follow.</p>
  
  <pre class="lang-py prettyprint-override"><code>import pandas as pd
  pandas_df = pd.DataFrame({"Letters":["X", "Y", "Z"]})
  spark_df = sqlContext.createDataFrame(pandas_df)
  spark_df.printSchema()
  </code></pre>
  
  <p>Till' this point everything is OK. The output is:</p>
  
  <blockquote>
    <p>root<br>
    |-- Letters: string (nullable = true)</p>
  </blockquote>
  
  <p>The problem comes when I try to print the DataFrame:</p>
  
  <pre class="lang-py prettyprint-override"><code>spark_df.show()
  </code></pre>
  
  <p>This is the result:</p>
  
  <blockquote>
    <p>An error occurred while calling o158.collectToPython. :
    org.apache.spark.SparkException: Job aborted due to stage failure:
    Task 0 in stage 5.0 failed 1 times, most recent failure: Lost task 0.0
    in stage 5.0 (TID 5, localhost, executor driver):
    org.apache.spark.SparkException:<br>
    Error from python worker:<br>
    Error
    executing Jupyter command 'pyspark.daemon': [Errno 2] No such file or
    directory PYTHONPATH was:<br>
    /home/roldanx/soft/spark-2.4.0-bin-hadoop2.7/python/lib/pyspark.zip:/home/roldanx/soft/spark-2.4.0-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip:/home/roldanx/soft/spark-2.4.0-bin-hadoop2.7/jars/spark-core_2.11-2.4.0.jar:/home/roldanx/soft/spark-2.4.0-bin-hadoop2.7/python/lib/py4j-0.10.7-src.zip:/home/roldanx/soft/spark-2.4.0-bin-hadoop2.7/python/:
    org.apache.spark.SparkException: No port number in pyspark.daemon's
    stdout</p>
  </blockquote>
  
  <p><strong>These are my Spark specifications:</strong></p>
  
  <p>SparkSession - hive</p>
  
  <p>SparkContext</p>
  
  <p>Spark UI</p>
  
  <p>Version:
      v2.4.0</p>
  
  <p>Master:
      local[*]</p>
  
  <p>AppName:
      PySparkShell</p>
  
  <p><strong>This are my venv:</strong></p>
  
  <p>export PYSPARK_PYTHON=jupyter</p>
  
  <p>export PYSPARK_DRIVER_PYTHON_OPTS='lab'</p>
  
  <p><strong>Fact:</strong></p>
  
  <p>As the error mentions, it has to do with running pyspark from Jupyter. Running it with 'PYSPARK_PYTHON=python2.7' and 'PYSPARK_PYTHON=python3.6' works fine</p>