title: Can you construct pyspark.pandas.DataFrame from pyspark.sql.dataframe.DataFrame?
body: <p>I am new to Spark / Databricks. My question is whether is it recommended / possible to mix sql and Pandas API dataframes? Is it possible to create a <a href="https://spark.apache.org/docs/latest/api/python/reference/pyspark.pandas/frame.html" rel="nofollow noreferrer">pyspark.pandas.DataFrame</a> directly from a <a href="https://spark.apache.org/docs/latest/api/python/reference/api/pyspark.sql.DataFrame.html" rel="nofollow noreferrer">pyspark.sql.dataframe.DataFrame</a>, or I need to re-read the parquet file?</p>
  <pre class="lang-py prettyprint-override"><code># Suppose you have an SQL dataframe (now I read Boston Safety Data from Microsoft Open Dataset)
  blob_account_name = &quot;azureopendatastorage&quot;
  blob_container_name = &quot;citydatacontainer&quot;
  blob_relative_path = &quot;Safety/Release/city=Boston&quot;
  blob_sas_token = r&quot;&quot;
  
  wasbs_path = 'wasbs://%s@%s.blob.core.windows.net/%s' % (blob_container_name, blob_account_name, blob_relative_path)
  spark.conf.set('fs.azure.sas.%s.%s.blob.core.windows.net' % (blob_container_name, blob_account_name), blob_sas_token)
  print('Remote blob path: ' + wasbs_path)
  
  df = spark.read.parquet(wasbs_path)
  
  # Convert df to pyspark.pandas.Dataframe
  df2 =   # ...?
  </code></pre>
  <p>Tried <code>df.toPandas()</code>, that is not good, because it converts to plain, undistributed <code>pandas.core.frame.DataFrame</code>.</p>
  <p>A workaround is to read the parquet again into a <code>pyspark.pandas.Dataframe</code> which I try to avoid.</p>
  <p>Thanks!</p>