title: Pyspark Dataframe not returning all rows while converting to pandas using toPandas or Pyarrow function in Pyspark
body: <p>On trying to convert pyspark dataframe into pandas dataframe using arrow function only half the rows are getting converted. Pyspark df contains 170,000 rows.</p>
  
  <pre><code>&gt;&gt; spark.conf.set("spark.sql.execution.arrow.enabled", "true")
  &gt;&gt; result_pdf = train_set.select("*").toPandas()
  &gt;&gt; result_pdf returns only 65000 rows.
  </code></pre>
  
  <p>I tried to install and update pyarrow using following commands:</p>
  
  <pre><code>&gt;&gt; conda install -c conda-forge pyarrow
  &gt;&gt; pip install pyarrow
  &gt;&gt; pip install pyspark[sql]
  </code></pre>
  
  <p>and then run </p>
  
  <pre><code>&gt;&gt; spark.conf.set("spark.sql.execution.arrow.enabled", "true")
  &gt;&gt; result_pdf = train_set.select("*").toPandas()
  &gt;&gt;spark.conf.set("spark.sql.execution.arrow.enabled", "true")
  &gt;&gt; result_pdf = train_set.select("*").toPandas()
  </code></pre>
  
  <p>I am getting following warning message each time while conversion:</p>
  
  <blockquote>
    <p>C:\Users\MUM1342.conda\envs\snakes\lib\site-packages\pyarrow__init__.py:152:
    UserWarning: pyarrow.open_stream is deprecated, please use
    pyarrow.ipc.open_stream   warnings.warn("pyarrow.open_stream is
    deprecated, please use "
    C:\Users\MUM1342.conda\envs\snakes\lib\site-packages\pyspark\sql\dataframe.py:2138:
    UserWarning: toPandas attempted Arrow optimization because
    'spark.sql.execution.arrow.enabled' is set to true, but has reached
    the error below and can not continue. Note that
    'spark.sql.execution.arrow.fallback.enabled' does not have an effect
    on failures in the middle of computation.</p>
  </blockquote>
  
  <p>Actual Output:</p>
  
  <pre><code>&gt; train_set.count
  &gt; 170256
  &gt; result_pdf.shape
  &gt; 6500
  </code></pre>
  
  <p>Expected Output:</p>
  
  <pre><code>&gt; train_set.count
  &gt; 170256
  &gt; result_pdf.shape
  &gt; 170256
  </code></pre>