title: pandas data types changed when reading from parquet file?
body: <p>I am brand new to pandas and the parquet file type. I have a python script that:</p>
  <ol>
  <li>reads in a hdfs parquet file</li>
  <li>converts it to a pandas dataframe</li>
  <li>loops through specific columns and changes some values</li>
  <li>writes the dataframe back to a parquet file</li>
  </ol>
  <p>Then the parquet file is imported back into hdfs using impala-shell.</p>
  <p>The issue I'm having appears to be with step 2. I have it print out the contents of the dataframe immediately after it reads it in and before any changes are made in step 3. It appears to be changing the datatypes and the data of some fields, which causes problems when it writes it back to a parquet file. Examples:</p>
  <ul>
  <li>fields that show up as <em>NULL</em> in the database are replaced with the <em>string</em> &quot;None&quot; (for string columns) or the <em>string</em> &quot;nan&quot; (for numeric columns) in the printout of the dataframe.</li>
  <li>fields that should be an Int with a value of 0 in the database are changed to &quot;0.00000&quot; and turned into a float in the dataframe.</li>
  </ul>
  <p>It appears that it is actually changing these values, because when it writes the parquet file and I import it into hdfs and run a query, I get errors like this:</p>
  <pre><code>WARNINGS: File '&lt;path&gt;/test.parquet' has an incompatible Parquet schema for column 
  '&lt;database&gt;.&lt;table&gt;.tport'. Column type: INT, Parquet schema:
  optional double tport [i:1 d:1 r:0]
  </code></pre>
  <p>I don't know why it would alter the data and not just leave it as-is. If this is what's happening, I don't know if I need to loop over every column and replace all these back to their original values, or if there is some other way to tell it to leave them alone.</p>
  <p>I have been using this reference page:
  <a href="http://arrow.apache.org/docs/python/parquet.html" rel="nofollow noreferrer">http://arrow.apache.org/docs/python/parquet.html</a></p>
  <p>It uses</p>
  <pre><code>pq.read_table(in_file) 
  </code></pre>
  <p>to read the parquet file and then</p>
  <pre><code>df = table2.to_pandas()
  </code></pre>
  <p>to convert to a dataframe that I can loop through and change the columns. I don't understand why it's changing the data, and I can't find a way to prevent this from happening. Is there a different way I need to read it than read_table?</p>
  <p>If I query the database, the data would look like this:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: right;">tport</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: right;">0</td>
  </tr>
  <tr>
  <td style="text-align: right;"></td>
  </tr>
  <tr>
  <td style="text-align: right;"></td>
  </tr>
  <tr>
  <td style="text-align: right;">1</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>My print(df) line for the same thing looks like this:</p>
  <div class="s-table-container">
  <table class="s-table">
  <thead>
  <tr>
  <th style="text-align: right;">tport</th>
  </tr>
  </thead>
  <tbody>
  <tr>
  <td style="text-align: right;">0.00000</td>
  </tr>
  <tr>
  <td style="text-align: right;">nan</td>
  </tr>
  <tr>
  <td style="text-align: right;">nan</td>
  </tr>
  <tr>
  <td style="text-align: right;">1.00000</td>
  </tr>
  </tbody>
  </table>
  </div>
  <p>Here is the relevant code. I left out the part that processes the command-line arguments since it was long and it doesn't apply to this problem. The file passed in is <em>in_file</em>:</p>
  <pre><code>import sys, getopt
  import random
  import re
  import math
  
  import pyarrow.parquet as pq
  import numpy as np
  import pandas as pd
  import pyarrow as pa
  import os.path
  
  # &lt;CLI PROCESSING SECTION HERE&gt;
  
  # GET LIST OF COLUMNS THAT MUST BE SCRAMBLED
  field_file = open('scrambler_columns.txt', 'r') 
  contents = field_file.read()
  scrambler_columns = contents.split('\n')
  
  def scramble_str(xstr):
      #print(xstr + '_scrambled!')
      return xstr + '_scrambled!'
  
  parquet_file = pq.ParquetFile(in_file)
  table2 = pq.read_table(in_file)
  metadata = pq.read_metadata(in_file)
  
  df = table2.to_pandas() #dataframe
  
  print('rows: ' + str(df.shape[0]))
  print('cols: ' + str(df.shape[1]))
  
  pd.set_option('display.max_rows', None)
  pd.set_option('display.max_columns', None)
  pd.set_option('display.width', None)
  pd.set_option('display.max_colwidth', -1)
  pd.set_option('display.float_format', lambda x: '%.5f' % x)
  
  #df.fillna(value='', inplace=True) # np.nan # \xa0 
  
  print(df) # print before making any changes
  
  
  cols = list(df)
  # https://pythonbasics.org/pandas-iterate-dataframe/
  for col_name, col_data in df.iteritems():
      #print(cols[index])
      if col_name in scrambler_columns:
          print('scrambling values in column '  + col_name)
  
          for i, val in col_data.items():
              df.at[i, col_name] = scramble_str(str(val))
  
          
      
  print(df) # print after making changes
  
  print(parquet_file.num_row_groups)
  print(parquet_file.read_row_group(0))
  
  # WRITE NEW PARQUET FILE
  new_table = pa.Table.from_pandas(df)
  writer = pq.ParquetWriter(out_file, new_table.schema)
  for i in range(1):
      writer.write_table(new_table)
  
  writer.close()
      
  if os.path.isfile(out_file) == True:
      print('wrote ' + out_file)
  else:
      print('error writing file ' + out_file)
  
  # READ NEW PARQUET FILE
  table3 = pq.read_table(out_file)
  df = table3.to_pandas() #dataframe
  print(df)
  </code></pre>
  <p><strong>EDIT</strong>
  Here are the datatypes for the 1st few columns in hdfs
  <a href="https://i.stack.imgur.com/H8M3R.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/H8M3R.png" alt="enter image description here" /></a></p>
  <p>and here are the same ones that are in the pandas dataframe:</p>
  <pre><code>id         object 
  col1       float64
  col2       object 
  col3       object 
  col4       float64
  col5       object 
  col6       object 
  col7       object 
  </code></pre>
  <p>It appears to convert</p>
  <pre><code>String to object
  Int    to float64
  bigint to float64
  </code></pre>
  <p>How can I tell pandas what data types the columns should be?</p>
  <p><strong>Edit 2:</strong> I was able to find a workaround by directly processing the pyarrow tables. Please see my question and answers here: <a href="https://stackoverflow.com/questions/65845694/how-to-update-data-in-pyarrow-table/65885612#65885612">How to update data in pyarrow table?</a></p>