title: Using parquet-avro library in Java to read parquet file written using pyarrow
body: <p>I am writing a dataframe to parquet using pyarrow in python.</p>
  
  <pre><code>import pyarrow as pa
  import pyarrow.parquet as pq
  import pandas as pd
  df = pd.DataFrame(
          {
           "numbers": [1, 2, 3],
           "colors": ["red", "white", "blue"],
           "dates":['2019-12-16', '2019-12-16', '2019-12-16'],
           "codes": [None, None, None]
          }
      )
  table = pa.Table.from_pandas(df)
  pq.write_table(table, "filename")
  </code></pre>
  
  <p>Parquet file when read in Java (or in Sublime Text configured with parquet-tools) is:</p>
  
  <pre><code>{"numbers":1,"codes":"R1"}
  {"numbers":2,"codes":"G1"}
  {"numbers":3,"codes":"B1"}
  </code></pre>
  
  <p>The code I am using to read the parquet is this:</p>
  
  <pre><code>class Parquet {
    private List&lt;SimpleGroup&gt; data;
    private List&lt;Type&gt; schema;
  
    public Parquet(List&lt;SimpleGroup&gt; data, List&lt;Type&gt; schema) {
      this.data = data;
      this.schema = schema;
    }
  
    public List&lt;SimpleGroup&gt; getData() {
      return data;
    }
  
    public List&lt;Type&gt; getSchema() {
      return schema;
    }
  }
  
   public static Parquet getParquetData(String filePath) throws IOException {
      List&lt;SimpleGroup&gt; simpleGroups = new ArrayList&lt;&gt;();
      ParquetFileReader reader = ParquetFileReader.open(HadoopInputFile.fromPath(new Path(filePath), new Configuration()));
      MessageType schema = reader.getFooter().getFileMetaData().getSchema();
      List&lt;Type&gt; fields = schema.getFields();
      PageReadStore pages;
      while ((pages = reader.readNextRowGroup()) != null) {
        long rows = pages.getRowCount();
        MessageColumnIO columnIO = new ColumnIOFactory().getColumnIO(schema);
        RecordReader recordReader = columnIO.getRecordReader(pages, new GroupRecordConverter(schema));
  
        for (int i = 0; i &lt; rows; i++) {
          SimpleGroup simpleGroup = (SimpleGroup) recordReader.read();
          simpleGroups.add(simpleGroup);
        }
      }
      reader.close();
      return new Parquet(simpleGroups, fields);
    }
  }
  </code></pre>
  
  <p>While debugging I found that though schema has all the columns, in data we see only non-null columns.
  <a href="https://i.stack.imgur.com/8IgR7.png" rel="nofollow noreferrer"><img src="https://i.stack.imgur.com/8IgR7.png" alt="enter image description here"></a></p>
  
  <p>Has anybody seen this behavior? Is there any option in the parquet-avro library to not have this  'optimization'?</p>
  
  <p>Thanks</p>