title: Converting TFRecords and tf.Examples to commonly used data types
body: <p>I am learning to create TensorFlow Extended pipelines and find them greatly useful.
  However, I have yet to figure out how to debug and test (tabular) data passing through these pipelines. I am aware TensorFlow uses TFRecords/tf.Examples, which are protobufs.</p>
  <p>These able humanly readable by using a <code>TFRecordDataset</code> and tf.Example's <code>parseFromString</code>.
  Still, this format is hard to read.</p>
  <p>How would one actually test the data? I feel like I'd need a pandas dataframe. And since we have 100+ columns and different use-cases, I can hardly define all columns each time I'd like to do this. Can I somehow use my schema for this? Thanks!</p>
  <p><strong>Edit</strong>: I'll accept @TheEngineer's answer, as it gave me the pivotal hint on how to achieve what I wanted. Still, I'd like to share <strong>my solution</strong>.</p>
  <p>Disclaimer: I'm using this code just for testing and to see what's going on within my pipeline. Be careful using this code in production. There might be better and safer ways.</p>
  <pre><code>import sys 
  import numpy as np
  import tensorflow_data_validation as tfdv 
  
  # Our default values for missing values within the tfrecord. We'll restore them later
  STR_NA_VALUE = &quot;NA&quot;
  INT_NA_VALUE = -sys.maxsize - 1
  FLOAT_NA_VALUE = float(&quot;nan&quot;)
  
  # Create a dict containing FixedLenFeatures using our schema
  def load_schema_as_feature_dict(schema_path):
      schema = tfdv.load_schema_text(schema_path)
  
      def convert_feature(feature):
          if feature.type == 1:
              return tf.io.FixedLenFeature((), tf.string, STR_NA_VALUE)
          if feature.type == 2:
              return tf.io.FixedLenFeature((), tf.int64, INT_NA_VALUE)
          if feature.type == 3:
              return tf.io.FixedLenFeature((), tf.float32, FLOAT_NA_VALUE)
          raise ValueError(&quot;Non-implemented type {}&quot;.format(feature.type))
  
      return dict((feature.name, convert_feature(feature)) for feature in schema.feature)  
  
  def as_pandas_frame(tfrecord_path, schema_path):
      feature_dict = load_schema_as_feature_dict(schema_path)
      dataset = tf.data.TFRecordDataset(tfrecord_path, compression_type=&quot;GZIP&quot;)
      parsed_dataset = dataset.map(lambda serialized_example: tf.io.parse_single_example(serialized_example, feature_dict))
      df = pd.DataFrame(list(parsed_dataset.as_numpy_iterator()))
      
      # Restore NA values from default_values we had to set
      for key, value in {np.object: str.encode(STR_NA_VALUE), np.int64: INT_NA_VALUE, np.float: FLOAT_NA_VALUE}.items():
          type_columns = df.select_dtypes(include=[key]).columns
          df[type_columns] = df[type_columns].replace({value:None})
      
      return df
  </code></pre>
  <p>Now, you'll just need to call this function with your stored tfrecords and your schema.pbtxt file:</p>
  <pre><code>df = as_pandas_frame(&quot;path/to/your/tfrecord.gz&quot;, &quot;path/to/your/schema.pbtxt&quot;)
  </code></pre>