Python Script: Converts .tsv to TFRecord The python code below converts a tab spaced values file (.tsv) into a TFRecord in hopes that this approach expedites future time spent loading the data. Is the performance of this task correct and optimal, specifically in terms of run-time?

The .tsv file is of the following format, including spaces to make it more readable (SEMEVAL STS'17 training dataset): float_value_from_0_to_5 \t string1 \t string2. The desired TFRecord's entries should optimally store the previous values as float rate, string sent1, and string sent2 for proper implementation with any arbitrary Neural Network that will embed the words once loaded from the TFRecord.

import tensorflow as tf
import nltk
import os
import csv

def tsv_to_tfrec(file_path):
  with open(file_path, 'rb') as tsv, tf.python_io.TFRecordWriter(
        os.path.splitext(file_path)[0] + ".tfrecords") as tfrec:
    read_tsv = csv.reader(tsv, delimiter = '\t', quoting = csv.QUOTE_NONE)

    for row in read_tsv:
      if len(row) != 3:
        print("\nERROR in # of indicies: ", len(row))
        for i in range(len(row)):
          print(i, " =", row[i])
      else:
        example = tf.train.Example(features = tf.train.Features(feature = {
            "rate": tf.train.Feature(float_list=tf.train.FloatList(
                value = [float(row[0])])),
            "sent1": tf.train.Feature(bytes_list=tf.train.BytesList(
                value = nltk.word_tokenize(row[1]))),
            "sent2": tf.train.Feature(bytes_list=tf.train.BytesList(
                value = nltk.word_tokenize(row[2])))
            }))

        tfrec.write(example.SerializeToString())
I am no expert in TensorFlow (even though started to grasp), but here are some things I noticed:


if this is a proper valid TSV file, you should not be worried about checking the length of each row - each row should have the same amount of columns
you may also try to use TextLineReader to read features from a csv file like noted here, this may lead to "unpacking" the features and avoiding getting them by index:

rate, sent1, sent2 = tf.decode_csv(value)

if, for some reason, TextLineReader is not applicable, you may unpack in the for loop:

for rate, sent1, sent2 in read_tsv:

you can also look into applying some micro-optimizations, like avoiding name lookups in the loop by pre-lookuping things before the loop - e.g. setting Example = tf.train.Example and then using Example in place of tf.train.Example
there is also a possibility of that classic trading off memory for the speed. Depending on how unique your features are, you, for instance, can "cache" the word tokenization results in a dictionary to avoid re-tokenizing the same sentences over again


Also check out the TensorFlow Performance Guide and Performance Tips.

There are also some minor PEP8 code style violations, like extra spaces around = in the keyword argument definitions - run your code through a static code analyzer like flake8 or pylint.