Split big file into separate files depending on column's distinct values I have a big file, which contains column names in the first row. The data is about 20GB and consists of 10s of millions of rows. Every other row contains data which I would like to append to other files depending on the distinct entries in a particular column (depending on index). When the program encounters a new distinct entry it creates the file other wise it should append the data. The following piece of code is my basic attempt to achieve this (I am Python newbie):

import re
import os

input_path_file1 = 'C:\InputData.txt'
output_path = r'C:\Output\Data'
unique_values = {}

# read in first line containing column names
with open(input_path_file1, 'r') as file:
    first_line = file.readline()

rows = 0
# 7 line_number
# 35 range
index = 35
with open(input_path_file1, mode='r') as file:

    # skip first row
    next(file)

    for line in file:
        splitted_data = re.split(r'\t+', line.rstrip('\t'))
        cell_data = splitted_data[index].replace('/', '')
        target = os.path.join(output_path, cell_data + '.txt')

        if cell_data in unique_values:
            unique_values[cell_data] += 1
        else:
            unique_values[cell_data] = 1
            with open(target, 'w') as outfile:
                outfile.write(first_line)

        with open(target, 'a') as outfile:
            outfile.write(line)

        rows += 1


Could this be made more efficient?
It would be more efficient to open each file only once,
instead of reopening every time you need it.
Two obvious approaches come to mind:


Accumulate all the lines you want to write to files into a dictionary of lists, and then write to the files one by one. Given that your content is very large, this alternative may consume too much memory and therefore not suitable for your case.
Keep a dictionary of open file handles to write to.
If the lines are ordered in such a way that the lines that should go to the same file are grouped together, then you don't even need a dictionary, you could just keep track of the previous filename, so that if the filename is the same then append to the currently open file, otherwise close the current file and open a new one.


And most certainly you don't want to open input_path_file1 twice to read the first line and then again to process the rest of the file.
Opening it once would suffice, and instead of next(file) to skip the first line,
you could store it with first_line = next(file).

The variable rows is not used, so it could be removed.

The variable unique_values is a dictionary of counts,
but the counts are not used for anything.
So this could have been a set instead of a dict.