I have a 20.2 GB dataset which is entirely consisting of 1 file. Need help on how to open the file Whenever I am dealing with datasets that have weird extensions that I have not encountered before, I typically open them with notepad to see how the data looks like before using pandas to analyze them. But this dataset (link), that has an extension of .tr (stands for trace files I suppose), is not supported by notepad due to its huge size (that's what the error message says).

Which software should I use to open the trace file? Also is there a safe way to segment the file into smaller parts, as I typically upload my data to kaggle and work on their kernel.

I am not sure if this is the right place to post, if not please point me towards the correct place before removing the post.
This file should be a plain text file with n rows and m columns separated by a tabulator. Here is another much smaller file which they offer for download and claim that it has the same data format.

Since the file is too big to fit in memory, you can use the data format HDF5. This will allow you to load slices of your dataset without having to load the whole file into memory. However, first, you need to convert the file to a .h5 file. This can be done with h5py. 

I put the following code together right now. It reads the text-file line by line and writes each line to the .h5-file. It works fine on the smaller file which they offer for download. I haven't tested it on a larger file, however. If you try it out, let me know how it goes.

import h5py
import numpy as np

N_COLS = 3  # adjust to your number of columns

n_rows_dataset = 1000

file = h5py.File('/path/to/your/output.h5', 'w')
dataset = file.create_dataset(
    'my_dataset', 
    (n_rows_dataset, N_COLS), 
    chunks=(5, N_COLS), 
    maxshape=(None, N_COLS))

current_row = 0
# this does not load the whole file but creates an iterator instead
for line in open("/path/to/the/large/file"):  
    if current_row == n_rows_dataset-1:
        n_rows_dataset = n_rows_dataset + 1000
        dataset.resize((n_rows_dataset, N_COLS))
    dset[current_row] = np.fromstring(line, sep=" ", dtype=np.float64)
    current_row += 1



Note: You can find out the number of colums of your dataset by printing the first lines of your file:

i = 0
for line in open("/path/to/the/large/file"):
    print(line)
    i += 1
    if i &gt;= 5:
        break