# HDF5 format for GPT dataloader#

## Expected Dataset Structure by the Static Dataloader#

The current HDF5 dataloader for GPT models expects the input dataset to meet certain structure criteria which are summarized below:

• The dataset files can be located in one directory or multiple directories passed as a list to the dataloader through the `data_dir` parameter. The files must have `.h5` extension (not `.hdf5` for example).

• Each HDF5 file must have an attribute `n_examples` that shows the number of input samples stored in the current HDF5 file.

• Each HDF5 file must have all input samples located in HDF5 dataset named data. The dataset data should have the dimensions `[n_examples, n_features, max_sequence_length]`. `n_features` must equal to 3 as it refers to the features `input_ids`, `attention_mask`, and `labels`. The features must be written and should be accessible in this exact order.

## Example (writing HDF5 file)#

```import h5py
import numpy as np

def write_hdf5_file():
n_examples = 50000
max_sequence_length = 2048
file_path = "/home/user/data_file.h5"
gpt_data = np.empty([n_examples, 3, max_sequence_length], dtype=np.int32)
with h5py.File(file_path, mode='w') as h5_file:
h5_file.attrs["n_examples"] = n_examples  # number of examples attribute
h5_file.create_dataset(
"data",                               # Name of dataset (must NOT change)
data=gpt_data,                        # Numpy array that contains data
dtype="i4",                           # Data type 32-bit int matches the data
chunks=(1, 3, max_sequence_length),   # chunk size equal to the data sample size
compression="gzip",                   # Enabling gzip compression
)
{"ft_lang_label":"__label__en","ft_lang_prob":0.7305833,"math_prob":0.6508138,"size":2654,"snap":"2023-40-2023-50","text_gpt3_token_len":625,"char_repetition_ratio":0.14603774,"word_repetition_ratio":0.027624309,"special_character_ratio":0.23247927,"punctuation_ratio":0.11007026,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.9563252,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2023-09-24T09:24:38Z\",\"WARC-Record-ID\":\"<urn:uuid:5461b622-752d-4a9c-aba7-c054088817c9>\",\"Content-Length\":\"19977\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:d75b3c01-b8ce-4df9-9b5f-be3afba8342f>\",\"WARC-Concurrent-To\":\"<urn:uuid:34d8e775-9e57-4d57-9b9f-ce9dc7cdf657>\",\"WARC-IP-Address\":\"104.18.0.163\",\"WARC-Target-URI\":\"https://docs.cerebras.net/en/latest/ai_model/content/hdf5_dataloader.html\",\"WARC-Payload-Digest\":\"sha1:FXO2DLEDRH23OPMEVQIISHLSRBPJX7CB\",\"WARC-Block-Digest\":\"sha1:FWM4BPMHL6U7OREBI6PHO4LW44YBQUT5\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2023/CC-MAIN-2023-40/CC-MAIN-2023-40_segments_1695233506632.31_warc_CC-MAIN-20230924091344-20230924121344-00331.warc.gz\"}"}