Support/Feature Request: Pre-processing very large corpus text file as tokens to train GPT Models.
Hi @abhaskumarsinha ,

If the text data is in a directory then you can use the API `tf.keras.utils.text_dataset_from_directory` to generate a tf.data.Dataset object from text files in the directory. Please refer the API [documentation](https://www.tensorflow.org/api_docs/python/tf/keras/utils/text_dataset_from_directory) for more details.

You can also refer this documentation [guide](https://www.tensorflow.org/guide/data?_gl=1*1orjui2*_ga*MTM4Mzg1ODAxNi4xNjY1OTkwMTQ1*_ga_W0YLR4190T*MTY4MzYxMTczMi44ODMuMS4xNjgzNjEzMjc0LjAuMC4w#consuming_text_data) on how to consume text data.

Please refer to an end to end tutorial for handling text data [here](https://www.tensorflow.org/tutorials/load_data/text).

Please go through and let us know if this is helpful. Thanks! Hello @SuryanarayanaY 

Thank you for your response.

I've already checked all of them before and I don't see how they would be helpful to me in this case.
1.  `tf.keras.utils.text_dataset_from_directory` loads the text from different files. The output is an array of strings in the order of text. This is typically in contrast to my requirement - where I require output in the form of an array of tokens (or words, separated with a spacebar), not a sentence. For example: `[b'Hello', b'World!', b'I', b'love', b'TensorFlow']` and then pre-process them back to sentence and token (as in the example in my first post).
2. The second guide link on consuming text data is similar. It processes the data in the form of sentences and not tokens.
3. I've also checked the handling text data tutorial before. The goal of the preprocessing I require is fundamentally different from the one discussed in the tutorial.

I also noticed `tf.strings.split` method to split the strings and map each sentence to one token, **this is again different from my requirement, where I actually require a flattened version of the tf.strings.split output** in the form of an array [1, m].

One way I can think to solve the issue is to write a generator function in Python that yields a pre-processed tokenized array of text tokens in the required array by reading a file in fragments each time and moving to the next iterator corresponding to the next fragment of the file, and use this generator with `tf.data`. But again, Python natively offers no such feature to **read files in the fragment of tokens** (but rather bytes, which I don't require). Hello @SuryanarayanaY 

The best I've got to do till now is this code:

```
dataset = './dataset/output_dataset.txt'

data = tf.data.TextLineDataset(dataset)

def split(string):
    return tf.strings.split(string, sep=' ')

def filter_empty_string(string):
    if tf.strings.length(tf.strings.reduce_join(string)) == 0:
        return False
    else:
        return True

data.map(split).filter(filter_empty_string)

for i in data.map(split).filter(filter_empty_string).take(10):
    print(i.numpy())
```


which produces an output like this:
```
[b'***' b'START' b'OF' b'THIS' b'PROJECT' b'GUTENBERG' b'EBOOK' b'COM@@'
 b'ING' b'AT@@' b'TRA@@' b'C@@' b'TION' b'***']
[b'Produced' b'by' b'Greg' b'Weeks,' b'Mary' b'Me@@' b'ehan' b'and' b'the'
 b'Online']
[b'Distributed' b'Proofreading' b'Team' b'at' b'http://www.pgdp.net']
[b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b''
 b'' b'' b'' b'' b'' b'' b'' b'' b'' b'Coming' b'At@@' b'trac@@' b'tion']
[b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b''
 b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'BY' b'F@@' b'R@@' b'IT@@' b'Z'
 b'LE@@' b'I@@' b'B@@' b'ER']
[b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b''
 b'' b'' b'' b'' b'' b'Illustr@@' b'ated' b'by' b'Paul' b'C@@' b'alle']
[b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'[T@@' b'ran@@' b"scriber's"
 b'Note:' b'This' b'et@@' b'ext' b'was' b'produced' b'from']
[b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b'' b''
 b'Gal@@' b'ax@@' b'y' b'Sci@@' b'ence' b'F@@' b'ic@@' b'tion' b'November'
 b'19@@' b'50@@' b'.']
[b'' b'' b'' b'' b'' b'' b'' b'' b'' b'Extensive' b'research' b'did'
 b'not' b'uncover' b'any' b'evidence' b'that']
[b'' b'' b'' b'' b'' b'' b'' b'' b'' b'the' b'U.S.' b'copyright' b'on'
 b'this' b'publication' b'was' b'rene@@' b'we@@' b'd.]']
```

This is far, far from what I need, that you can see in the example I've provided in the very first post. Somewhat similar to this:
```
[b'*** START OF' ], [b'THIS'] 
[b'START OF THIS'], [b'PROJECT']
[b'OF THIS PROJECT'], [b'GUTENBERG'] 
...
```

for `gpt_input = 3` Hi @abhaskumarsinha ,

For data preprocessing of `tf.data.Dataset` outputs you need to use `pre-processing` layers.

For text vectorization TF have `tf.keras.layers.TextVectorization` layer and it has some built-in methods which may be helpful. Please refer the documentation [source](https://www.tensorflow.org/guide/keras/preprocessing_layers#text_preprocessing) for more details.

Thanks! Hello @SuryanarayanaY 

`tf.keras.layers.TextVectorization` has methods to deal with pre-processing **texts** into the matrices of numbers and arrays. But, for casting the text into numbers of the fixed array, I must get them in the desired format (or order), this is what `TextVectorization` doesn't seem to offer here.

Thank you. Hello @SuryanarayanaY 

Here's my latest effort to help me load dataset using `tf.data.Dataset.from_generator`:

```
starting_chunk = 5
ending_chunk = 10
chunk_size = 1024

def read_file(f, vectorizer, chunk_size = 1024, starting_chunk = 0, ending_chunk = 2, gpt_input = 10):
    i = 0
    chunk = []
    
    while True:
        data = f.read(chunk_size)
        
        if not data or i > ending_chunk:
            break
        
        if i >= starting_chunk and i <= ending_chunk:
            file_contents = data.split()
            input_tokens, output_tokens = [], []
            for i in range(len(file_contents) - gpt_input - 1):
                input_tokens += [file_contents[i : i + gpt_input]]
                output_tokens += [file_contents[i + gpt_input]]
               
            
            X = [' '.join(input_tokens[i]) for i in range(len(input_tokens))]
            Y = output_tokens
            
            X = vectorizer(X)
            Y = vectorizer(Y)
            
            output = tf.concat([X, Y], 1)
        
            yield output
        
        i += 1
```
        
and using the generator above with `tf.data.Dataset.from.generator`:

```
dataset = tf.data.Dataset.from_generator(
    read_file,
    args = (tf.io.read_file("./dataset/output_dataset.txt"), vectorizer),
    output_signature = tf.TensorSpec(shape = (None, None, None), dtype = tf.int64)
    )
```

Nevertheless, I get greeted with an error:

`ValueError: Attempt to convert a value (<keras.layers.preprocessing.text_vectorization.TextVectorization object at 0x0000016EC7E3F370>) with an unsupported type (<class 'keras.layers.preprocessing.text_vectorization.TextVectorization'>) to a Tensor.`


It sounds like it is almost impossible to do it using TensorFlow library functions...

 Thank You.
<details><summary>Click to expand!</summary> 
 
 ### Issue Type

Feature Request

### Have you reproduced the bug with TF nightly?

No

### Source

binary

### Tensorflow Version

v2.9.0-18-gd8ce9f9c301 2.9.1

### Custom Code

Yes

### OS Platform and Distribution

Windows 11

### Mobile device

_No response_

### Python version

3.9.5

### Bazel version

_No response_

### GCC/Compiler version

_No response_

### CUDA/cuDNN version

_No response_

### GPU model and memory

_No response_

### Current Behaviour?

Suppose I've a very simple Python code like this:


```
corpus = file.read()
            file_contents = corpus.split()[token_start : token_end]


input_tokens, output_tokens = [], []
        for i in tqdm(range(len(file_contents) - gpt_input - 1)):
            input_tokens += [file_contents[i : i + gpt_input]]
            output_tokens += [file_contents[i + gpt_input]]
               
            
        X = [' '.join(input_tokens[i]) for i in tqdm(range(len(input_tokens)))]
        Y = output_tokens
```
The code does three things:

1. Load a file into RAM, split the contents of the file into words, i.e. - we have a list of words from the file in the order of the sentences.
2. Next, use two variables - input_tokens, output_tokens as list and append list of first `gpt_input` words in input_token and `gpt-input`-th word in output_token. This ensures that we have all `i` to `i + gpt_input` words in input_tokens and `i + 1` tokens in output_tokens, for all i = 0 to i = `total_tokens - 1`.
3. Now, we reconstruct sentences with words input_tokens, i.e. - we condensate gpt_input words back to the sentences.

Example:

If the file has contents  like this:
```
Hello World, I'm writing a new cool code in TensorFlow, please don't forget to check it!
```

The end result:
input_tokens for gpt_input = 3:
```
Hello World, I'm
World, I'm writing
I'm writing a
writing a new
a new cool
...
```
output_tokens for gpt_input = 3:
```
writing
a
new
cool
code
...
```

So, now the problem is - the file or the text corpus which is needed to train a GPT Model can be very large! like upto - 200-300 GB and can't be loaded into RAM/memory directly. So, TensorFlow offers - tf.data class, with the set of tools to help loading, caching and training from very large datasets. But the problem is that, I don't see any way to create and pre-process text file corpus using tf.data class from the documentation. To me, it seems pretty much impossible to do. If there is any way to load corpus fragments with a window size defined by words, kindly let me know.

Thank you in advance.

### Standalone code to reproduce the issue

```shell
corpus = file.read()
            file_contents = corpus.split()[token_start : token_end]

input_tokens, output_tokens = [], []
        for i in tqdm(range(len(file_contents) - gpt_input - 1)):
            input_tokens += [file_contents[i : i + gpt_input]]
            output_tokens += [file_contents[i + gpt_input]]
               
            
        X = [' '.join(input_tokens[i]) for i in tqdm(range(len(input_tokens)))]
        Y = output_tokens
```


### Relevant log output

_No response_</details>