<p dir="auto">Hello All,</p>
<p dir="auto">I am trying to load a big text file for pertaining a BERT model from scratch the size of the txt file is about 11GB and trying to load the model for pertaining is exhausting all the RAM on the system.</p>
<p dir="auto">Is it possible to load all the data in batches and then perform training.</p>
<p dir="auto">I am a bit new to hugging face ecosystem so I would request you all to help me around if you have any clue about this.<br>
I am using Google Colab for the purpose.</p>
<p dir="auto">Please share code snippets, If possible.</p>
<p dir="auto">Cheers !</p>
<p dir="auto">`#construct dataset<br>
from transformers import LineByLineTextDataset</p>
<p dir="auto">file_path="/content/drive/MyDrive/full_text_data.txt"<br>
dataset = LineByLineTextDataset(<br>
tokenizer=tokenizer,<br>
file_path=file_path,<br>
block_size=32)`</p>
<h3 dir="auto">System Info</h3>
<p dir="auto">MacBook Pro (13-inch, M1, 2020)<br>
Chip Apple M1<br>
Memory 16 GB</p>
<p dir="auto">(Also tried with Google Colab)</p>
<h3 dir="auto">Who can help?</h3>
<p dir="auto">copy of <a class="issue-link js-issue-link" data-error-text="Failed to load title" data-id="1382818942" data-permission-text="Title is private" data-url="https://github.com/huggingface/transformers/issues/19161" data-hovercard-type="issue" data-hovercard-url="/huggingface/transformers/issues/19161/hovercard" href="https://github.com/huggingface/transformers/issues/19161">#19161</a></p>
<p dir="auto">Hello All,</p>
<p dir="auto">A bit new to HuggingFace environment.</p>
<p dir="auto">I am trying to load a big text file for pertaining a BERT model from scratch the size of the txt file is about 11GB and trying to load the model for pertaining is exhausting all the RAM on the system.</p>
<p dir="auto">Is it possible to load all the data in batches and then perform training.</p>
<p dir="auto">I am a bit new to hugging face ecosystem so I would request you all to help me around if you have any clue about this.<br>
I am using Google Colab for the purpose.</p>
<p dir="auto">Please share code snippets, If possible.</p>
<p dir="auto">Cheers !</p>
<p dir="auto">`#construct dataset<br>
from transformers import LineByLineTextDataset</p>
<p dir="auto">file_path="/content/drive/MyDrive/full_text_data.txt"<br>
dataset = LineByLineTextDataset(<br>
tokenizer=tokenizer,<br>
file_path=file_path,<br>
block_size=32)`</p>
<h3 dir="auto">Information</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> The official example scripts</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own modified scripts</li>
</ul>
<h3 dir="auto">Tasks</h3>
<ul class="contains-task-list">
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox"> An officially supported task in the <code class="notranslate">examples</code> folder (such as GLUE/SQuAD, ...)</li>
<li class="task-list-item"><input type="checkbox" id="" disabled="" class="task-list-item-checkbox" checked=""> My own task or dataset (give details below)</li>
</ul>
<h3 dir="auto">Reproduction</h3>
<p dir="auto">try to load the data with a big text file</p>
<h3 dir="auto">Expected behavior</h3>
<p dir="auto">The data should be read in the batches to combat OOM.</p>