Space Connection Error - Dataset Map
Hi, i have created a Streamlit space with a CPU UPGRADED and i try to do a Dataset.map, after about 20 minute the space go on “Connection timeout” on container log i see “stopping…” and my procedure of Dataset.map stop and i need to restart…On the screen i see a popup with message: “Sorry, there is an error on our side.”How can i resolve? I have tried also to do a tokenized_dataset local on my pc but it is 45 GB and i don’t know how to load it on huggingface to use on space and so skip this step of dataset.mapThank you
2024-12-30T09:14:52Z
To resolve the connection timeout and error duringDataset.mapin your Streamlit Space:Increase Resources: Ensure your Streamlit Space has sufficient CPU and memory for processing large datasets.Optimize Dataset Processing: Process the dataset in smaller batches using a chunked approach to avoid timeouts.Pre-tokenize Locally: Tokenize the dataset locally, save it in a format like Parquet/JSON, and upload it to Hugging Face. Load the tokenized dataset directly in your Streamlit Space.UsemapOptimizations: Usebatched=Trueandnum_procinDataset.mapfor better performance.Check Logs: Review container logs for detailed error messages to identify specific issues.
2024-12-30T09:30:50Z
Hi,i have loaded my dataset and converted it on parquet formatdataset = load_dataset(“Cicciokr/CC-100-Latin”, revision=“refs/convert/parquet”)i have also tokenized locally my dataset and is about 45GB but is not possible to upload it on the Spaces, error “reach limit 1GB”.To use the Dataset it in the Trainer function, i need to do a mapping with tokenizer right or is not necessary and i can use directly the dataset?tokenizer = RobertaTokenizerFast(vocab_file=“./vocab.json”,*merges_file=“./merges.txt”,*)def preprocess_function(examples):return tokenizer(examples[‘text’], truncation=True, padding=‘max_length’, max_length=512)*tokenized_dataset = dataset.map(preprocess_function, batched=True, num_proc=8)trainer = Trainer(train_dataset=tokenized_dataset[‘train’]*)In the log i have no other information only connection timeout, and i use the CPU Upgraded, is not possible to have connection timeout while is in run mode with no other error, i pay the hourly use of this resources.Thank you
2024-12-30T09:34:47Z
Yes, to use your dataset with theTrainerfunction, you need to map the tokenizer over the dataset to tokenize the text data. You’ve already set up the tokenizer and preprocessing function correctly. By using thedataset.mapfunction, you’re applying the tokenizer to your dataset, which prepares it for use with the Trainer.
2024-12-30T10:16:30Z
i have also tokenized locally my dataset and is about 45GB but is not possible to upload it on the Spaces, error “reach limit 1GB”.Model repos and dataset repos are large and fast, but you cannot upload large files to Spaces.It is possible to download from model repos and dataset repos after starting Spaces, and it is fast, so it is good to put datasets in dataset repos.Edit:However, the free Spaces disk space is still 50GB even after starting up, so if you only want to read a 45GB data set, it should be fine, but it might be difficult to process… it’s 50-50.
2024-12-30T10:19:50Z
How can i do to load “tokenized dataset” from the dataset repos?And also my question is… if the space can’t even run a dataset map, is it possible to run training on a T4 small or will it always go into connection timeout?Now i’m trying to do a Dataset.map with a T4 Small, but is slower then a CPU Upgraded ( T4 Small about 7000 examples/s, CPU Upgraded 20000 examples/s )Thank you
2024-12-30T10:23:37Z
How can i do to load “tokenized dataset” from the dataset repos?I’ve found a way to save it. However, I think it would be more reliable tosplit the data set into several partsand run the training in several sessions if possible. It would be difficult if the data cannot be divided due to its nature…github.com/huggingface/transformersCan we save tokenized datasets?opened05:44AM - 28 Oct 21 UTCclosed03:01PM - 05 Dec 21 UTConjas-buidlIt takes a lot of time to tokenize my dataset, is there a way to save it and loa…d it?

Let's say I'm using the IMDB toy dataset, How to save the `inputs` object?
```
from datasets import load_dataset
raw_datasets = load_dataset("imdb")
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
inputs = tokenizer(sentences, padding="max_length", truncation=True)Another option would be to customize the Trainer’s DataCollator to incorporate a tokenizer. Anduse trainer with IterableDataset.stackoverflow.comHow to use huggingface HF trainer train with custom collate function?python, huggingface-transformers, huggingface, huggingface-datasetsasked byCharlie Parkeron11:22PM - 10 Aug 23 UTCDataCollator vs. Tokenizers🤗Transformershey@hamel, welcome to theforum! 
you’re spot on about using data collators to do padding on-the-fly. to understand why this helps, consider the following scenarios: 

use the tokenizer to pad each example in the dataset to the length of the longest example in the dataset
use the tokenizer and DataCollatorWithPadding (docs) to pad each example in a batch to the length of the longest example in the batch

clearly, scenario 2 is more efficient, especially in cases where a few examples happ…