Very slow data loading on large dataset
Hi ! Currently it is that slow because we haven't implemented parallelism for the dataset generation yet.
Though we will definitely work on this :)

For now I'd recommend loading the dataset shard by shard in parallel, and then concatenate them:
```python
# in one process, load first 100 files for english
shard1 = load_dataset("allenai/c4", data_files="multilingual/c4-en.tfrecord-000**.json.gz")
# in another process load next 100 files for english
shard2 = load_dataset("allenai/c4", data_files="multilingual/c4-en.tfrecord-001**.json.gz")

# finally
concatenate_datasets([shard1, shard2, ...])
I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

Is there something that I am missing ?
Very slow data loading on large dataset 
 I made a simple python script to check the NLP library speed, which loads 1.1 TB of textual data.
It has been 8 hours and still, it is on the loading steps.
It does work when the text dataset size is small about  1 GB, but it doesn't scale.
It also uses a single thread during the data loading step.

```
train_files = glob.glob("xxx/*.txt",recursive=True)
random.shuffle(train_files)

print(train_files)

dataset = nlp.load_dataset('text', 
                           data_files=train_files,
                           name="customDataset",
                           version="1.0.0",
                           cache_dir="xxx/nlp")
```

Is there something that I am missing ? 
 Hi ! Currently it is that slow because we haven't implemented parallelism for the dataset generation yet.
Though we will definitely work on this :)

For now I'd recommend loading the dataset shard by shard in parallel, and then concatenate them:
```python
# in one process, load first 100 files for english
shard1 = load_dataset("allenai/c4", data_files="multilingual/c4-en.tfrecord-000**.json.gz")
# in another process load next 100 files for english
shard2 = load_dataset("allenai/c4", data_files="multilingual/c4-en.tfrecord-001**.json.gz")

# finally
concatenate_datasets([shard1, shard2, ...])