Updated TTC4900 Dataset

- The source address of the TTC4900 dataset of [@savasy](https://github.com/savasy) has been updated for direct download.
- Updated readme.


Adding to_tf_dataset method

Hi @lewtun, thanks for reporting.

It seems that our library fails at inferring the dtype of the columns:
- `milestone`
- `performed_via_github_app`

(and assigns them `null` dtype).

Oh my **god** do not merge this yet, it's just a draft.

I've added a method (via a mixin) to the `arrow_dataset.Dataset` class that automatically converts our Dataset classes to TF Dataset classes ready for training. It hopefully has most of the features we want, including streaming from disk (no need to load the whole dataset in memory!), correct shuffling, variable-length batches to reduce compute, and correct support for unusual padding. It achieves that by calling the tokenizer `pad` method in the middle of a TF compute graph via a very hacky call to `tf.py_function`, which is heretical but seems to work.

A number of issues need to be resolved before it's ready to merge, though:

1) Is a MixIn the right way to do this? Do other classes besides `arrow_dataset.Dataset` need this method too?
2) Needs an argument to support constant-length batches for TPU training - this is easy to add and I'll do it soon.
3) Needs the user to supply the list of columns to drop from the arrow `Dataset`. Is there some automatic way to get the columns we want, or see which columns were added by the tokenizer?
4) Assumes the label column is always present and always called "label" - this is probably not great, but I'm not sure what the 'correct' thing to do here is.


Update CommonVoice with new release

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** CommonVoice mid-2021 release
- **Description:** more data in CommonVoice: Languages that have increased the most by percentage are Thai (almost 20x growth, from 12 hours to 250 hours), Luganda (almost 9x growth, from 8 to 80), Esperanto (7x growth, from 100 to 840), and Tamil (almost 8x, from 24 to 220).
- **Paper:** https://discourse.mozilla.org/t/common-voice-2021-mid-year-dataset-release/83812
- **Data:** https://commonvoice.mozilla.org/en/datasets
- **Motivation:** More data and more varied. I think we just need to add configs in the existing dataset script.

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Fix IndexError while loading Arabic Billion Words dataset

bug

Something isn't working

Catch `IndexError` and ignore that record.

Close #2727.


Concurrent use of same dataset (already downloaded)

Something isn't working

## Describe the bug
When launching several jobs at the same time loading the same dataset trigger some errors see (last comments).

## Steps to reproduce the bug
export HF_DATASETS_CACHE=/gpfswork/rech/toto/datasets
for MODEL in "bert-base-uncased" "roberta-base" "distilbert-base-cased"; do #  "bert-base-uncased" "bert-large-cased" "roberta-large" "albert-base-v1" "albert-large-v1"; do
  for TASK_NAME in "mrpc" "rte" 'imdb' "paws" "mnli"; do
    export OUTPUT_DIR=${MODEL}_${TASK_NAME}
    sbatch --job-name=${OUTPUT_DIR} \
      --gres=gpu:1 \
      --no-requeue \
      --cpus-per-task=10 \
      --hint=nomultithread \
      --time=1:00:00 \
      --output=jobinfo/${OUTPUT_DIR}_%j.out \
      --error=jobinfo/${OUTPUT_DIR}_%j.err \
      --qos=qos_gpu-t4 \
      --wrap="module purge; module load pytorch-gpu/py3/1.7.0 ; export HF_DATASETS_OFFLINE=1; export HF_DATASETS_CACHE=/gpfswork/rech/toto/datasets;  python compute_measures.py --seed=$SEED --saving_path=results --batch_size=$BATCH_SIZE --task_name=$TASK_NAME --model_name=/gpfswork/rech/toto/transformers_models/$MODEL"

  done
done

```python
# Sample code to reproduce the bug
        dataset_train = load_dataset('imdb', split='train', download_mode="reuse_cache_if_exists")
        dataset_train = dataset_train.map(lambda e: tokenizer(e['text'], truncation=True, padding='max_length'),
                                          batched=True).select(list(range(args.filter)))

        dataset_val = load_dataset('imdb', split='train', download_mode="reuse_cache_if_exists")
        dataset_val = dataset_val.map(lambda e: tokenizer(e['text'], truncation=True, padding='max_length'),
                                      batched=True).select(list(range(args.filter, args.filter + 5000)))

        dataset_test = load_dataset('imdb', split='test', download_mode="reuse_cache_if_exists")
        dataset_test = dataset_test.map(lambda e: tokenizer(e['text'], truncation=True, padding='max_length'),
                                        batched=True)
```

## Expected results
I believe I am doing something wrong with the objects.

## Actual results
Traceback (most recent call last):
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 983, in _prepare_split
    check_duplicates=True,
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/arrow_writer.py", line 192, in __init__
    self.stream = pa.OSFile(self._path, "wb")
  File "pyarrow/io.pxi", line 829, in pyarrow.lib.OSFile.__cinit__
  File "pyarrow/io.pxi", line 844, in pyarrow.lib.OSFile._open_writable
  File "pyarrow/error.pxi", line 122, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 97, in pyarrow.lib.check_status
FileNotFoundError: [Errno 2] Failed to open local file '/gpfswork/rech/tts/unm25jp/datasets/paws/labeled_final/1.1.0/09d8fae989bb569009a8f5b879ccf2924d3e5cd55bfe2e89e6dab1c0b50ecd34.incomplete/paws-test.arrow'. Detail: [errno 2] No such file or directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "compute_measures.py", line 181, in <module>
    train_loader, val_loader, test_loader = get_dataloader(args)
  File "/gpfsdswork/projects/rech/toto/intRAOcular/dataset_utils.py", line 69, in get_dataloader
    dataset_train = load_dataset('paws', "labeled_final", split='train', download_mode="reuse_cache_if_exists")
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/gpfslocalsup/pub/anaconda-py3/2020.02/envs/pytorch-gpu-1.7.0/lib/python3.7/site-packages/datasets/builder.py", line 658, in _download_and_prepare
    + str(e)
OSError: Cannot find data file.
Original error:
[Errno 2] Failed to open local file '/gpfswork/rech/toto/datasets/paws/labeled_final/1.1.0/09d8fae989bb569009a8f5b879ccf2924d3e5cd55bfe2e89e6dab1c0b50ecd34.incomplete/paws-test.arrow'. Detail: [errno 2] No such file or directory

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: datasets==1.8.0
- Platform: linux (jeanzay)
- Python version: pyarrow==2.0.0
- PyArrow version: 3.7.8


Error in loading the Arabic Billion Words Corpus

Something isn't working

## Describe the bug
I get `IndexError: list index out of range` when trying to load the `Techreen` and `Almustaqbal` configs of the dataset.

## Steps to reproduce the bug
```python
load_dataset("arabic_billion_words", "Techreen")
load_dataset("arabic_billion_words", "Almustaqbal")
```

## Expected results
The datasets load succefully.

## Actual results
```python
_extract_tags(self, sample, tag)
    139             if len(out) > 0:
    140                 break
--> 141         return out[0]
    142
    143     def _clean_text(self, text):

IndexError: list index out of range
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.2
- Platform: Ubuntu 18.04.5 LTS
- Python version: 3.7.11
- PyArrow version: 3.0.0


Typo fix `tokenize_exemple`

There is a small typo in the main README.md


Pass use_auth_token to request_etags

Fix #2724.


404 Error when loading remote data files from private repo

Something isn't working

## Describe the bug
When loading remote data files from a private repo, a 404 error is raised.

## Steps to reproduce the bug
```python
url = hf_hub_url("lewtun/asr-preds-test", "preds.jsonl", repo_type="dataset")
dset = load_dataset("json", data_files=url, use_auth_token=True)
# HTTPError: 404 Client Error: Not Found for url: https://huggingface.co/datasets/lewtun/asr-preds-test/resolve/main/preds.jsonl
```

## Expected results
Load dataset.

## Actual results
404 Error.


Fix en subset by modifying dataset_info with correct validation infos

- Related to: #2682

We correct the values of `en` subset concerning the expected validation values (both `num_bytes` and `num_examples`.

Instead of having:

`{"name": "validation", "num_bytes": 828589180707, "num_examples": 364868892, "dataset_name": "c4"}`

We replace with correct values:

`{"name": "validation", "num_bytes": 825767266, "num_examples": 364608, "dataset_name": "c4"}`

There are still issues with validation with other subsets, but I can't download all the files, unzip to check for the correct number of bytes. (If you have a fast way to obtain those values for other subsets, I can do this in this PR ... otherwise I can't spend those resources)


Missing cache file

Something isn't working

Strangely missing cache file after I  restart my program again.

`glue_dataset = datasets.load_dataset('glue', 'sst2')`

`FileNotFoundError: [Errno 2] No such file or directory: /Users/chris/.cache/huggingface/datasets/glue/sst2/1.0.0/dacbe3125aa31d7f70367a07a8a9e72a5a0bfeb5fc42e75c9db75b96d6053ad/dataset_info.json'`


Deal with the bad check in test_load.py

This PR removes a check that's been added in #2684. My intention with this check was to capture an URL in the error message, but instead, it captures a substring of the previous regex match in the test function. Another option would be to replace this check with:
```python
m_paths = re.findall(r"\S*_dummy/_dummy.py\b", str(exc_info.value))  # on Linux this will match an URL as well as a local_path due to different os.sep, so take the last element (an URL always comes last in the list)
assert len(m_paths) > 0 and is_remote_url(m_paths[-1])  # is_remote_url comes from datasets.utils.file_utils
```

@lhoestq Let me know which one of these two approaches (delete or replace) do you prefer?


fix: 🐛 fix two typos


Use ETag in streaming mode to detect resource updates

enhancement

New feature or request

dataset-viewer

Related to the dataset viewer on huggingface.co

**Is your feature request related to a problem? Please describe.**

I want to cache data I generate from processing a dataset I've loaded in streaming mode, but I've currently no way to know if the remote data has been updated or not, thus I don't know when to invalidate my cache.

**Describe the solution you'd like**

Take the ETag of the data files into account and provide it (directly or through a hash) to give a signal that I can invalidate my cache.

**Describe alternatives you've considered**

None


New documentation structure

Organize Datasets documentation into four documentation types to improve clarity and discoverability of content.

**Content to add in the very short term (feel free to add anything I'm missing):**
- A discussion on why Datasets uses Arrow that includes some context and background about why we use Arrow. Would also be great to talk about Datasets speed and performance here, and if you can share any benchmarking/tests you did, that would be awesome! Finally, a discussion about how memory-mapping frees the user from RAM constraints would be very helpful.
- Explain why you would want to disable or override verifications when loading a dataset.
- If possible, include a code sample of when the number of elements in the field of an output dictionary aren’t the same as the other fields in the output dictionary (taken from the [note](https://huggingface.co/docs/datasets/processing.html#augmenting-the-dataset) here).


Fix shuffle on IterableDataset that disables batching in case any functions were mapped

Made a very minor change to fix the issue#2716. Added the missing argument in the constructor call.

As discussed in the bug report, the change is made to prevent the `shuffle` method call from resetting the value of `batched` attribute in `MappedExamplesIterable`

Fix #2716.


Calling shuffle on IterableDataset will disable batching in case any functions were mapped

bug

Something isn't working

When using dataset in streaming mode, if one applies `shuffle` method on the dataset and `map` method for which `batched=True` than the batching operation will not happen, instead `batched` will be set to `False`

I did RCA on the dataset codebase, the problem is emerging from [this line of code](https://github.com/huggingface/datasets/blob/d25a0bf94d9f9a9aa6cabdf5b450b9c327d19729/src/datasets/iterable_dataset.py#L197) here as it is
`self.ex_iterable.shuffle_data_sources(seed), function=self.function, batch_size=self.batch_size`, as one can see it is missing batched argument, which means that the iterator fallsback to default constructor value, which in this case is `False`.
To remedy the problem we can change this line to
`self.ex_iterable.shuffle_data_sources(seed), function=self.function, batched=self.batched, batch_size=self.batch_size`


Update PAN-X data URL in XTREME dataset

Related to #2710, #2691.


add more precise information for size

enhancement

New feature or request

For the import into ELG, we would like a more precise description of the size of the dataset, instead of the current size categories. The size can be expressed in bytes, or any other preferred size unit. As suggested in the slack channel, perhaps this could be computed with a regex for existing datasets.


Enumerate all ner_tags values in WNUT 17 dataset

This PR does:
- Enumerate all ner_tags in dataset card Data Fields section
- Add all metadata tags to dataset card

Close #2709.


Update WikiANN data URL

WikiANN data source URL is no longer accessible: 404 error from Dropbox.

We have decided to host it at Hugging Face. This PR updates the data source URL, the metadata JSON file and the dataset card.

Close #2691.


Missing documentation for wnut_17 (ner_tags)

bug

Something isn't working

On the info page of the wnut_17 data set (https://huggingface.co/datasets/wnut_17), the model output of ner-tags is only documented for these 5 cases:

`ner_tags: a list of classification labels, with possible values including O (0), B-corporation (1), I-corporation (2), B-creative-work (3), I-creative-work (4).`

I trained a model with the data and it gives me 13 classes:

```
"id2label": {
    "0": 0,
    "1": 1,
    "2": 2,
    "3": 3,
    "4": 4,
    "5": 5,
    "6": 6,
    "7": 7,
    "8": 8,
    "9": 9,
    "10": 10,
    "11": 11,
    "12": 12
  }

  "label2id": {
    "0": 0,
    "1": 1,
    "10": 10,
    "11": 11,
    "12": 12,
    "2": 2,
    "3": 3,
    "4": 4,
    "5": 5,
    "6": 6,
    "7": 7,
    "8": 8,
    "9": 9
  }
```
The paper (https://www.aclweb.org/anthology/W17-4418.pdf) explains those 6 categories, but the ordering does not match:

```
1. person
2. location (including GPE, facility)
3. corporation
4. product (tangible goods, or well-defined
services)
5. creative-work (song, movie, book and
so on)
6. group (subsuming music band, sports team,
and non-corporate organisations)
```
I would be very helpful for me, if somebody could clarify the model ouputs and explain the "B-" and "I-" prefixes to me.

Really great work with that and the other packages, I couldn't believe that training the model with that data was basically a one-liner!


QASC: incomplete training set

Something isn't working

## Describe the bug
The training instances are not loaded properly.

## Steps to reproduce the bug
```python
from datasets import load_dataset

dataset = load_dataset("qasc", script_version='1.10.2')

def load_instances(split):
    instances = dataset[split]
    print(f"split: {split} - size: {len(instances)}")
    for x in instances:
        print(json.dumps(x))

load_instances('test')
load_instances('validation')
load_instances('train')
```

##  results
For test and validation, we can see the examples in the output (which is good!):
```
split: test - size: 920
{"answerKey": "", "choices": {"label": ["A", "B", "C", "D", "E", "F", "G", "H"], "text": ["Anthax", "under water", "uterus", "wombs", "two", "moles", "live", "embryo"]}, "combinedfact": "", "fact1": "", "fact2": "", "formatted_question": "What type of birth do therian mammals have? (A) Anthax (B) under water (C) uterus (D) wombs (E) two (F) moles (G) live (H) embryo", "id": "3C44YUNSI1OBFBB8D36GODNOZN9DPA", "question": "What type of birth do therian mammals have?"}
{"answerKey": "", "choices": {"label": ["A", "B", "C", "D", "E", "F", "G", "H"], "text": ["Corvidae", "arthropods", "birds", "backbones", "keratin", "Jurassic", "front paws", "Parakeets."]}, "combinedfact": "", "fact1": "", "fact2": "", "formatted_question": "By what time had mouse-sized viviparous mammals evolved? (A) Corvidae (B) arthropods (C) birds (D) backbones (E) keratin (F) Jurassic (G) front paws (H) Parakeets.", "id": "3B1NLC6UGZVERVLZFT7OUYQLD1SGPZ", "question": "By what time had mouse-sized viviparous mammals evolved?"}
{"answerKey": "", "choices": {"label": ["A", "B", "C", "D", "E", "F", "G", "H"], "text": ["Reduced friction", "causes infection", "vital to a good life", "prevents water loss", "camouflage from consumers", "Protection against predators", "spur the growth of the plant", "a smooth surface"]}, "combinedfact": "", "fact1": "", "fact2": "", "formatted_question": "What does a plant's skin do? (A) Reduced friction (B) causes infection (C) vital to a good life (D) prevents water loss (E) camouflage from consumers (F) Protection against predators (G) spur the growth of the plant (H) a smooth surface", "id": "3QRYMNZ7FYGITFVSJET3PS0F4S0NT9", "question": "What does a plant's skin do?"}
...
```
However, only a few instances are loaded for the training split, which is not correct.

## Environment info
- `datasets` version: '1.10.2'
- Platform: MaxOS
- Python version:3.7
- PyArrow version: 3.0.0


404 Not Found Error when loading LAMA dataset

The [LAMA](https://huggingface.co/datasets/viewer/?dataset=lama) probing dataset is not available for download:

Steps to Reproduce:

1. `from datasets import load_dataset`
2. `dataset = load_dataset('lama', 'trex')`.

Results:
`FileNotFoundError: Couldn't find file locally at lama/lama.py, or remotely at https://raw.githubusercontent.com/huggingface/datasets/1.1.2/datasets/lama/lama.py or https://s3.amazonaws.com/datasets.huggingface.co/datasets/datasets/lama/lama.py`


Update BibTeX entry

Update BibTeX entry.


404 not found error on loading WIKIANN dataset

Something isn't working

## Describe the bug
Unable to retreive wikiann English dataset

## Steps to reproduce the bug
```python
from datasets import list_datasets, load_dataset, list_metrics, load_metric
WIKIANN = load_dataset("wikiann","en")
```

## Expected results
Colab notebook should display successful download status

## Actual results
FileNotFoundError: Couldn't find file at https://www.dropbox.com/s/12h3qqog6q4bjve/panx_dataset.tar?dl=1

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.1
- Platform: Linux-5.4.104+-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.11
- PyArrow version: 3.0.0


Fix pick default config name message

The error message to tell which config name to load is not displayed.

This is because in the code it was considering the config kwargs to be non-empty, which is a special case for custom configs created on the fly. It appears after this change: https://github.com/huggingface/datasets/pull/2659

I fixed that by making the config kwargs empty by default, even if default parameters are passed

Fix https://github.com/huggingface/datasets/issues/2703


Bad message when config name is missing

When loading a dataset that have several configurations, we expect to see an error message if the user doesn't specify a config name.

However in `datasets` 1.10.0 and 1.10.1 it doesn't show the right message:

```python
import datasets

datasets.load_dataset("glue")
```
raises
```python
AttributeError: 'BuilderConfig' object has no attribute 'text_features'
```
instead of
```python
ValueError: Config name is missing.
Please pick one among the available configs: ['cola', 'sst2', 'mrpc', 'qqp', 'stsb', 'mnli', 'mnli_mismatched', 'mnli_matched', 'qnli', 'rte', 'wnli', 'ax']
Example of usage:
        `load_dataset('glue', 'cola')`
```


Fix download_mode docstrings

documentation

Improvements or additions to documentation

Fix `download_mode` docstrings.


from datasets import Dataset is failing

bug

Something isn't working

## Describe the bug
A clear and concise description of what the bug is.

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
from datasets import Dataset
```

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.
/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in <module>()
     25 import posixpath
     26 import requests
---> 27 from tqdm.contrib.concurrent import thread_map
     28
     29 from .. import __version__, config, utils

ModuleNotFoundError: No module named 'tqdm.contrib.concurrent'

---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.

To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: latest version as of 07/21/2021
- Platform: Google Colab
- Python version: 3.7
- PyArrow version:


cannot combine splits merging and streaming?

Something isn't working

this does not work:
`dataset = datasets.load_dataset('mc4','iw',split='train+validation',streaming=True)`
with error:
`ValueError: Bad split: train+validation. Available splits: ['train', 'validation']`

these work:
`dataset = datasets.load_dataset('mc4','iw',split='train+validation')`
`dataset = datasets.load_dataset('mc4','iw',split='train',streaming=True)`
`dataset = datasets.load_dataset('mc4','iw',split='validation',streaming=True)`

i could not find a reference to this in the documentation and the error message is confusing. also would be nice to allow streaming for the merged splits


Ignore empty batch when writing

This prevents an schema update with unknown column types, as reported in #2644.

This is my first attempt at fixing the issue. I tested the following:
- First batch returned by a batched map operation is empty.
- An intermediate batch is empty.
- `python -m unittest tests.test_arrow_writer` passes.

However, `arrow_writer` looks like a pretty generic interface, I'm not sure if there are other uses I may have overlooked. Let me know if that's the case, or if a better approach would be preferable.


Fix import on Colab

Fix #2695, fix #2700.


Add support for disable_progress_bar on Windows

This PR is a continuation of #2667 and adds support for `utils.disable_progress_bar()` on Windows when using multiprocessing. This [answer](https://stackoverflow.com/a/6596695/14095927) on SO explains it nicely why the current approach (with calling `utils.is_progress_bar_enabled()` inside `Dataset._map_single`) would not work on Windows.


Cannot import load_dataset on Colab

Something isn't working

## Describe the bug
Got tqdm concurrent module not found error during importing load_dataset from datasets.

## Steps to reproduce the bug
Here [colab notebook](https://colab.research.google.com/drive/1pErWWnVP4P4mVHjSFUtkePd8Na_Qirg4?usp=sharing) to reproduce the error

On colab:
```python
!pip install datasets
from datasets import load_dataset
```

## Expected results
Works without error

## Actual results
Specify the actual results or traceback.
```
ModuleNotFoundError                       Traceback (most recent call last)
<ipython-input-2-8cc7de4c69eb> in <module>()
----> 1 from datasets import load_dataset, load_metric, Metric, MetricInfo, Features, Value
      2 from sklearn.metrics import mean_squared_error

/usr/local/lib/python3.7/dist-packages/datasets/__init__.py in <module>()
     31     )
     32
---> 33 from .arrow_dataset import Dataset, concatenate_datasets
     34 from .arrow_reader import ArrowReader, ReadInstruction
     35 from .arrow_writer import ArrowWriter

/usr/local/lib/python3.7/dist-packages/datasets/arrow_dataset.py in <module>()
     40 from tqdm.auto import tqdm
     41
---> 42 from datasets.tasks.text_classification import TextClassification
     43
     44 from . import config, utils

/usr/local/lib/python3.7/dist-packages/datasets/tasks/__init__.py in <module>()
      1 from typing import Optional
      2
----> 3 from ..utils.logging import get_logger
      4 from .automatic_speech_recognition import AutomaticSpeechRecognition
      5 from .base import TaskTemplate

/usr/local/lib/python3.7/dist-packages/datasets/utils/__init__.py in <module>()
     19
     20 from . import logging
---> 21 from .download_manager import DownloadManager, GenerateMode
     22 from .file_utils import DownloadConfig, cached_path, hf_bucket_url, is_remote_url, temp_seed
     23 from .mock_download_manager import MockDownloadManager

/usr/local/lib/python3.7/dist-packages/datasets/utils/download_manager.py in <module>()
     24
     25 from .. import config
---> 26 from .file_utils import (
     27     DownloadConfig,
     28     cached_path,

/usr/local/lib/python3.7/dist-packages/datasets/utils/file_utils.py in <module>()
     25 import posixpath
     26 import requests
---> 27 from tqdm.contrib.concurrent import thread_map
     28
     29 from .. import __version__, config, utils

ModuleNotFoundError: No module named 'tqdm.contrib.concurrent'
```
## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.10.0
- Platform: Colab
- Python version: 3.7.11
- PyArrow version: 3.0.0


fix: 🐛 change string format to allow copy/paste to work in bash

Before: copy/paste resulted in an error because the square bracket
characters `[]` are special characters in bash


Fix OSCAR Esperanto

The Esperanto part (original) of OSCAR has the wrong number of examples:
```python
from datasets import load_dataset
raw_datasets = load_dataset("oscar", "unshuffled_original_eo")
```
raises
```python
NonMatchingSplitsSizesError:
[{'expected': SplitInfo(name='train', num_bytes=314188336, num_examples=121171, dataset_name='oscar'),
'recorded': SplitInfo(name='train', num_bytes=314064514, num_examples=121168, dataset_name='oscar')}]
```

I updated the number of expected examples in dataset_infos.json

cc @sgugger


Update BibTeX entry

Update BibTeX entry


xtreme / pan-x cannot be downloaded

Something isn't working

## Describe the bug

Dataset xtreme / pan-x cannot be loaded

Seems related to https://github.com/huggingface/datasets/pull/2326

## Steps to reproduce the bug

```python
dataset = load_dataset("xtreme", "PAN-X.fr")
```

## Expected results

Load the dataset

## Actual results

```
FileNotFoundError: Couldn't find file at https://www.dropbox.com/s/12h3qqog6q4bjve/panx_dataset.tar?dl=1
```

## Environment info

- `datasets` version: 1.9.0
- Platform: macOS-11.4-x86_64-i386-64bit
- Python version: 3.8.11
- PyArrow version: 4.0.1


Docs details

Some comments here:

- the code samples assume the expected libraries have already been installed. Maybe add a section at start, or add it to every code sample. Something like `pip install datasets transformers torch  'datasets[streaming]'` (maybe just link to https://huggingface.co/docs/datasets/installation.html + a one-liner that installs all the requirements / alternatively a requirements.txt file)
- "If you’d like to play with the examples, you must install it from source." in https://huggingface.co/docs/datasets/installation.html: it's not clear to me what this means (what are these "examples"?)
- in https://huggingface.co/docs/datasets/loading_datasets.html: "or AWS bucket if it’s not already stored in the library". It's the only place in the doc (aside from the docstring https://huggingface.co/docs/datasets/package_reference/loading_methods.html?highlight=aws bucket#datasets.list_datasets) where the "AWS bucket" is mentioned. It's not easy to understand what this means. Maybe explain more, and link to https://s3.amazonaws.com/datasets.huggingface.co and/or https://huggingface.co/docs/datasets/filesystems.html.
- example in https://huggingface.co/docs/datasets/loading_datasets.html#manually-downloading-files is obsoleted by https://github.com/huggingface/datasets/pull/2326. Also: see https://github.com/huggingface/datasets/issues/2691 for a bug on this specific dataset.
- in https://huggingface.co/docs/datasets/loading_datasets.html#manually-downloading-files the doc says "After you’ve downloaded the files, you can point to the folder hosting them locally with the data_dir argument as follows:", but the following example does not show how to use `data_dir`
- in https://huggingface.co/docs/datasets/loading_datasets.html#csv-files, it would be nice to have an URL to the csv loader reference (but I'm not sure there is one in the API reference). This comment applies in many places in the doc: I would want the API reference to contain doc for all the code/functions/classes... and I would want a lot more links inside the doc pointing to the API entries.
- in the API reference (docstrings) I would prefer "SOURCE" to link to github instead of a copy of the code inside the docs site (eg. https://github.com/huggingface/datasets/blob/master/src/datasets/load.py#L711 instead of https://huggingface.co/docs/datasets/_modules/datasets/load.html#load_dataset)
- it seems like not all the API is exposed in the doc. For example, there is no doc for [`disable_progress_bar`](https://github.com/huggingface/datasets/search?q=disable_progress_bar), see https://huggingface.co/docs/datasets/search.html?q=disable_progress_bar, even if the code contains docstrings. Does it mean that the function is not officially supported? (otherwise, maybe it also deserves a mention in https://huggingface.co/docs/datasets/package_reference/logging_methods.html)
- in https://huggingface.co/docs/datasets/loading_datasets.html?highlight=most%20efficient%20format%20have%20json%20files%20consisting%20multiple%20json%20objects#json-files, "The most efficient format is to have JSON files consisting of multiple JSON objects, one per line, representing individual data rows:", maybe link to https://en.wikipedia.org/wiki/JSON_streaming#Line-delimited_JSON and give it a name ("line-delimited JSON"? "JSON Lines" as in https://huggingface.co/docs/datasets/processing.html#exporting-a-dataset-to-csv-json-parquet-or-to-python-objects ?)
- in https://huggingface.co/docs/datasets/loading_datasets.html, for the local files sections, it would be nice to provide sample csv / json / text files to download, so that it's easier for the reader to try to load them (instead: they won't try)
- the doc explains how to shard a dataset, but does not explain why and when a dataset should be sharded (I have no idea... for [parallelizing](https://huggingface.co/docs/datasets/processing.html#multiprocessing)?). It does neither give an idea of the number of shards a dataset typically should have and why.
- the code example in https://huggingface.co/docs/datasets/processing.html#mapping-in-a-distributed-setting does not work, because `training_args` has not been defined before in the doc.


cannot save the dataset to disk after rename_column

Something isn't working

## Describe the bug
If you use `rename_column` and do no other modification, you will be unable to save the dataset using `save_to_disk`

## Steps to reproduce the bug
```python
# Sample code to reproduce the bug
In [1]: from datasets import Dataset, load_from_disk
In [5]: dataset=Dataset.from_dict({'foo': [0]})
In [7]: dataset.save_to_disk('foo')
In [8]: dataset=load_from_disk('foo')
In [10]: dataset=dataset.rename_column('foo', 'bar')
In [11]: dataset.save_to_disk('foo')
---------------------------------------------------------------------------
PermissionError                           Traceback (most recent call last)
<ipython-input-11-a3bc0d4fc339> in <module>
----> 1 dataset.save_to_disk('foo')

/mnt/beegfs/projects/meerqat/anaconda3/envs/meerqat/lib/python3.7/site-packages/datasets/arrow_dataset.py in save_to_disk(self, dataset_path
, fs)
    597             if Path(dataset_path, config.DATASET_ARROW_FILENAME) in cache_files_paths:
    598                 raise PermissionError(
--> 599                     f"Tried to overwrite {Path(dataset_path, config.DATASET_ARROW_FILENAME)} but a dataset can't overwrite itself."
    600                 )
    601             if Path(dataset_path, config.DATASET_INDICES_FILENAME) in cache_files_paths:

PermissionError: Tried to overwrite foo/dataset.arrow but a dataset can't overwrite itself.
```

N. B. I created the dataset from dict to enable easy reproduction but the same happens if you load an existing dataset (e.g. starting from `In [8]`)

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.8.0
- Platform: Linux-3.10.0-1160.11.1.el7.x86_64-x86_64-with-centos-7.9.2009-Core
- Python version: 3.7.10
- PyArrow version: 3.0.0


hebrew language codes he and iw should be treated as aliases

Something isn't working

https://huggingface.co/datasets/mc4 not listed when searching for hebrew datasets (he) as it uses the older language code iw, preventing discoverability.


Minor documentation fix

Currently, [Writing a dataset loading script](https://huggingface.co/docs/datasets/add_dataset.html) page has a small error. A link to `matinf` dataset in [_Dataset scripts of reference_](https://huggingface.co/docs/datasets/add_dataset.html#dataset-scripts-of-reference) section actually leads to `xsquad`, instead. This PR fixes that.


Fix bad config ids that name cache directories

`data_dir=None` was considered a dataset config parameter, hence creating a special config_id for all dataset being loaded.
Since the config_id is used to name the cache directories, this leaded to datasets being regenerated for users.

I fixed this by ignoring the value of `data_dir` when it's `None` when computing the config_id.
I also added a test to make sure the cache directories are not unexpectedly renamed in the future.

Fix https://github.com/huggingface/datasets/issues/2683


Fix Blog Authorship Corpus dataset

This PR:
- Update the JSON metadata file, which previously was raising a `NonMatchingSplitsSizesError`
- Fix the codec of the data files (`latin_1` instead of `utf-8`), which previously was raising ` UnicodeDecodeError` for some files

Close #2679.


Print absolute local paths in load_dataset error messages

Use absolute local paths in the error messages of `load_dataset` as per @stas00's suggestion in https://github.com/huggingface/datasets/pull/2500#issuecomment-874891223


Cache directories changed due to recent changes in how config kwargs are handled

Since #2659 I can see weird cache directory names with hashes in the config id, even though no additional config kwargs are passed. For example:

```python
from datasets import load_dataset_builder

c4_builder = load_dataset_builder("c4", "en")
print(c4_builder.cache_dir)
# /Users/quentinlhoest/.cache/huggingface/datasets/c4/en-174d3b7155eb68db/0.0.0/...

# instead of
# /Users/quentinlhoest/.cache/huggingface/datasets/c4/en/0.0.0/...
```
This issue could be annoying since it would simply ignore old cache directories for users, and regenerate datasets

cc @stas00 this is what you experienced a few days ago


Fix c4 expected files

Some files were not registered in the list of expected files to download

Fix https://github.com/huggingface/datasets/issues/2677


5 duplicate datasets

Something isn't working

## Describe the bug

In 5 cases, I could find a dataset on Paperswithcode which references two Hugging Face datasets as dataset loaders. They are:

- https://paperswithcode.com/dataset/multinli -> https://huggingface.co/datasets/multi_nli and https://huggingface.co/datasets/multi_nli_mismatch

  <img width="838" alt="Capture d’écran 2021-07-20 à 16 33 58" src="https://user-images.githubusercontent.com/1676121/126342757-4625522a-f788-41a3-bd1f-2a8b9817bbf5.png">

- https://paperswithcode.com/dataset/squad -> https://huggingface.co/datasets/squad and https://huggingface.co/datasets/squad_v2
- https://paperswithcode.com/dataset/narrativeqa -> https://huggingface.co/datasets/narrativeqa and https://huggingface.co/datasets/narrativeqa_manual
- https://paperswithcode.com/dataset/hate-speech-and-offensive-language -> https://huggingface.co/datasets/hate_offensive and https://huggingface.co/datasets/hate_speech_offensive
- https://paperswithcode.com/dataset/newsph-nli -> https://huggingface.co/datasets/newsph and https://huggingface.co/datasets/newsph_nli

Possible solutions:
- don't fix (it works)
- for each pair of duplicate datasets, remove one, and create an alias to the other.

## Steps to reproduce the bug

Visit the Paperswithcode links, and look at the "Dataset Loaders" section

## Expected results

There should only be one reference to a Hugging Face dataset loader

## Actual results

Two Hugging Face dataset loaders


feat: 🎸 add paperswithcode id for qasper dataset

The reverse reference exists on paperswithcode:
https://paperswithcode.com/dataset/qasper


Cannot load the blog_authorship_corpus due to codec errors

Something isn't working

## Describe the bug
A codec error is raised while loading the blog_authorship_corpus.

## Steps to reproduce the bug
```
from datasets import load_dataset
raw_datasets = load_dataset("blog_authorship_corpus")
```

## Expected results
Loading the dataset without errors.

## Actual results
An error similar to the one below was raised for (what seems like) every XML file.
/home/izaskr/.cache/huggingface/datasets/downloads/extracted/7cf52524f6517e168604b41c6719292e8f97abbe8f731e638b13423f4212359a/blogs/788358.male.24.Arts.Libra.xml cannot be loaded. Error message: 'utf-8' codec can't decode byte 0xe7 in position 7551: invalid continuation byte

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/izaskr/anaconda3/envs/local_vae_older/lib/python3.8/site-packages/datasets/load.py", line 856, in load_dataset
    builder_instance.download_and_prepare(
  File "/home/izaskr/anaconda3/envs/local_vae_older/lib/python3.8/site-packages/datasets/builder.py", line 583, in download_and_prepare
    self._download_and_prepare(
  File "/home/izaskr/anaconda3/envs/local_vae_older/lib/python3.8/site-packages/datasets/builder.py", line 671, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/home/izaskr/anaconda3/envs/local_vae_older/lib/python3.8/site-packages/datasets/utils/info_utils.py", line 74, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='train', num_bytes=610252351, num_examples=532812, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='train', num_bytes=614706451, num_examples=535568, dataset_name='blog_authorship_corpus')}, {'expected': SplitInfo(name='validation', num_bytes=37500394, num_examples=31277, dataset_name='blog_authorship_corpus'), 'recorded': SplitInfo(name='validation', num_bytes=32553710, num_examples=28521, dataset_name='blog_authorship_corpus')}]

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-4.15.0-132-generic-x86_64-with-glibc2.10
- Python version: 3.8.8
- PyArrow version: 4.0.1


Import Error in Kaggle notebook

Something isn't working

## Describe the bug
Not able to import datasets library in kaggle notebooks

## Steps to reproduce the bug
```python
!pip install datasets
import datasets
```

## Expected results
No such error

## Actual results
```
ImportError                               Traceback (most recent call last)
<ipython-input-9-652e886d387f> in <module>
----> 1 import datasets

/opt/conda/lib/python3.7/site-packages/datasets/__init__.py in <module>
     31     )
     32
---> 33 from .arrow_dataset import Dataset, concatenate_datasets
     34 from .arrow_reader import ArrowReader, ReadInstruction
     35 from .arrow_writer import ArrowWriter

/opt/conda/lib/python3.7/site-packages/datasets/arrow_dataset.py in <module>
     36 import pandas as pd
     37 import pyarrow as pa
---> 38 import pyarrow.compute as pc
     39 from multiprocess import Pool, RLock
     40 from tqdm.auto import tqdm

/opt/conda/lib/python3.7/site-packages/pyarrow/compute.py in <module>
     16 # under the License.
     17
---> 18 from pyarrow._compute import (  # noqa
     19     Function,
     20     FunctionOptions,

ImportError: /opt/conda/lib/python3.7/site-packages/pyarrow/_compute.cpython-37m-x86_64-linux-gnu.so: undefined symbol: _ZNK5arrow7compute15KernelSignature8ToStringEv
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Kaggle
- Python version: 3.7.10
- PyArrow version: 4.0.1


Error when downloading C4

Something isn't working

Hi,
I am trying to download `en` corpus from C4 dataset. However, I get an error caused by validation files download (see image). My code is very primitive:
`datasets.load_dataset('c4', 'en')`

Is this a bug or do I have some configurations missing on my server?
Thanks!

<img width="1014" alt="Снимок экрана 2021-07-20 в 11 37 17" src="https://user-images.githubusercontent.com/36672861/126289448-6e0db402-5f3f-485a-bf74-eb6e0271fc25.png">


Increase json reader block_size automatically

Currently some files can't be read with the default parameters of the JSON lines reader.
For example this one:
https://huggingface.co/datasets/thomwolf/codeparrot/resolve/main/file-000000000006.json.gz

raises a pyarrow error:
```python
ArrowInvalid: straddling object straddles two block boundaries (try to increase block size?)
```

The block size that is used is the default one by pyarrow (related to this [jira issue](https://issues.apache.org/jira/browse/ARROW-9612)).

To fix this issue I changed the block_size to increase automatically if there is a straddling issue when parsing a batch of json lines.

By default the value is `chunksize // 32` in order to leverage multithreading, and it doubles every time a straddling issue occurs. The block_size is then reset for each file.

cc @thomwolf @albertvillanova


Parallelize ETag requests

Since https://github.com/huggingface/datasets/pull/2628 we use the ETag or the remote data files to compute the directory in the cache where a dataset is saved. This is useful in order to reload the dataset from the cache only if the remote files haven't changed.

In this I made the ETag requests parallel using multithreading. There is also a tqdm progress bar that shows up if there are more than 16 data files.


Fix sacrebleu parameter name

DONE:
- Fix parameter name: `smooth` to `smooth_method`.
- Improve kwargs description.
- Align docs on using a metric.
- Add example of passing additional arguments in using metrics.

Related to #2669.


Fix potential DuplicatedKeysError in SQuAD

DONE:
- Fix potential DiplicatedKeysError by ensuring keys are unique.
- Align examples in the docs with SQuAD code.

We should promote as a good practice, that the keys should be programmatically generated as unique, instead of read from data (which might be not unique).


Fix potential DuplicatedKeysError in LibriSpeech

DONE:
- Fix unnecessary path join.
- Fix potential DiplicatedKeysError by ensuring keys are unique.

We should promote as a good practice, that the keys should be programmatically generated as unique, instead of read from data (which might be not unique).


Mesinesp development and training data sets have been added.

https://zenodo.org/search?page=1&size=20&q=mesinesp, Mesinesp has Medical Semantic Indexed records in Spanish.  Indexing is done using DeCS codes, a sort of Spanish equivalent to MeSH terms.
The Mesinesp (Spanish BioASQ track, see https://temu.bsc.es/mesinesp) development set has a total of 750 records.
The Mesinesp (Spanish BioASQ track, see https://temu.bsc.es/mesinesp) training set has a total of 369,368 records.


Using sharding to parallelize indexing

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Creating an elasticsearch index on large dataset could be quite long and cannot be parallelized on shard (the index creation is colliding)

**Describe the solution you'd like**
When working on dataset shards, if an index already exists, its mapping should be checked and if compatible, the indexing process should continue with the shard data.

Additionally, at the end of the process, the `_indexes` dict should be send back to the original dataset object (from which the shards have been created) to allow to use the index for later filtering on the whole dataset.

**Describe alternatives you've considered**
Each dataset shard could created independent partial indices. then on the whole dataset level, indices should be all referred in `_indexes` dict and be used in querying through `get_nearest_examples()`. The drawback is that the scores will be computed independently on the partial indices leading to inconsistent values for most scoring based on corpus level statistics (tf/idf, BM25).

**Additional context**
The objectives is to parallelize the index creation to speed-up the process (ie surcharging the ES server which is fine to handle large load) while later enabling search on the whole dataset.


Metric kwargs are not passed to underlying external metric f1_score

bug

Something isn't working

## Describe the bug
When I want to use F1 score with average="min", this keyword argument does not seem to be passed through to the underlying sklearn metric. This is evident because [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) throws an error telling me so.

## Steps to reproduce the bug
```python
import datasets
f1 = datasets.load_metric("f1", keep_in_memory=True, average="min")
f1.add_batch(predictions=[0,2,3], references=[1, 2, 3])
f1.compute()
```

## Expected results
No error, because `average="min"` should be passed correctly to f1_score in sklearn.

## Actual results

```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\datasets\metric.py", line 402, in compute
    output = self._compute(predictions=predictions, references=references, **kwargs)
  File "C:\Users\bramv\.cache\huggingface\modules\datasets_modules\metrics\f1\82177930a325d4c28342bba0f116d73f6d92fb0c44cd67be32a07c1262b61cfe\f1.py", line 97, in _compute
    "f1": f1_score(
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\utils\validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\metrics\_classification.py", line 1071, in f1_score
    return fbeta_score(y_true, y_pred, beta=1, labels=labels,
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\utils\validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\metrics\_classification.py", line 1195, in fbeta_score
    _, _, f, _ = precision_recall_fscore_support(y_true, y_pred,
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\utils\validation.py", line 63, in inner_f
    return f(*args, **kwargs)
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\metrics\_classification.py", line 1464, in precision_recall_fscore_support
    labels = _check_set_wise_labels(y_true, y_pred, average, labels,
  File "C:\Users\bramv\.virtualenvs\pipeline-TpEsXVex\lib\site-packages\sklearn\metrics\_classification.py", line 1294, in _check_set_wise_labels
    raise ValueError("Target is %s but average='binary'. Please "
ValueError: Target is multiclass but average='binary'. Please choose another average setting, one of [None, 'micro', 'macro', 'weighted'].
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Windows-10-10.0.19041-SP0
- Python version: 3.9.2
- PyArrow version: 4.0.1


Add Russian SuperGLUE

Hi,

This adds the [Russian SuperGLUE](https://russiansuperglue.com/) dataset. For the most part I reused the code for the original SuperGLUE, although there are some relatively minor differences in the structure that I accounted for.


Use tqdm from tqdm_utils

This PR replaces `tqdm` from the `tqdm` lib with `tqdm` from `datasets.utils.tqdm_utils`. With this change, it's possible to disable progress bars just by calling `disable_progress_bar`. Note this doesn't work on Windows when using multiprocessing due to how global variables are shared between processes. Currently, there is no easy way to disable progress bars in a multiprocess setting on Windows (patching logging with `datasets.utils.logging.get_verbosity = lambda: datasets.utils.logging.NOTSET` doesn't seem to work as well), so adding support for this is a future goal. Additionally, this PR adds a unit ("ba" for batches) to the bar printed by `Dataset.to_json` (this change is motivated by https://github.com/huggingface/datasets/issues/2657).


Adds CodeClippy dataset [WIP]

CodeClippy is an opensource code dataset scrapped from github during flax-jax-community-week
https://the-eye.eu/public/AI/training_data/code_clippy_data/


Adds APPS dataset to the hub [WIP]

A loading script for [APPS dataset](https://github.com/hendrycks/apps)


[`to_json`] add multi-proc sharding support

enhancement

New feature or request

As discussed on slack it appears that `to_json` is quite slow on huge datasets like OSCAR.

I implemented sharded saving, which is much much faster - but the tqdm bars all overwrite each other, so it's hard to make sense of the progress, so if possible ideally this multi-proc support could be implemented internally in `to_json` via `num_proc` argument. I guess `num_proc` will be the number of shards?

I think the user will need to use this feature wisely, since too many processes writing to say normal style HD is likely to be slower than one process.

I'm not sure whether the user should be responsible to concatenate the shards at the end  or `datasets`, either way works for my needs.

The code I was using:

```
from multiprocessing import cpu_count, Process, Queue

[...]

filtered_dataset = concat_dataset.map(filter_short_documents, batched=True, batch_size=256, num_proc=cpu_count())

DATASET_NAME = "oscar"
SHARDS = 10
def process_shard(idx):
    print(f"Sharding {idx}")
    ds_shard = filtered_dataset.shard(SHARDS, idx, contiguous=True)
    # ds_shard = ds_shard.shuffle() # remove contiguous=True above if shuffling
    print(f"Saving {DATASET_NAME}-{idx}.jsonl")
    ds_shard.to_json(f"{DATASET_NAME}-{idx}.jsonl", orient="records", lines=True, force_ascii=False)

queue = Queue()
processes = [Process(target=process_shard, args=(idx,)) for idx in range(SHARDS)]
for p in processes:
    p.start()

for p in processes:
    p.join()
```

Thank you!

@lhoestq


Load Dataset from the Hub (NO DATASET SCRIPT)

## Load the data from any Dataset repository on the Hub

This PR adds support for loading datasets from any dataset repository on the hub, without requiring any dataset script.

As a user it's now possible to create a repo and upload some csv/json/text/parquet files, and then be able to load the data in one line. Here is an example with the `allenai/c4` repository that contains a lot of compressed json lines files:
```python
from datasets import load_dataset

data_files = {"train": "en/c4-train.*.json.gz"}
c4 = load_dataset("allenai/c4", data_files=data_files, split="train", streaming=True)

print(c4.n_shards)
# 1024
print(next(iter(c4)))
# {'text': 'Beginners BBQ Class Takin...'}
```

By default it loads all the files, but as shown in the example you can choose the ones you want with unix style patterns.

Of course it's still possible to use dataset scripts since they offer the most flexibility.

## Implementation details

It uses `huggingface_hub` to list the files in a dataset repository.

If you provide a path to a local directory instead of a repository name, it works the same way but it uses `glob`.

Depending on the data files available, or passed in the `data_files` parameter, one of the available builders will be used among the csv, json, text and parquet builders.

Because of this, it's not possible to load both csv and json files at once. In this case you have to load them separately and then concatenate the two datasets for example.

## TODO

- [x] tests
- [x] docs
- [x] when huggingface_hub gets a new release, update the CI and the setup.py

Close https://github.com/huggingface/datasets/issues/2629


Add SD task for SUPERB

Include the SD (Speaker Diarization) task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#sd-speaker-diarization).

TODO:
- [x] Generate the LibriMix corpus
- [x] Prepare the corpus for diarization
- [x] Upload these files to the superb-data repo
- [x] Transcribe the corresponding s3prl processing of these files into our superb loading script
- [x] README: tags + description sections
- ~~Add DER metric~~ (we leave the DER metric for a follow-up PR)

Related to #2619.

Close #2653.

cc: @lewtun


Move checks from _map_single to map

The goal of this PR is to remove duplicated checks in the `map` logic to execute them only once whenever possible (`fn_kwargs`, `input_columns`, ...). Additionally, this PR improves the consistency (to align it with `input_columns`) of the `remove_columns` check by adding support for a single string value, which is then wrapped into a list.


Allow dataset config kwargs to be None

Close https://github.com/huggingface/datasets/issues/2658

The dataset config kwargs that were set to None we simply ignored.
This was an issue when None has some meaning for certain parameters of certain builders, like the `sep` parameter of the "csv" builder that allows to infer to separator.

cc @SBrandeis


Can't pass `sep=None` to load_dataset("csv", ...) to infer the separator via pandas.read_csv

When doing `load_dataset("csv", sep=None)`, the `sep` passed to `pd.read_csv` is still the default `sep=","` instead, which makes it impossible to make the csv loader infer the separator.

Related to https://github.com/huggingface/datasets/pull/2656

cc @SBrandeis


`to_json` reporting enhancements

New feature or request

While using `to_json` 2 things came to mind that would have made the experience easier on the user:

1. Could we have a `desc` arg for the tqdm use and a fallback to just `to_json` so that it'd be clear to the user what's happening? Surely, one can just print the description before calling json, but I thought perhaps it'd help to have it self-identify like you did for other progress bars recently.

2. It took me a while to make sense of the reported numbers:
```
 22%|██▏       | 1536/7076 [12:30:57<44:09:42, 28.70s/it]
```
So iteration here happens to be 10K samples, and the total is 70M records. But the user does't know that, so the progress bar is perfect, but the numbers it reports are meaningless until one discovers that 1it=10K samples. And one still has to convert these in the head - so it's not quick. Not exactly sure what's the best way to approach this, perhaps it can be part of `desc`? or report M or K, so it'd be built-in if it were to print, e.g.:
```
 22%|██▏       | 15360K/70760K [12:30:57<44:09:42, 28.70s/it]
```
or
```
 22%|██▏       | 15.36M/70.76M [12:30:57<44:09:42, 28.70s/it]
```
(while of course remaining friendly to small datasets)

I forget if tqdm lets you add a magnitude identifier to the running count.

Thank you!


Change `from_csv` default arguments

Passing `sep=None` to pandas's `read_csv` lets pandas guess the CSV file's separator

This PR allows users to use this pandas's feature by passing `sep=None` to `Dataset.from_csv`:

```python
Dataset.from_csv(
    ...,
    sep=None
)
```


Allow the selection of multiple columns at once

New feature or request

**Is your feature request related to a problem? Please describe.**

Similar to pandas, it would be great if we could select multiple columns at once.

**Describe the solution you'd like**
```python
my_dataset = ...  # Has columns ['idx', 'sentence', 'label']
idx, label = my_dataset[['idx', 'label']]
```

**Describe alternatives you've considered**
we can do `[dataset[col] for col in ('idx', 'label')]`

**Additional context**
This is of course very minor.


Give a user feedback if the dataset he loads is streamable or not

New feature or request

**Is your feature request related to a problem? Please describe.**
I would love to know if a `dataset` is with the current implementation streamable or not.

**Describe the solution you'd like**
We could show a warning when a dataset is loaded with `load_dataset('...',streaming=True)` when its lot streamable, e.g. if it is an archive.

**Describe alternatives you've considered**
Add a new metadata tag for "streaming"


Add SD task for SUPERB

dataset request

Requesting to add a new dataset

1.11

Next minor release

Include the SD (Speaker Diarization) task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/master/s3prl/downstream#sd-speaker-diarization).

Steps:
- [x] Generate the LibriMix corpus
- [x] Prepare the corpus for diarization
- [x] Upload these files to the superb-data repo
- [x] Transcribe the corresponding s3prl processing of these files into our superb loading script
- [ ] README: tags + description sections

Related to #2619.

cc: @lewtun


Fix logging docstring

Remove "no tqdm bars" from the docstring in the logging module to align it with the changes introduced in #2534.


Setting log level higher than warning does not suppress progress bar

bug

Something isn't working

## Describe the bug
I would like to disable progress bars for `.map` method (and other methods like `.filter` and `load_dataset` as well).
According to #1627 one can suppress it by setting log level higher than `warning`, however doing so doesn't suppress it with version 1.9.0.

I also tried to set `DATASETS_VERBOSITY` environment variable to `error` or `critical` but it also didn't work.

## Steps to reproduce the bug
```python
import datasets

from datasets.utils.logging import set_verbosity_error

set_verbosity_error()

def dummy_map(batch):
    return batch

common_voice_train = datasets.load_dataset("common_voice", "de", split="train")
common_voice_test = datasets.load_dataset("common_voice", "de", split="test")

common_voice_train.map(dummy_map)
```

## Expected results
- The progress bar for `.map` call won't be shown

## Actual results
- The progress bar for `.map` is still shown

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-5.4.0-1045-aws-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.7.5
- PyArrow version: 4.0.1


[load_dataset] shard and parallelize the process

enhancement

New feature or request

- Some huge datasets take forever to build the first time. (e.g. oscar/en) as it's done in a single cpu core.
- If the build crashes, everything done up to that point gets lost

Request: Shard the build over multiple arrow files, which would enable:
- much faster build by parallelizing the build process
- if the process crashed, the completed arrow files don't need to be re-built again

Thank you!

@lhoestq


adding progress bar / ETA for `load_dataset`

New feature or request

Please consider:
```
Downloading and preparing dataset oscar/unshuffled_deduplicated_en (download: 462.40 GiB, generated: 1.18 TiB, post-processed: Unknown size, total: 1.63 TiB) to cache/oscar/unshuffled_deduplicated_en/1.0.0/84838bd49d2295f62008383b05620571535451d84545037bb94d6f3501651df2...
HF google storage unreachable. Downloading and preparing it from source
```
and no indication whatsoever of whether things work well or when it'll be done. It's important to have an estimated completion time for when doing slurm jobs since some instances have a cap on run-time.

I think for this particular job it sat for 30min in total silence and then after 30min it started generating:
```
897850 examples [07:24, 10286.71 examples/s]
```
which is already great!

Request:
1. ETA - knowing how many hours to allocate for a slurm job
2. progress bar - helps to know things are working and aren't stuck and where we are at.

Thank you!

@lhoestq


Add web_split dataset for Paraphase and Rephrase benchmark

New feature or request

## Describe:
For getting simple sentences from complex sentence there are dataset and task like wiki_split that is available in hugging face datasets. This web_split is a very similar dataset. There some research paper which states that by combining these two datasets we if we train the model it will yield better results on both tests data.

This dataset is made from web NLG data.

All the dataset related details are provided in the below repository

Github link: https://github.com/shashiongithub/Split-and-Rephrase


Fix anchor in README

1.10

Next minor release

I forgot to push this fix in #2611, so I'm sending it now.


downloading of yahoo_answers_topics dataset failed

bug

Something isn't working

## Describe the bug
I get an error datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files when I try to download the yahoo_answers_topics dataset

## Steps to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')
# Sample code to reproduce the bug
   self.dataset = load_dataset(
                'yahoo_answers_topics', cache_dir=self.config['yahoo_cache_dir'], split='train[:90%]')

## Expected results
A clear and concise description of the expected results.

## Actual results
Specify the actual results or traceback.
datasets.utils.info_utils.NonMatchingChecksumError: Checksums didn't match for dataset source files


load_dataset processing failed with OS error after downloading a dataset

Something isn't working

## Describe the bug
After downloading a dataset like opus100, there is a bug that
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS

## Steps to reproduce the bug
```python
from datasets import load_dataset
this_dataset = load_dataset('opus100', 'af-en')
```

## Expected results
there is no error when running load_dataset.

## Actual results
Specify the actual results or traceback.

Traceback (most recent call last):
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 652, in _download_and_prep
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 989, in _prepare_split
    example = self.info.features.encode_example(record)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 952, in encode_example
    example = cast_to_python_objects(example)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 219, in cast_to_python_ob
    return _cast_to_python_objects(obj)[0]
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/features.py", line 165, in _cast_to_python_o
    import torch
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 188, in <module>
    _load_global_deps()
  File "/home/anaconda3/lib/python3.6/site-packages/torch/__init__.py", line 141, in _load_global_deps
    ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
  File "/home/anaconda3/lib/python3.6/ctypes/__init__.py", line 348, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: dlopen: cannot load any more object with static TLS

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "download_hub_opus100.py", line 9, in <module>
    this_dataset = load_dataset('opus100', language_pair)
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 575, in download_and_prepa
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/anaconda3/lib/python3.6/site-packages/datasets/builder.py", line 658, in _download_and_prep
    + str(e)
OSError: Cannot find data file.
Original error:
dlopen: cannot load any more object with static TLS

## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-3.13.0-32-generic-x86_64-with-debian-jessie-sid
- Python version: 3.6.6
- PyArrow version: 3.0.0


Batched `map` not allowed to return 0 items

Something isn't working

## Describe the bug
I'm trying to use `map` to filter a large dataset by selecting rows that match an expensive condition (files referenced by one of the columns need to exist in the filesystem, so we have to `stat` them). According to [the documentation](https://huggingface.co/docs/datasets/processing.html#augmenting-the-dataset), `a batch mapped function can take as input a batch of size N and return a batch of size M where M can be greater or less than N and can even be zero`.

However, when the returned batch has a size of zero (neither item in the batch fulfilled the condition), we get an `index out of bounds` error. I think that `arrow_writer.py` is [trying to infer the returned types using the first element returned](https://github.com/huggingface/datasets/blob/master/src/datasets/arrow_writer.py#L100), but no elements were returned in this case.

For this error to happen, I'm returning a dictionary that contains empty lists for the keys I want to keep, see below. If I return an empty dictionary instead (no keys), then a different error eventually occurs.

## Steps to reproduce the bug
```python
def select_rows(examples):
    # `key` is a column name that exists in the original dataset
    # The following line simulates no matches found, so we return an empty batch
    result = {'key': []}
    return result

filtered_dataset = dataset.map(
    select_rows,
    remove_columns = dataset.column_names,
    batched = True,
    num_proc = 1,
    desc = "Selecting rows with images that exist"
)
```

The code above immediately triggers the exception. If we use the following instead:

```python
def select_rows(examples):
    # `key` is a column name that exists in the original dataset
    result = {'key': []}   # or defaultdict or whatever

    # code to check for condition and append elements to result
    # some_items_found will be set to True if there were any matching elements in the batch

    return result if some_items_found else {}
```

Then it _seems_ to work, but it eventually fails with some sort of schema error. I believe it may happen when an empty batch is followed by a non-empty one, but haven't set up a test to verify it.

In my opinion, returning a dictionary with empty lists and valid column names should be accepted as a valid result with zero items.

## Expected results
The dataset would be filtered and only the matching fields would be returned.

## Actual results
An exception is encountered, as described. Using a workaround makes it fail further along the line.

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.1.dev0
- Platform: Linux-5.4.0-53-generic-x86_64-with-glibc2.17
- Python version: 3.8.10
- PyArrow version: 4.0.1


Enum used in map functions will raise a RecursionError with dill.

Something isn't working

## Describe the bug

Enums used in functions pass to `map` will fail at pickling with a maximum recursion exception as described here: https://github.com/uqfoundation/dill/issues/250#issuecomment-852566284

In my particular case, I use an enum to define an argument with fixed options using the `TraininigArguments` dataclass as base class and the `HfArgumentParser`. In the same file I use a `ds.map` that tries to pickle the content of the module including the definition of the enum that runs into the dill bug described above.

## Steps to reproduce the bug
```python
from datasets import load_dataset
from enum import Enum

class A(Enum):
    a = 'a'

def main():
    a = A.a

    def f(x):
        return {} if a == a.a else x

    ds = load_dataset('cnn_dailymail', '3.0.0')['test']
    ds = ds.map(f, num_proc=15)

if __name__ == "__main__":
    main()
```

## Expected results
The known problem with dill could be prevented as explained in the link above (workaround.) Since `HFArgumentParser` nicely uses the enum class for choices it makes sense to also deal with this bug under the hood.

## Actual results

```python
  File "/home/xxxx/miniconda3/lib/python3.8/site-packages/dill/_dill.py", line 1373, in save_type
    pickler.save_reduce(_create_type, (type(obj), obj.__name__,
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 690, in save_reduce
    save(args)
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 558, in save
    f(self, obj)  # Call unbound method with explicit self
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 899, in save_tuple
    save(element)
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 534, in save
    self.framer.commit_frame()
  File "/home/xxxx/miniconda3/lib/python3.8/pickle.py", line 220, in commit_frame
    if f.tell() >= self._FRAME_SIZE_TARGET or force:
RecursionError: maximum recursion depth exceeded while calling a Python object
```

## Environment info
- `datasets` version: 1.8.0
- Platform: Linux-5.9.0-4-amd64-x86_64-with-glibc2.10
- Python version: 3.8.5
- PyArrow version: 3.0.0


Support multi-worker with streaming dataset (IterableDataset).

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
The current `.map` does not support multi-process, CPU can become bottleneck if the pre-processing is complex (e.g. t5 span masking).

**Describe the solution you'd like**
Ideally `.map` should support multi-worker like tfds, with `AUTOTUNE`.

**Describe alternatives you've considered**
A simpler solution is to shard the dataset and process it in parallel with pytorch dataloader. The shard does not need to be of equal size.
* https://pytorch.org/docs/stable/data.html#torch.utils.data.IterableDataset

**Additional context**


load_dataset("financial_phrasebank") NonMatchingChecksumError

bug

Something isn't working

## Describe the bug
Attempting to download the financial_phrasebank dataset results in a NonMatchingChecksumError

## Steps to reproduce the bug
```python
from datasets import load_dataset
dataset = load_dataset("financial_phrasebank", 'sentences_allagree')
```

## Expected results
I expect to see the financial_phrasebank dataset downloaded successfully

## Actual results
NonMatchingChecksumError: Checksums didn't match for dataset source files:
['https://www.researchgate.net/profile/Pekka_Malo/publication/251231364_FinancialPhraseBank-v10/data/0c96051eee4fb1d56e000000/FinancialPhraseBank-v10.zip']

## Environment info
- `datasets` version: 1.9.0
- Platform: Linux-4.14.232-177.418.amzn2.x86_64-x86_64-with-debian-10.6
- Python version: 3.7.10
- PyArrow version: 4.0.1


Fix docstrings

1.10

Next minor release

Fix rendering of some docstrings.


Refactor patching to specific submodule

Minor reorganization of the code, so that additional patching functions (not related to streaming) might be created.

In relation with the initial approach followed in #2631.


Streaming for the Json loader

It was not using `open` in the builder. Therefore `pyarrow.json.read_json` was downloading the full file to start yielding rows.

Moreover, it appeared that `pyarrow.json.read_json` was not really suited for streaming as it was downloading too much data and failing if `block_size` was not properly configured (related to #2573).

So I switched to using `open` which is extended to support reading from remote file progressively, and I removed the pyarrow json reader which was not practical.
Instead, I'm using the classical `json.loads` from the standard library.


Add the CIDEr metric?

enhancement

New feature or request

Hi,
I find the api in https://huggingface.co/metrics  quite useful.
I am playing around with video/image captioning task, where CIDEr is a popular metric.
Do you plan to add this into the HF ```datasets``` library?
Thanks.


Streaming for the Pandas loader

It was not using open in the builder. Therefore pd.read_pickle could fail when streaming from a private repo for example.

Indeed, when streaming, open is extended to support reading from remote files and handles authentication to the HF Hub


Streaming for the CSV loader

It was not using `open` in the builder. Therefore `pd.read_csv` was downloading the full file to start yielding rows.

Indeed, when streaming, `open` is extended to support reading from remote file progressively.


Inject ASR template for lj_speech dataset

1.10

Next minor release

Related to: #2565, #2633.

cc: @lewtun


Update ASR tags

1.10

Next minor release

This PR updates the ASR tags of the 5 datasets added in #2565 following the change of task categories in #2620


add image-classification task template

Snippet below is the tl;dr, but you can try it out directly here:

[![Open In Collab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/gist/nateraw/005c025d41f0e48ae3d4ee61c0f20b70/image-classification-task-template-demo.ipynb)

```python
from datasets import load_dataset

ds = load_dataset('nateraw/image-folder', data_files='PetImages/')
# DatasetDict({
#     train: Dataset({
#         features: ['file', 'labels'],
#         num_rows: 23410
#     })
# })

ds = ds.prepare_for_task('image-classification')
# DatasetDict({
#     train: Dataset({
#         features: ['image_file_path', 'labels'],
#         num_rows: 23410
#     })
# })
```


Delete extracted files when loading dataset

Close #2481, close #2604, close #2591.

cc: @stas00, @thomwolf, @BirgerMoell


Progress bars are not properly rendered in Jupyter notebook

bug

Something isn't working

## Describe the bug
The progress bars are not Jupyter widgets; regular progress bars appear (like in a terminal).

## Steps to reproduce the bug
```python
ds.map(tokenize, num_proc=10)
```

## Expected results
Jupyter widgets displaying the progress bars.

## Actual results
Simple plane progress bars.

cc: Reported by @thomwolf


Load datasets from the Hub without requiring a dataset script

As a user I would like to be able to upload my csv/json/text/parquet/etc. files in a dataset repository on the Hugging Face Hub and be able to load this dataset with `load_dataset` without having to implement a dataset script.

Moreover I would like to be able to specify which file goes into which split using the `data_files` argument.

This feature should be compatible with private repositories and dataset streaming.

This can be implemented by checking the extension of the files in the dataset repository and then by using the right dataset builder that is already packaged in the library (csv/json/text/parquet/etc.)


Use ETag of remote data files

1.10

Next minor release

Use ETag of remote data files to create config ID.

Related to #2616.


Minor fix tests with Windows paths

1.10

Next minor release

Minor fix tests with Windows paths.


Use correct logger in metrics.py

1.10

Next minor release

Fixes #2624


can't set verbosity for `metric.py`

Something isn't working

## Describe the bug
```
[2021-07-10 20:13:11,528][datasets.utils.filelock][INFO] - Lock 139705371374976 acquired on /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.lock
[2021-07-10 20:13:11,529][datasets.arrow_writer][INFO] - Done writing 32 examples in 6100 bytes /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow.
[2021-07-10 20:13:11,531][datasets.arrow_dataset][INFO] - Set __getitem__(key) output type to python objects for no columns  (when key is int or slice) and don't output other (un-formatted) columns.
[2021-07-10 20:13:11,543][/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric.py][INFO] - Removing /root/.cache/huggingface/metrics/seqeval/default/default_experiment-1-0.arrow
```
As you can see, `datasets` logging come from different places.
`filelock`, `arrow_writer` & `arrow_dataset` comes from `datasets.*` which are expected
However, `metric.py` logging comes from `/conda/envs/myenv/lib/python3.8/site-packages/datasets/`

So when setting `datasets.utils.logging.set_verbosity_error()`,  it still logs the last message which is annoying during evaluation.

I had to do
```
logging.getLogger("/conda/envs/myenv/lib/python3.8/site-packages/datasets/metric").setLevel(logging.ERROR)
```
to fully mute these messages

## Expected results
it shouldn't log these messages when setting `datasets.utils.logging.set_verbosity_error()`

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: tried both 1.8.0 & 1.9.0
- Platform: Ubuntu 18.04.5 LTS
- Python version: 3.8.10
- PyArrow version: 3.0.0


[Metrics] added wiki_split metrics

Fixes: #2606

This pull request adds combine metrics for the wikisplit or English sentence split task

Reviewer: @patrickvonplaten


Integration with AugLy

enhancement

New feature or request

**Is your feature request related to a problem? Please describe.**
Facebook recently launched a library, [AugLy](https://github.com/facebookresearch/AugLy) , that has a unified API for augmentations for image, video and text.

It would be pretty exciting to have it hooked up to HF libraries so that we can make NLP models robust to misspellings or to punctuation, or emojis etc. Plus, with Transformers supporting more CV use cases, having augmentations support becomes crucial.

**Describe the solution you'd like**
The biggest difference between augmentations and preprocessing is that preprocessing happens only once, but you are running augmentations once per epoch. AugLy operates on text directly, so this breaks the typical workflow where we would run the tokenizer once, set format to pt tensors and be ready for the Dataloader.

**Describe alternatives you've considered**

One possible way of implementing these is to make a custom Dataset class where getitem(i) runs the augmentation and the tokenizer every time, though this would slow training down considerably given we wouldn't even run the tokenizer in batches.


Use prefix to allow exceed Windows MAX_PATH

By using this prefix, you can exceed the Windows MAX_PATH limit.

See: https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file?redirectedfrom=MSDN#win32-file-namespaces

Related to #2524, #2220.


Add speech processing tasks

This PR replaces the `automatic-speech-recognition` task category with a broader `speech-processing` category.

The tasks associated with this category are derived from the [SUPERB benchmark](https://arxiv.org/abs/2105.01051), and ASR is included in this set.


Add ASR task for SUPERB

1.10

Next minor release

This PR starts building up the SUPERB benchmark by including the ASR task as described in the [SUPERB paper](https://arxiv.org/abs/2105.01051) and `s3prl` [instructions](https://github.com/s3prl/s3prl/tree/v0.2.0/downstream#asr-automatic-speech-recognition).

Usage:

```python
from datasets import load_dataset

asr = load_dataset("superb", "asr")
# DatasetDict({
#     train: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 28539
#     })
#     validation: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 2703
#     })
#     test: Dataset({
#         features: ['file', 'text', 'speaker_id', 'chapter_id', 'id'],
#         num_rows: 2620
#     })
# })
```

I've used the GLUE benchmark as a guide for filling out the README.

To move fast during the evaluation PoC I propose to merge one task at a time, so we can continue building the training / evaluation framework in parallel.

Note: codewise this PR is ready for review - I'll add the missing YAML tags once #2620 is merged :)


`filelock.py` Error

bug

Something isn't working

## Describe the bug

It seems that the `filelock.py` went error.

```
>>> ds=load_dataset('xsum')

^CTraceback (most recent call last):
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 402, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
OSError: [Errno 37] No locks available
```

According to error log, it is OSError, but there is an `except` in the `_acquire` function.

```
    def _acquire(self):
        open_mode = os.O_WRONLY | os.O_CREAT | os.O_EXCL | os.O_TRUNC
        try:
            fd = os.open(self._lock_file, open_mode)
        except (IOError, OSError):
            pass
        else:
            self._lock_file_fd = fd
        return None
```

I don't know why it stucked rather than `pass` directly.

I am not quite familiar with filelock operation, so any help is highly appriciated.

## Steps to reproduce the bug
```python

ds = load_dataset('xsum')
```

## Expected results
A clear and concise description of the expected results.

## Actual results
```
>>> ds=load_dataset('xsum')

^CTraceback (most recent call last):
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 402, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
OSError: [Errno 37] No locks available

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/load.py", line 818, in load_dataset
    use_auth_token=use_auth_token,
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/load.py", line 470, in prepare_module
    with FileLock(lock_path):
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 323, in __enter__
    self.acquire()
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 272, in acquire
    self._acquire()
  File "/user/HS502/yl02706/.conda/envs/lyc/lib/python3.6/site-packages/datasets/utils/filelock.py", line 402, in _acquire
    fcntl.flock(fd, fcntl.LOCK_EX | fcntl.LOCK_NB)
KeyboardInterrupt
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.0
- Platform: Linux-4.15.0-135-generic-x86_64-with-debian-buster-sid
- Python version: 3.6.13
- PyArrow version: 4.0.1


Fix missing EOL issue in to_json for old versions of pandas

1.10

Next minor release

Some versions of pandas don't add an EOL at the end of the output of `to_json`.
Therefore users could end up having two samples in the same line

Close https://github.com/huggingface/datasets/issues/2615


Support remote data files

enhancement

New feature or request

1.10

Next minor release

Add support for (streaming) remote data files:

```python
data_files = f"https://huggingface.co/datasets/{repo_id}/resolve/main/{relative_file_path}"
ds = load_dataset("json", split="train", data_files=data_files, streaming=True)
```

cc: @thomwolf


Jsonlines export error

bug

Something isn't working

## Describe the bug
When exporting large datasets in jsonlines (c4 in my case) the created file has an error every 9999 lines: the 9999th and 10000th are concatenated, thus breaking the jsonlines format. This sounds like it is related to batching, which is by 10000 by default

## Steps to reproduce the bug
This what I'm running:

in python:

```
from datasets import load_dataset
ptb = load_dataset("ptb_text_only")
ptb["train"].to_json("ptb.jsonl")
```

then out of python:

```
head -10000 ptb.jsonl
```

## Expected results
Properly separated lines

## Actual results
The last line is a concatenation of two lines

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->

- `datasets` version: 1.9.1.dev0
- Platform: Linux-5.4.0-1046-gcp-x86_64-with-Ubuntu-18.04-bionic
- Python version: 3.6.9
- PyArrow version: 4.0.1


Convert numpy scalar to python float in Pearsonr output

1.10

Next minor release

Following of https://github.com/huggingface/datasets/pull/2612


Use ndarray.item instead of ndarray.tolist

1.10

Next minor release

This PR follows up on #2612 to use `numpy.ndarray.item` instead of `numpy.ndarray.tolist` as the latter is somewhat confusing to the developer (even though it works).

Judging from the `numpy` docs, `ndarray.item` is closer to what we want: https://numpy.org/doc/stable/reference/generated/numpy.ndarray.item.html#numpy-ndarray-item

PS. Sorry for the duplicate work here. I should have read the numpy docs more carefully in #2612


Return Python float instead of numpy.float64 in sklearn metrics

1.10

Next minor release

This PR converts the return type of all `sklearn` metrics to be Python `float` instead of `numpy.float64`.

The reason behind this is that our Hub evaluation framework relies on converting benchmark-specific metrics to YAML ([example](https://huggingface.co/datasets/autonlp/autonlp-benchmark-raft-neelalex__raft-test-neelalex__raft-predictions-3/blob/main/README.md#L11)) and the `numpy.float64` format produces garbage like:

```python
import yaml
from datasets import load_metric

metric = load_metric("accuracy")
score = metric.compute(predictions=[0,1], references=[0,1])
print(yaml.dump(score["accuracy"])) # output below
# !!python/object/apply:numpy.core.multiarray.scalar
# - !!python/object/apply:numpy.dtype
#   args:
#   - f8
#   - false
#   - true
#   state: !!python/tuple
#   - 3
#   - <
#   - null
#   - null
#   - null
#   - -1
#   - -1
#   - 0
# - !!binary |
#   AAAAAAAA8D8=
```


More consistent naming

As per @stas00's suggestion in #2500, this PR inserts a space between the logo and the lib name (`🤗Datasets` -> `🤗 Datasets`) for consistency with the Transformers lib. Additionally, more consistent names are used for Datasets Hub, etc.


Add missing WikiANN language tags

1.10

Next minor release

Add missing language tags for WikiANN datasets.


Fix potential DuplicatedKeysError

1.10

Next minor release

Fix potential DiplicatedKeysError by ensuring keys are unique.

We should promote as a good practice, that the keys should be programmatically generated as unique, instead of read from data (which might be not unique).


Support streaming JSON files

1.10

Next minor release

Use open in JSON dataset builder, so that it can be patched with xopen for streaming.

Close #2607.


Streaming local gzip compressed JSON line files is not working

Something isn't working

## Describe the bug
Using streaming to iterate on local gzip compressed JSON files raise a file not exist error

## Steps to reproduce the bug
```python
from datasets import load_dataset

streamed_dataset = load_dataset('json', split='train', data_files=data_files, streaming=True)

next(iter(streamed_dataset))
```

## Actual results
```
FileNotFoundError                         Traceback (most recent call last)
<ipython-input-6-27a664e29784> in <module>
----> 1 next(iter(streamed_dataset))

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in __iter__(self)
    336
    337     def __iter__(self):
--> 338         for key, example in self._iter():
    339             if self.features:
    340                 # we encode the example for ClassLabel feature types for example

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in _iter(self)
    333         else:
    334             ex_iterable = self._ex_iterable
--> 335         yield from ex_iterable
    336
    337     def __iter__(self):

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in __iter__(self)
     76
     77     def __iter__(self):
---> 78         for key, example in self.generate_examples_fn(**self.kwargs):
     79             yield key, example
     80

~/Documents/GitHub/datasets/src/datasets/iterable_dataset.py in wrapper(**kwargs)
    282     def wrapper(**kwargs):
    283         python_formatter = PythonFormatter()
--> 284         for key, table in generate_tables_fn(**kwargs):
    285             batch = python_formatter.format_batch(table)
    286             for i, example in enumerate(_batch_to_examples(batch)):

~/Documents/GitHub/datasets/src/datasets/packaged_modules/json/json.py in _generate_tables(self, files, original_files)
     85                         file,
     86                         read_options=self.config.pa_read_options,
---> 87                         parse_options=self.config.pa_parse_options,
     88                     )
     89                 except pa.ArrowInvalid as err:

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/_json.pyx in pyarrow._json.read_json()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/_json.pyx in pyarrow._json._get_reader()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.get_input_stream()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.get_native_file()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.OSFile.__cinit__()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/io.pxi in pyarrow.lib.OSFile._open_readable()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.pyarrow_internal_check_status()

~/miniconda2/envs/datasets/lib/python3.7/site-packages/pyarrow/error.pxi in pyarrow.lib.check_status()

FileNotFoundError: [Errno 2] Failed to open local file 'gzip://file-000000000000.json::/Users/thomwolf/github-dataset/file-000000000000.json.gz'. Detail: [errno 2] No such file or directory
```

## Environment info
- `datasets` version: 1.9.1.dev0
- Platform: Darwin-19.6.0-x86_64-i386-64bit
- Python version: 3.7.7
- PyArrow version: 1.0.0


[Metrics] addition of wiki_split metrics

enhancement

New feature or request

metric request

Requesting to add a new metric

**Is your feature request related to a problem? Please describe.**
While training the model on sentence split the task in English we require to evaluate the trained model on `Exact Match`, `SARI` and `BLEU` score
like this
![image](https://user-images.githubusercontent.com/26653468/124746876-ff5a3380-df3e-11eb-9a01-4b48db7a6694.png)
While training we require metrics which can give all the output

Currently, we don't have an exact match for text normalized data

**Describe the solution you'd like**
A custom metrics for wiki_split that can calculate these three values and provide it in the form of a single dictionary
For exact match, we can refer to [this](https://github.com/huggingface/transformers/blob/master/src/transformers/data/metrics/squad_metrics.py)

**Describe alternatives you've considered**
Two metrics are already present one more can be added for an exact match then we can run all three metrics in training script

#self-assign


Make any ClientError trigger retry in streaming mode (e.g. ClientOSError)

1.10

Next minor release

During the FLAX sprint some users have this error when streaming datasets:
```python
aiohttp.client_exceptions.ClientOSError: [Errno 104] Connection reset by peer
```
This error must trigger a retry instead of directly crashing

Therefore I extended the error type that triggers the retry to be the base aiohttp error type: `ClientError`
In particular both `ClientOSError` and `ServerDisconnectedError` inherit from `ClientError`.


Add option to delete temporary files (e.g. extracted files) when loading dataset

enhancement

New feature or request

1.10

Next minor release

I'm loading a dataset constituted of 44 GB of compressed JSON files.

When loading the dataset with the JSON script, extracting the files create about 200 GB of uncompressed files before creating the 180GB of arrow cache tables

Having a simple way to delete the extracted files after usage (or even better, to stream extraction/delete) would be nice to avoid disk cluter.

I can maybe tackle this one in the JSON script unless you want a more general solution.


Fix DuplicatedKeysError in omp

1.10

Next minor release

Close #2598.


Remove import of transformers

1.10

Next minor release

When pickling a tokenizer within multiprocessing, check that is instance of transformers PreTrainedTokenizerBase without importing transformers.

Related to huggingface/transformers#12549 and #502.


Fix `filter` with multiprocessing in case all samples are discarded

1.10

Next minor release

Fixes #2600

Also I moved the check for `num_proc` larger than dataset size added in #2566 up so that multiprocessing is not used with one process.


Crash when using multiprocessing (`num_proc` > 1) on `filter` and all samples are discarded

bug

Something isn't working

## Describe the bug
If `filter` is applied to a dataset using multiprocessing (`num_proc` > 1) and all sharded datasets are empty afterwards (due to all samples being discarded), the program crashes.

## Steps to reproduce the bug
```python
from datasets import Dataset
data = Dataset.from_dict({'id': [0,1]})
data.filter(lambda x: False, num_proc=2)
```

## Expected results
An empty table should be returned without crashing.

## Actual results
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 185, in wrapper
    out: Union["Dataset", "DatasetDict"] = func(self, *args, **kwargs)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/fingerprint.py", line 397, in wrapper
    out = func(self, *args, **kwargs)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 2143, in filter
    return self.map(
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 1738, in map
    result = concatenate_datasets(transformed_shards)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/arrow_dataset.py", line 3267, in concatenate_datasets
    table = concat_tables(tables_to_concat, axis=axis)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/table.py", line 853, in concat_tables
    return ConcatenationTable.from_tables(tables, axis=axis)
  File "/home/user/venv/lib/python3.8/site-packages/datasets/table.py", line 713, in from_tables
    blocks = to_blocks(tables[0])
IndexError: list index out of range
```

## Environment info
<!-- You can run the command `datasets-cli env` and copy-and-paste its output below. -->
- `datasets` version: 1.9.0
- Platform: Linux-5.12.11-300.fc34.x86_64-x86_64-with-glibc2.2.5
- Python version: 3.8.10
- PyArrow version: 3.0.0


Update processing.rst with other export formats

1.10

Next minor release

Add other supported export formats than CSV in the docs.


Unable to download omp dataset

Something isn't working

## Describe the bug
The omp dataset cannot be downloaded because of a DuplicatedKeysError

## Steps to reproduce the bug
from datasets import load_dataset
omp = load_dataset('omp', 'posts_labeled')
print(omp)

## Expected results
This code should download the omp dataset and print the dictionary

## Actual results
Downloading and preparing dataset omp/posts_labeled (download: 1.27 MiB, generated: 13.31 MiB, post-processed: Unknown size, total: 14.58 MiB) to /home/erika_distefano/.cache/huggingface/datasets/omp/posts_labeled/1.1.0/2fe5b067be3bff1d4588d5b0cbb9b5b22ae1b9d5b026a8ff572cd389f862735b...
0 examples [00:00, ? examples/s]2021-07-06 09:43:55.868815: I tensorflow/stream_executor/platform/default/dso_loader.cc:48] Successfully opened dynamic library libcudart.so.11.0
Traceback (most recent call last):
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 990, in _prepare_split
    writer.write(example, key)
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 338, in write
    self.check_duplicate_keys()
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 349, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 3326
Keys should be unique and deterministic in nature

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "hf_datasets.py", line 32, in <module>
    omp = load_dataset('omp', 'posts_labeled')
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/load.py", line 748, in load_dataset
    use_auth_token=use_auth_token,
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 575, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 652, in _download_and_prepare
    self._prepare_split(split_generator, **prepare_split_kwargs)
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/builder.py", line 992, in _prepare_split
    num_examples, num_bytes = writer.finalize()
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 409, in finalize
    self.check_duplicate_keys()
  File "/home/erika_distefano/.local/lib/python3.6/site-packages/datasets/arrow_writer.py", line 349, in check_duplicate_keys
    raise DuplicatedKeysError(key)
datasets.keyhash.DuplicatedKeysError: FAILURE TO GENERATE DATASET !
Found duplicate Key: 3326
Keys should be unique and deterministic in nature

## Environment info
- `datasets` version: 1.8.0
- Platform: Ubuntu 18.04.4 LTS
- Python version: 3.6.9
- PyArrow version: 3.0.0