Add Facebook SimpleQuestionV2

Add simple questions v2: https://research.fb.com/downloads/babi/


Add GoEmotions

Hi @lewtun, thanks for reporting.

It seems that our library fails at inferring the dtype of the columns:
- `milestone`
- `performed_via_github_app`

(and assigns them `null` dtype).

Adds the GoEmotions dataset, a nice emotion classification dataset with 27 (multi-)label annotations on reddit comments. Includes both a large raw version and a narrowed version with predefined train/test/val splits, which I've included as separate configs with the latter as a default.

- Webpage/repo: https://github.com/google-research/google-research/tree/master/goemotions
- Paper: https://arxiv.org/abs/2005.00547


Add siswati ner corpus


Adding NCHLT dataset


Add KorNLU

Added Korean NLU datasets. The link to the dataset can be found [here](https://github.com/kakaobrain/KorNLUDatasets) and the paper can be found [here](https://arxiv.org/abs/2004.03289)

**Note**: The MNLI tsv file is broken, so this code currently excludes the file. Please suggest other alternative if any @lhoestq

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


Dataset.map() turns tensors into lists?

I apply `Dataset.map()` to a function that returns a dict of torch tensors (like a tokenizer from the repo transformers). However, in the mapped dataset, these tensors have turned to lists!

```import datasets
import torch
from datasets import load_dataset
print("version datasets", datasets.__version__)

dataset = load_dataset("snli", split='train[0:50]')

def tokenizer_fn(example):
    # actually uses a tokenizer which does something like:
    return {'input_ids': torch.tensor([[0, 1, 2]])}

print("First item in dataset:\n", dataset[0])
tokenized = tokenizer_fn(dataset[0])
print("Tokenized hyp:\n", tokenized)
dataset_tok = dataset.map(tokenizer_fn, batched=False,
        remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
dataset_tok = dataset.map(tokenizer_fn, batched=False,
                          remove_columns=['label', 'premise', 'hypothesis'])
print("Tokenized using map:\n", dataset_tok[0])
print(type(tokenized['input_ids']), type(dataset_tok[0]['input_ids']))
```

The output is:

```
version datasets 1.1.3
Reusing dataset snli (/home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c)
First item in dataset:
 {'premise': 'A person on a horse jumps over a broken down airplane.', 'hypothesis': 'A person is training his horse for a competition.', 'label': 1}
Tokenized hyp:
 {'input_ids': tensor([[0, 1, 2]])}
Loading cached processed dataset at /home/tom/.cache/huggingface/datasets/snli/plain_text/1.0.0/bb1102591c6230bd78813e229d5dd4c7fbf4fc478cec28f298761eb69e5b537c/cache-fe38f449fe9ac46f.arrow
Tokenized using map:
 {'input_ids': [[0, 1, 2]]}
<class 'torch.Tensor'> <class 'list'>
```

Or am I doing something wrong?


Add xitsonga ner corpus


Add AMTTL Chinese Word Segmentation Dataset


Add TSAC: Tunisian Sentiment Analysis Corpus

github: https://github.com/fbougares/TSAC

paper: https://www.aclweb.org/anthology/W17-1307/


Add Big Patent dataset

- More info on the dataset: https://evasharma.github.io/bigpatent/
- There's another raw version of the dataset available from tfds. However, they're quite large so I don't have the resources to fully test all the configs for that version yet. We'll try to add it later.
- ~Currently, there are no dummy data for this dataset yet as I'm facing some problems with generating them. I'm trying to add them later.~


Add SuperGLUE metric

Adds a new metric for the SuperGLUE benchmark (similar to the GLUE benchmark metric).


Add UN Universal Declaration of Human Rights (UDHR)

Universal declaration of human rights with translations in 464 languages and dialects.

- UN page: https://www.ohchr.org/EN/UDHR/Pages/UDHRIndex.aspx
- Raw data source: https://unicode.org/udhr/index.html

Each instance of the dataset corresponds to one translation of the document. Since there's only one instance per language (and because there are 500 languages so the dummy data would be messy), I opted to just include them all under the same single config. I wasn't able to find any kind of license so I just copied the copyright notice.

I was pretty careful careful generating the language tags so they _should_ all be correct & consistent BCP-47 codes per the docs.


Update ADD NEW DATASET

This PR adds a couple of detail on cloning/rebasing the repo.


add med_hop

This PR adds the MedHop dataset from the QAngaroo multi hop reading comprehension datasets

More info:
http://qangaroo.cs.ucl.ac.uk/index.html


Fix docs indentation issues

Replace tabs with spaces.


Add PerSenT

Added [Person's SentimenT](https://stonybrooknlp.github.io/PerSenT/) dataset.


add wiki_hop

This PR adds the WikiHop dataset from the QAngaroo multi hop reading comprehension datasets

More info:
http://qangaroo.cs.ucl.ac.uk/index.html


add scb_mt_enth_2020

## scb-mt-en-th-2020: A Large English-Thai Parallel Corpus

The primary objective of our work is to build a large-scale English-Thai dataset for machine translation.
We construct an English-Thai machine translation dataset with over 1 million segment pairs, curated from various sources,
namely news, Wikipedia articles, SMS messages, task-based dialogs, web-crawled data and government documents.
Methodology for gathering data, building parallel texts and removing noisy sentence pairs are presented in a reproducible manner.
We train machine translation models based on this dataset. Our models' performance are comparable to that of
Google Translation API (as of May 2020) for Thai-English and outperform Google when the Open Parallel Corpus (OPUS) is
included in the training data for both Thai-English and English-Thai translation.
The dataset, pre-trained models, and source code to reproduce our work are available for public use.


Add support for ".txm" format

In dummy data generation, add support for XML-like ".txm" file format.

Also support filenames with additional compression extension: ".txm.gz".


IIT B English to Hindi machine translation dataset

Adding IIT Bombay English-Hindi Corpus dataset
more info : http://www.cfilt.iitb.ac.in/iitb_parallel/


add crows_pairs

This PR adds CrowS-Pairs datasets.

More info:
https://github.com/nyu-mll/crows-pairs/
https://arxiv.org/pdf/2010.00133.pdf


allegro_reviews dataset

- **Name:** *allegro_reviews*
- **Description:** *Allegro Reviews is a sentiment analysis dataset, consisting of 11,588 product reviews written in Polish and extracted from Allegro.pl - a popular e-commerce marketplace. Each review contains at least 50 words and has a rating on a scale from one (negative review) to five (positive review).*
- **Data:** *https://github.com/allegro/klejbenchmark-allegroreviews*
- **Motivation:** *The KLEJ benchmark (Kompleksowa Lista Ewaluacji Językowych) is a set of nine evaluation tasks for the Polish language understanding.*


Add PEC

A persona-based empathetic conversation dataset.


Add ASSET dataset for text simplification evaluation

Adding the ASSET dataset from https://github.com/facebookresearch/asset

One config for the simplification data, one for the human ratings of quality.

The README.md borrows from that written by @juand-r


Hi

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Lío o

````l`````````

```
O
```
`````
Ño
```
````

```


Add Sesotho Ner


Add ZEST: ZEroShot learning from Task descriptions

Adds the ZEST dataset on zero-shot learning from task descriptions from AI2.

- Webpage: https://allenai.org/data/zest
- Paper: https://arxiv.org/abs/2011.08115

The nature of this dataset made the supported task tags tricky if you wouldn't mind giving any feedback @yjernite. Also let me know if you think we should have a `other-task-generalization` or something like that...


Add Schema Guided Dialogue dataset

This PR adds the Schema Guided Dialogue dataset created for the DSTC8 challenge
- https://github.com/google-research-datasets/dstc8-schema-guided-dialogue

A bit simpler than MultiWOZ, the only tricky thing is the sequence of dictionaries that had to be linearized. There is a config for the data proper, and a config for the schemas.


add MRQA

MRQA (shared task 2019)
out of distribution generalization
Framed as extractive question answering
Dataset is the concatenation (of subsets) of existing QA datasets processed to match the SQuAD format


Add Gutenberg time references dataset

This PR adds the gutenberg_time dataset: https://arxiv.org/abs/2011.04124


Add Setswana NER


Add caWaC dataset

Add dataset.


Add Sepedi NER

This is a new branch created for this dataset


Specify file encoding

If not specified, Python uses system default, which for Windows is not "utf-8".


Add CLINC150 dataset

Added CLINC150 Dataset. The link to the dataset can be found [here](https://github.com/clinc/oos-eval) and the paper can be found [here](https://www.aclweb.org/anthology/D19-1131.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


add hard dataset

Hotel Reviews in Arabic language.


Add SciTLDR Dataset (Take 2)

Adds the SciTLDR Dataset by AI2
Added the `README.md` card with tags to the best of my knowledge

Multi-target summaries or TLDRs of Scientific Documents

Continued from #986


Adding CS restaurants dataset

This PR adds the CS restaurants dataset; this is a re-opening of a previous PR with a chaotic commit history.


Adding  Evidence Inference Data:

http://evidence-inference.ebm-nlp.com/download/
https://arxiv.org/pdf/2005.04177.pdf


Add Bilingual Corpus of Arabic-English Parallel Tweets

Added Bilingual Corpus of Arabic-English Parallel Tweets. The link to the dataset can be found [here](https://alt.qcri.org/wp-content/uploads/2020/08/Bilingual-Corpus-of-Arabic-English-Parallel-Tweets.zip) and the paper can be found [here](https://www.aclweb.org/anthology/2020.bucc-1.3.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


Add NoReC: Norwegian Review Corpus


Adding C3 dataset: the first free-form multiple-Choice Chinese machine reading Comprehension dataset.

https://github.com/nlpdata/c3
https://arxiv.org/abs/1904.09679


Adding C3 dataset: the first free-form multiple-Choice Chinese machine reading Comprehension dataset.      https://github.com/nlpdata/c3 https://arxiv.org/abs/1904.09679


Include license file in source distribution

It would be helpful to include the license file in the source distribution.


add yahoo_answers_topics

This PR adds yahoo answers topic classification dataset.

More info:
https://github.com/LC-John/Yahoo-Answers-Topic-Classification-Dataset

cc @joeddav, @yjernite


Adding Autshumato South african langages:


how large datasets are handled under the hood

Hi
I want to use multiple large datasets with a mapping style dataloader, where they cannot fit into memory, could you tell me how you handled the datasets under the hood? is this you bring all in memory in case of mapping style ones? or is this some sharding under the hood and you bring in memory when necessary, thanks


Add multi_x_science_sum

Add Multi-XScience Dataset.

github repo: https://github.com/yaolu/Multi-XScience
paper: [Multi-XScience: A Large-scale Dataset for Extreme Multi-document Summarization of Scientific Articles](https://arxiv.org/abs/2010.14235)


Adding Medal: MeDAL: Medical Abbreviation Disambiguation Dataset for Natural Language Understanding Pretraining


UM005: Urdu <> English Translation Dataset

Adds Urdu-English dataset for machine translation: http://ufal.ms.mff.cuni.cz/umc/005-en-ur/


add generated_reviews_enth

`generated_reviews_enth` is created as part of [scb-mt-en-th-2020](https://arxiv.org/pdf/2007.03541.pdf) for machine translation task. This dataset (referred to as `generated_reviews_yn` in [scb-mt-en-th-2020](https://arxiv.org/pdf/2007.03541.pdf)) are English product reviews generated by [CTRL](https://arxiv.org/abs/1909.05858), translated by Google Translate API and annotated as accepted or rejected (`correct`) based on fluency and adequacy of the translation by human annotators. This allows it to be used for English-to-Thai translation quality esitmation (binary label), machine translation, and sentiment analysis.


adding yahoo_answers_qa

Adding Yahoo Answers QA dataset.

More info:
https://ciir.cs.umass.edu/downloads/nfL6/


Microsoft CodeXGlue

Datasets from https://github.com/microsoft/CodeXGLUE

This contains 13 datasets:

code_x_glue_cc_clone_detection_big_clone_bench
code_x_glue_cc_clone_detection_poj_104
code_x_glue_cc_cloze_testing_all
code_x_glue_cc_cloze_testing_maxmin
code_x_glue_cc_code_completion_line
code_x_glue_cc_code_completion_token
code_x_glue_cc_code_refinement
code_x_glue_cc_code_to_code_trans
code_x_glue_cc_defect_detection
code_x_glue_ct_code_to_text
code_x_glue_tc_nl_code_search_adv
code_x_glue_tc_text_to_code
code_x_glue_tt_text_to_text


NotADirectoryError while loading the CNN/Dailymail dataset

Downloading and preparing dataset cnn_dailymail/3.0.0 (download: 558.32 MiB, generated: 1.28 GiB, post-processed: Unknown size, total: 1.82 GiB) to /root/.cache/huggingface/datasets/cnn_dailymail/3.0.0/3.0.0/0128610a44e10f25b4af6689441c72af86205282d26399642f7db38fa7535602...

---------------------------------------------------------------------------

NotADirectoryError                        Traceback (most recent call last)

<ipython-input-9-cd4bf8bea840> in <module>()
     22
     23
---> 24 train = load_dataset('cnn_dailymail', '3.0.0', split='train')
     25 validation = load_dataset('cnn_dailymail', '3.0.0', split='validation')
     26 test = load_dataset('cnn_dailymail', '3.0.0', split='test')

5 frames

/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/0128610a44e10f25b4af6689441c72af86205282d26399642f7db38fa7535602/cnn_dailymail.py in _find_files(dl_paths, publisher, url_dict)
    132     else:
    133         logging.fatal("Unsupported publisher: %s", publisher)
--> 134     files = sorted(os.listdir(top_dir))
    135
    136     ret_files = []

NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'


added dataset circa

Dataset Circa added. Only README.md and dataset card left


Add Sepedi ner corpus


Problem downloading amazon_reviews_multi

Thanks for adding the dataset.
After trying to load the dataset, I am getting the following error:
`ConnectionError: Couldn't reach https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_fr_train.json
`
I used the following code to load the dataset:
`load_dataset(
            dataset_name,
            "all_languages",
            cache_dir=".data"
        )`

I am using version 1.1.3 of `datasets`

Note that I can perform a successfull `wget https://amazon-reviews-ml.s3-us-west-2.amazonaws.com/json/train/dataset_fr_train.json`


Add CAIL 2018 dataset


Adding farsi_news dataset (https://github.com/sci2lab/Farsi-datasets)


Add E2E NLG

Adding the E2E NLG dataset.

More info here : http://www.macs.hw.ac.uk/InteractionLab/E2E/

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template and at least fill the tags
- [x] Both tests for the real data and the dummy data pass.


Fix SV -> NO

This PR fixes the small typo as seen in #956


making sure datasets are not loaded in memory and distributed training of them

Hi
I am dealing with large-scale datasets which I need to train distributedly, I used the shard function to divide the dataset across the cores, without any sampler, this does not work for distributed training and does not become any faster than 1 TPU core. 1) how I can make sure data is not loaded in memory 2) in case of distributed training with iterative datasets which measures needs to be taken? Is this all sharding the data only. I was wondering if there can be possibility for me to discuss this with someone with distributed training with iterative datasets using dataset library. thanks


Add OPUS DOGC dataset


Add SciTLDR Dataset

Adds the SciTLDR Dataset by AI2
Added README card with tags to the best of my knowledge

Multi-target summaries or TLDRs of Scientific Documents


Add GAP dataset

GAP dataset
Gender bias coreference resolution


committing Whoa file


add mc taco

MC-TACO
Temporal commonsense knowledge


add prachathai67k take2

I decided it will be faster to create a new pull request instead of fixing the rebase issues.
continuing from https://github.com/huggingface/datasets/pull/954


add wisesight_sentiment take2

Take 2 since last time the rebase issues were taking me too much time to fix as opposed to just open a new one.


Wongnai - Thai reviews dataset

40,000 reviews, previously released on GitHub ( https://github.com/wongnai/wongnai-corpus ) with an LGPL license, and on a closed Kaggle competition ( https://www.kaggle.com/c/wongnai-challenge-review-rating-prediction/ )


[WIP] Add multi woz

This PR adds version 2.2 of the Multi-domain Wizard of OZ dataset: https://github.com/budzianowski/multiwoz/tree/master/data/MultiWOZ_2.2

It was a pretty big chunk of work to figure out the structure, so I stil have tol add the description to the README.md

On the plus side the structure is broadly similar to that of the Google Schema Guided dialogue [dataset](https://github.com/google-research-datasets/dstc8-schema-guided-dialogue), so will take care of that one next.


Add code refinement

### OVERVIEW
Millions of open-source projects with numerous bug fixes
are available in code repositories. This proliferation
of software development histories can be leveraged to
learn how to fix common programming bugs
Code refinement aims to automatically fix bugs in the code,
which can contribute to reducing the cost of bug-fixes for developers.
Given a piece of Java code with bugs,
the task is to remove the bugs to output the refined code.


Add ROPES dataset

ROPES dataset
Reasoning over paragraph effects in situations - testing a system's ability to apply knowledge from a passage of text to a new situation. The task is framed into a reading comprehension task following squad-style extractive qa.

One thing to note: labels of the test set are hidden (leaderboard submission) so I encoded that as an empty list (ropes.py:L125)


Arabic pos dialect

A README.md and loading script for the Arabic POS Dialect dataset. The README is missing the sections on personal information, biases, and limitations, as it would probably be better for those to be filled by someone who can read the contents of the dataset and is familiar with Arabic NLP.


add MeTooMA dataset

This PR adds the #MeToo MA dataset. It presents multi-label data points for tweets mined in the backdrop of the #MeToo movement. The dataset includes data points in the form of Tweet ids and appropriate labels. Please refer to the accompanying paper for detailed information regarding annotation, collection, and guidelines.

Paper: https://ojs.aaai.org/index.php/ICWSM/article/view/7292
Dataset Link: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/JN4EYU

---
annotations_creators:
- expert-generated
language_creators:
- found
languages:
- en
multilinguality:
- monolingual
size_categories:
- 1K<n<10K
source_datasets:
- original
task_categories:
- text-classification
- text-retrieval
task_ids:
- multi-class-classification
- multi-label-classification
---

# Dataset Card for #MeTooMA dataset

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-instances)
  - [Data Splits](#data-instances)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)

## Dataset Description

- **Homepage:** https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/JN4EYU
- **Paper:** https://ojs.aaai.org//index.php/ICWSM/article/view/7292
- **Point of Contact:** https://github.com/midas-research/MeTooMA

### Dataset Summary

- The dataset consists of tweets belonging to #MeToo movement on Twitter, labeled into different categories.
- This dataset includes more data points and has more labels than any of the previous datasets that contain social media
posts about sexual abuse disclosures. Please refer to the Related Datasets of the publication for detailed information about this.
- Due to Twitter's development policies, the authors provide only the tweet IDs and corresponding labels,
other data can be fetched via Twitter API.
- The data has been labeled by experts, with the majority taken into the account for deciding the final label.
- The authors provide these labels for each of the tweets.
  - Relevance
  - Directed Hate
  - Generalized Hate
  - Sarcasm
  - Allegation
  - Justification
  - Refutation
  - Support
  - Oppose
- The definitions for each task/label are in the main publication.
- Please refer to the accompanying paper https://aaai.org/ojs/index.php/ICWSM/article/view/7292 for statistical analysis on the textual data
extracted from this dataset.
- The language of all the tweets in this dataset is English
- Time period: October 2018 - December 2018
- Suggested Use Cases of this dataset:
  - Evaluating usage of linguistic acts such as hate-speech and sarcasm in the context of public sexual abuse disclosures.
  - Extracting actionable insights and virtual dynamics of gender roles in sexual abuse revelations.
  - Identifying how influential people were portrayed on the public platform in the
  events of mass social movements.
  - Polarization analysis based on graph simulations of social nodes of users involved
  in the #MeToo movement.

### Supported Tasks and Leaderboards

Multi-Label and Multi-Class Classification

### Languages

English

## Dataset Structure
- The dataset is structured into CSV format with TweetID and accompanying labels.
- Train and Test sets are split into respective files.

### Data Instances

Tweet ID and the appropriate labels

### Data Fields

Tweet ID and appropriate labels (binary label applicable for a data point) and multiple labels for each Tweet ID

### Data Splits

- Train: 7979
- Test: 1996

## Dataset Creation

### Curation Rationale

- Twitter was the major source of all the public disclosures of sexual abuse incidents during the #MeToo movement.
- People expressed their opinions over issues that were previously missing from the social media space.
- This provides an option to study the linguistic behaviors of social media users in an informal setting,
therefore the authors decide to curate this annotated dataset.
- The authors expect this dataset would be of great interest and use to both computational and socio-linguists.
- For computational linguists, it provides an opportunity to model three new complex dialogue acts (allegation, refutation, and justification) and also to study how these acts interact with some of the other linguistic components like stance, hate, and sarcasm. For socio-linguists, it provides an opportunity to explore how a movement manifests in social media.

### Source Data
- Source of all the data points in this dataset is a Twitter social media platform.

#### Initial Data Collection and Normalization

- All the tweets are mined from Twitter with initial search parameters identified using keywords from the #MeToo movement.
- Redundant keywords were removed based on manual inspection.
- Public streaming APIs of Twitter was used for querying with the selected keywords.
- Based on text de-duplication and cosine similarity score, the set of tweets were pruned.
- Non-English tweets were removed.
- The final set was labeled by experts with the majority label taken into the account for deciding the final label.
- Please refer to this paper for detailed information: https://ojs.aaai.org//index.php/ICWSM/article/view/7292

#### Who are the source language producers?

Please refer to this paper for detailed information: https://ojs.aaai.org//index.php/ICWSM/article/view/7292

### Annotations

#### Annotation process

- The authors chose against crowdsourcing for labeling this dataset due to its highly sensitive nature.
- The annotators are domain experts having degrees in advanced clinical psychology and gender studies.
- They were provided a guidelines document with instructions about each task and its definitions, labels, and examples.
- They studied the document, worked on a few examples to get used to this annotation task.
- They also provided feedback for improving the class definitions.
- The annotation process is not mutually exclusive, implying that the presence of one label does not mean the
absence of the other one.

#### Who are the annotators?

- The annotators are domain experts having a degree in clinical psychology and gender studies.
- Please refer to the accompanying paper for a detailed annotation process.

### Personal and Sensitive Information

- Considering Twitter's policy for distribution of data, only Tweet ID and applicable labels are shared for public use.
- It is highly encouraged to use this dataset for scientific purposes only.
- This dataset collection completely follows the Twitter mandated guidelines for distribution and usage.

## Considerations for Using the Data

### Social Impact of Dataset

- The authors of this dataset do not intend to conduct a population-centric analysis of the #MeToo movement on Twitter.
- The authors acknowledge that findings from this dataset cannot be used as-is for any direct social intervention, these
should be used to assist already existing human intervention tools and therapies.
- Enough care has been taken to ensure that this work comes off as trying to target a specific person for their
the personal stance of issues pertaining to the #MeToo movement.
- The authors of this work do not aim to vilify anyone accused in the #MeToo movement in any manner.
- Please refer to the ethics and discussion section of the mentioned publication for appropriate sharing of this dataset
and the social impact of this work.

### Discussion of Biases

- The #MeToo movement acted as a catalyst for implementing social policy changes to benefit the members of
the community affected by sexual abuse.
- Any work undertaken on this dataset should aim to minimize the bias against minority groups which
might amplify in cases of a sudden outburst of public reactions over sensitive social media discussions.

### Other Known Limitations

- Considering privacy concerns, social media practitioners should be aware of making automated interventions
to aid the victims of sexual abuse as some people might not prefer to disclose their notions.
- Concerned social media users might also repeal their social information if they found out that their
information is being used for computational purposes, hence it is important to seek subtle individual consent
before trying to profile authors involved in online discussions to uphold personal privacy.

## Additional Information

Please refer to this link: https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/JN4EYU

### Dataset Curators

- If you use the corpus in a product or application, then please credit the authors
and [Multimodal Digital Media Analysis Lab - Indraprastha Institute of Information Technology, New Delhi]
(http://midas.iiitd.edu.in) appropriately.
Also, if you send us an email, we will be thrilled to know about how you have used the corpus.
- If interested in the commercial use of the corpus, send an email to midas@iiitd.ac.in.
- Multimodal Digital Media Analysis Lab - Indraprastha Institute of Information Technology, New Delhi, India
disclaims any responsibility for the use of the corpus and does not provide technical support.
However, the contact listed above will be happy to respond to queries and clarifications
- Please feel free to send us an email:
  - with feedback regarding the corpus.
  - with information on how you have used the corpus.
  - if interested in having us analyze your social media data.
  - if interested in a collaborative research project.

### Licensing Information

[More Information Needed]

### Citation Information

Please cite the following publication if you make use of the dataset: https://ojs.aaai.org/index.php/ICWSM/article/view/7292

```

@article{Gautam_Mathur_Gosangi_Mahata_Sawhney_Shah_2020, title={#MeTooMA: Multi-Aspect Annotations of Tweets Related to the MeToo Movement}, volume={14}, url={https://aaai.org/ojs/index.php/ICWSM/article/view/7292}, abstractNote={<p>In this paper, we present a dataset containing 9,973 tweets related to the MeToo movement that were manually annotated for five different linguistic aspects: relevance, stance, hate speech, sarcasm, and dialogue acts. We present a detailed account of the data collection and annotation processes. The annotations have a very high inter-annotator agreement (0.79 to 0.93 k-alpha) due to the domain expertise of the annotators and clear annotation instructions. We analyze the data in terms of geographical distribution, label correlations, and keywords. Lastly, we present some potential use cases of this dataset. We expect this dataset would be of great interest to psycholinguists, socio-linguists, and computational linguists to study the discursive space of digitally mobilized social movements on sensitive issues like sexual harassment.</p&gt;}, number={1}, journal={Proceedings of the International AAAI Conference on Web and Social Media}, author={Gautam, Akash and Mathur, Puneet and Gosangi, Rakesh and Mahata, Debanjan and Sawhney, Ramit and Shah, Rajiv Ratn}, year={2020}, month={May}, pages={209-216} }

```


Add MeTooMA Dataset


Adding The Microsoft Terminology Collection dataset.


Add Children's Book Test (CBT) dataset

Add the Children's Book Test (CBT) from Facebook (Hill et al. 2016).

Sentence completion given a few sentences as context from a children's book.


add piqa

Physical Interaction: Question Answering (commonsense)
https://yonatanbisk.com/piqa/


Add SWAG

Commonsense NLI -> https://rowanzellers.com/swag/


Add wiki auto dataset

This PR adds the WikiAuto sentence simplification dataset

https://github.com/chaojiang06/wiki-auto

This is also a prospective GEM task, hence the README.md


ADD Afrikaans NER

Afrikaans NER corpus


Add CS Restaurants dataset

This PR adds the Czech restaurants dataset for Czech NLG.


Add CLINC150 Dataset

Added CLINC150 Dataset. The link to the dataset can be found [here](https://github.com/clinc/oos-eval) and the paper can be found [here](https://www.aclweb.org/anthology/D19-1131.pdf)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


Add CLINC150 Dataset


Adding the WebNLG dataset

This PR adds data from the WebNLG challenge, with one configuration per release and challenge iteration.

More information can be found [here](https://webnlg-challenge.loria.fr/)

Unfortunately, the data itself comes from a pretty large number of small XML files, so the dummy data ends up being quite large (8.4 MB even keeping only one example per file).


add CODAH dataset

Adding CODAH dataset.

More info:
https://github.com/Websail-NU/CODAH


Add Danish Political Comments Dataset


sample multiple datasets

Hi
I am dealing with multiple datasets, I need to have a dataloader over them with a condition that in each batch data samples are coming from one of the datasets. My main question is:
-  I need to have a way to sample the datasets first with some weights, lets say 2x dataset1 1x dataset2, could you point me how I can do it

sub-questions:
- I want to concat sampled datasets and define one dataloader on it, then I need a way to make sure batches come from 1 dataset in each iteration, could you assist me how I can do?
- I use iterative-type of datasets, but I need a method of shuffling still since it brings accuracy performance issues if not doing it, thanks for the help.


Add code to automate parts of the dataset card

Most parts of the "Dataset Structure" section can be generated automatically. This PR adds some code to do so.


Add Tunizi Dataset


dataset(ncslgr): add initial loading script

clean #789


Isixhosa ner corpus


Add Norwegian NER

This PR adds the [Norwegian NER](https://github.com/ljos/navnkjenner) dataset.

I have added the `conllu` package as a test dependency. This is required to properly parse the `.conllu` files.


Added PragmEval benchmark


add prachathai67k

`prachathai-67k`: News Article Corpus and Multi-label Text Classificdation from Prachathai.com
The prachathai-67k dataset was scraped from the news site Prachathai.
We filtered out those articles with less than 500 characters of body text, mostly images and cartoons.
It contains 67,889 articles wtih 12 curated tags from August 24, 2004 to November 15, 2018.
The dataset was originally scraped by @lukkiddd and cleaned by @cstorm125.
You can also see preliminary exploration at https://github.com/PyThaiNLP/prachathai-67k/blob/master/exploration.ipynb


added health_fact dataset

Added dataset Explainable Fact-Checking for Public Health Claims (dataset_id: health_fact)


Add orange sum

Add OrangeSum a french abstractive summarization dataset.

Paper: [BARThez: a Skilled Pretrained French Sequence-to-Sequence Model](https://arxiv.org/abs/2010.12321)


Prachathai67k

Add `prachathai-67k`: News Article Corpus and Multi-label Text Classificdation from Prachathai.com

The `prachathai-67k` dataset was scraped from the news site [Prachathai](prachathai.com). We filtered out those articles with less than 500 characters of body text, mostly images and cartoons. It contains 67,889 articles wtih 12 curated tags from August 24, 2004 to November 15, 2018. The dataset was originally scraped by [@lukkiddd](https://github.com/lukkiddd) and cleaned by [@cstorm125](https://github.com/cstorm125). Download the dataset [here](https://www.dropbox.com/s/fsxepdka4l2pr45/prachathai-67k.zip?dl=1). You can also see preliminary exploration in [exploration.ipynb](https://github.com/PyThaiNLP/prachathai-67k/blob/master/exploration.ipynb).

This dataset is a part of [pyThaiNLP](https://github.com/PyThaiNLP/) Thai text [classification-benchmarks](https://github.com/PyThaiNLP/classification-benchmarks). For the benchmark, we selected the following tags with substantial volume that resemble **classifying types of articles**:

* `การเมือง` - politics
* `สิทธิมนุษยชน` - human_rights
* `คุณภาพชีวิต` - quality_of_life
* `ต่างประเทศ` - international
* `สังคม` - social
* `สิ่งแวดล้อม` - environment
* `เศรษฐกิจ` - economics
* `วัฒนธรรม` - culture
* `แรงงาน` - labor
* `ความมั่นคง` - national_security
* `ไอซีที` - ict
* `การศึกษา` - education


Support .xz file format

Add support to extract/uncompress files in .xz format.


Add GermaNER Dataset


docs(ADD_NEW_DATASET): correct indentation for script


Add europeana newspapers

This PR adds the [Europeana newspapers](https://github.com/EuropeanaNewspapers/ner-corpora) dataset.


add PEC dataset

A persona-based empathetic conversation dataset published at EMNLP 2020.


Adding Babi dataset - English version

Adding the English version of bAbI.

Samples are taken from ParlAI for consistency with the main users at the moment.


Add German Legal Entity Recognition Dataset


The FLUE Benchmark

This PR adds the [FLUE](https://github.com/getalp/Flaubert/tree/master/flue) benchmark which is a set of different datasets to evaluate models for French content.

Two datasets are missing, the French Treebank that we can use only for research purpose and we are not allowed to distribute, and the Word Sense disambiguation for Nouns that will be added later.


D

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://github.com/huggingface/datasets/blob/master/ADD_NEW_DATASET.md).


Add People's Daily NER dataset


Add MSRA NER dataset


add wisesight_sentiment

Add `wisesight_sentiment` Social media messages in Thai language with sentiment label (positive, neutral, negative, question)

Model Card:
---
YAML tags:
annotations_creators:
- expert-generated
language_creators:
- found
languages:
- th
licenses:
- cc0-1.0
multilinguality:
- monolingual
size_categories:
- 10K<n<100K
source_datasets:
- original
task_categories:
- text-classification
task_ids:
- sentiment-classification
---

# Dataset Card for wisesight_sentiment

## Table of Contents
- [Dataset Description](#dataset-description)
  - [Dataset Summary](#dataset-summary)
  - [Supported Tasks](#supported-tasks-and-leaderboards)
  - [Languages](#languages)
- [Dataset Structure](#dataset-structure)
  - [Data Instances](#data-instances)
  - [Data Fields](#data-instances)
  - [Data Splits](#data-instances)
- [Dataset Creation](#dataset-creation)
  - [Curation Rationale](#curation-rationale)
  - [Source Data](#source-data)
  - [Annotations](#annotations)
  - [Personal and Sensitive Information](#personal-and-sensitive-information)
- [Considerations for Using the Data](#considerations-for-using-the-data)
  - [Social Impact of Dataset](#social-impact-of-dataset)
  - [Discussion of Biases](#discussion-of-biases)
  - [Other Known Limitations](#other-known-limitations)
- [Additional Information](#additional-information)
  - [Dataset Curators](#dataset-curators)
  - [Licensing Information](#licensing-information)
  - [Citation Information](#citation-information)

## Dataset Description

- **Homepage:** https://github.com/PyThaiNLP/wisesight-sentiment
- **Repository:** https://github.com/PyThaiNLP/wisesight-sentiment
- **Paper:**
- **Leaderboard:** https://www.kaggle.com/c/wisesight-sentiment/
- **Point of Contact:** https://github.com/PyThaiNLP/

### Dataset Summary

Wisesight Sentiment Corpus: Social media messages in Thai language with sentiment label (positive, neutral, negative, question)
- Released to public domain under Creative Commons Zero v1.0 Universal license.
- Labels: {"pos": 0, "neu": 1, "neg": 2, "q": 3}
- Size: 26,737 messages
- Language: Central Thai
- Style: Informal and conversational. With some news headlines and advertisement.
- Time period: Around 2016 to early 2019. With small amount from other period.
- Domains: Mixed. Majority are consumer products and services (restaurants, cosmetics, drinks, car, hotels), with some current affairs.
- Privacy:
    - Only messages that made available to the public on the internet (websites, blogs, social network sites).
    - For Facebook, this means the public comments (everyone can see) that made on a public page.
    - Private/protected messages and messages in groups, chat, and inbox are not included.
- Alternations and modifications:
    - Keep in mind that this corpus does not statistically represent anything in the language register.
    - Large amount of messages are not in their original form. Personal data are removed or masked.
    - Duplicated, leading, and trailing whitespaces are removed. Other punctuations, symbols, and emojis are kept intact.
    (Mis)spellings are kept intact.
    - Messages longer than 2,000 characters are removed.
    - Long non-Thai messages are removed. Duplicated message (exact match) are removed.
- More characteristics of the data can be explore [this notebook](https://github.com/PyThaiNLP/wisesight-sentiment/blob/master/exploration.ipynb)

### Supported Tasks and Leaderboards

Sentiment analysis / [Kaggle Leaderboard](https://www.kaggle.com/c/wisesight-sentiment/)

### Languages

Thai

## Dataset Structure

### Data Instances

```
{'category': 'pos', 'texts': 'น่าสนนน'}
{'category': 'neu', 'texts': 'ครับ #phithanbkk'}
{'category': 'neg', 'texts': 'ซื้อแต่ผ้าอนามัยแบบเย็นมาค่ะ แบบว่าอีห่ากูนอนไม่ได้'}
{'category': 'q', 'texts': 'มีแอลกอฮอลมั้ยคะ'}
```

### Data Fields

- `texts`: texts
- `category`: sentiment of texts ranging from `pos` (positive; 0), `neu` (neutral; 1), `neg` (negative; 2) and `q` (question; 3)

### Data Splits

|           | train | valid | test  |
|-----------|-------|-------|-------|
| # samples | 21628 | 2404  | 2671  |
| # neu     | 11795 | 1291  | 1453  |
| # neg     | 5491  | 637   | 683   |
| # pos     | 3866  | 434   | 478   |
| # q       | 476   | 42    | 57    |
| avg words | 27.21 | 27.18 | 27.12 |
| avg chars | 89.82 | 89.50 | 90.36 |

## Dataset Creation

### Curation Rationale

Originally, the dataset was conceived for the [In-class Kaggle Competition](https://www.kaggle.com/c/wisesight-sentiment/) at Chulalongkorn university by [Ekapol Chuangsuwanich](https://www.cp.eng.chula.ac.th/en/about/faculty/ekapolc/) (Faculty of Engineering, Chulalongkorn University). It has since become one of the  benchmarks for sentiment analysis in Thai.

### Source Data

#### Initial Data Collection and Normalization

- Style: Informal and conversational. With some news headlines and advertisement.
- Time period: Around 2016 to early 2019. With small amount from other period.
- Domains: Mixed. Majority are consumer products and services (restaurants, cosmetics, drinks, car, hotels), with some current affairs.
- Privacy:
  - Only messages that made available to the public on the internet (websites, blogs, social network sites).
  - For Facebook, this means the public comments (everyone can see) that made on a public page.
  - Private/protected messages and messages in groups, chat, and inbox are not included.
  - Usernames and non-public figure names are removed
  - Phone numbers are masked (e.g. 088-888-8888, 09-9999-9999, 0-2222-2222)
  - If you see any personal data still remain in the set, please tell us - so we can remove them.
- Alternations and modifications:
  - Keep in mind that this corpus does not statistically represent anything in the language register.
  - Large amount of messages are not in their original form. Personal data are removed or masked.
  - Duplicated, leading, and trailing whitespaces are removed. Other punctuations, symbols, and emojis are kept intact.
  - (Mis)spellings are kept intact.
  - Messages longer than 2,000 characters are removed.
  - Long non-Thai messages are removed. Duplicated message (exact match) are removed.

#### Who are the source language producers?

Social media users in Thailand

### Annotations

#### Annotation process

- Sentiment values are assigned by human annotators.
- A human annotator put his/her best effort to assign just one label, out of four, to a message.
- Agreement, enjoyment, and satisfaction are positive. Disagreement, sadness, and disappointment are negative.
- Showing interest in a topic or in a product is counted as positive. In this sense, a question about a particular product could has a positive sentiment value, if it shows the interest in the product.
- Saying that other product or service is better is counted as negative.
- General information or news title tend to be counted as neutral.

#### Who are the annotators?

Outsourced annotators hired by [Wisesight (Thailand) Co., Ltd.](https://github.com/wisesight/)

### Personal and Sensitive Information

- We trying to exclude any known personally identifiable information from this data set.
- Usernames and non-public figure names are removed
- Phone numbers are masked (e.g. 088-888-8888, 09-9999-9999, 0-2222-2222)
- If you see any personal data still remain in the set, please tell us - so we can remove them.

## Considerations for Using the Data

### Social Impact of Dataset

- `wisesight_sentiment` is the first and one of the few open datasets for sentiment analysis of social media data in Thai
- There are risks of personal information that escape the anonymization process

### Discussion of Biases

- A message can be ambiguous. When possible, the judgement will be based solely on the text itself.
  - In some situation, like when the context is missing, the annotator may have to rely on his/her own world knowledge and just guess.
  - In some cases, the human annotator may have an access to the message's context, like an image. These additional information are not included as part of this corpus.

### Other Known Limitations

- The labels are imbalanced; over half of the texts are `neu` (neutral) whereas there are very few `q` (question).
- Misspellings in social media texts make word tokenization process for Thai difficult, thus impacting the model performance

## Additional Information

### Dataset Curators

Thanks [PyThaiNLP](https://github.com/PyThaiNLP/pythainlp) community, [Kitsuchart Pasupa](http://www.it.kmitl.ac.th/~kitsuchart/) (Faculty of Information Technology, King Mongkut's Institute of Technology Ladkrabang), and [Ekapol Chuangsuwanich](https://www.cp.eng.chula.ac.th/en/about/faculty/ekapolc/) (Faculty of Engineering, Chulalongkorn University) for advice. The original Kaggle competition, using the first version of this corpus, can be found at https://www.kaggle.com/c/wisesight-sentiment/

### Licensing Information

- If applicable, copyright of each message content belongs to the original poster.
- **Annotation data (labels) are released to public domain.**
- [Wisesight (Thailand) Co., Ltd.](https://github.com/wisesight/) helps facilitate the annotation, but does not necessarily agree upon the labels made by the human annotators. This annotation is for research purpose and does not reflect the professional work that Wisesight has been done for its customers.
- The human annotator does not necessarily agree or disagree with the message. Likewise, the label he/she made to the message does not necessarily reflect his/her personal view towards the message.

### Citation Information

Please cite the following if you make use of the dataset:

Arthit Suriyawongkul, Ekapol Chuangsuwanich, Pattarawat Chormai, and Charin Polpanumas. 2019. **PyThaiNLP/wisesight-sentiment: First release.** September.

BibTeX:
```
@software{bact_2019_3457447,
  author       = {Suriyawongkul, Arthit and
                  Chuangsuwanich, Ekapol and
                  Chormai, Pattarawat and
                  Polpanumas, Charin},
  title        = {PyThaiNLP/wisesight-sentiment: First release},
  month        = sep,
  year         = 2019,
  publisher    = {Zenodo},
  version      = {v1.0},
  doi          = {10.5281/zenodo.3457447},
  url          = {https://doi.org/10.5281/zenodo.3457447}
}
```


V-1.0.0 of isizulu_ner_corpus


Local machine/cluster Beam Datasets example/tutorial

Hi,

I'm wondering if https://huggingface.co/docs/datasets/beam_dataset.html has an non-GCP or non-Dataflow version example/tutorial? I tried to migrate it to run on DirectRunner and SparkRunner, however, there were way too many runtime errors that I had to fix during the process, and even so I wasn't able to get either runner correctly producing the desired output.

Thanks!
Shang


Added HANS parses and categories

This pull request adds HANS missing information: the sentence parses, as well as the heuristic category.


add PIB dataset

This pull request will add PIB dataset.


small updates to the "add new dataset" guide

small updates (corrections/typos) to the "add new dataset" guide


Add NumerSense

Adds the NumerSense dataset
- Webpage/leaderboard: https://inklab.usc.edu/NumerSense/
- Paper: https://arxiv.org/abs/2005.00683
- Description: NumerSense is a new numerical commonsense reasoning probing task, with a diagnostic dataset consisting of 3,145 masked-word-prediction probes. Basically, it's a benchmark to see whether your MLM can figure out the right number in a fill-in-the-blank task based on commonsense knowledge (a bird has **two** legs)


adding metooma dataset


[WIP] complex_webqa - Error zipfile.BadZipFile: Bad CRC-32

Have a string `zipfile.BadZipFile: Bad CRC-32 for file 'web_snippets_train.json'` error when downloading the largest file from dropbox: `https://www.dropbox.com/sh/7pkwkrfnwqhsnpo/AABVENv_Q9rFtnM61liyzO0La/web_snippets_train.json.zip?dl=1`

Didn't managed to see how to solve that.

Putting aside for now.


Lambada

Added LAMBADA dataset.

A couple of points of attention (mostly because I am not sure)
- The training data are compressed in a .tar file inside the main tar.gz file. I had to manually un-tar the training file to access the examples.
- The dev and test splits don't have the `category` field so I put `None` by default.

Happy to make changes if it doesn't respect the guidelines!

Victor


Add weibo NER dataset


Add the Multilingual Amazon Reviews Corpus

- **Name:** Multilingual Amazon Reviews Corpus* (`amazon_reviews_multi`)
- **Description:** A collection of Amazon reviews in English, Japanese, German, French, Spanish and Chinese.
- **Paper:** https://arxiv.org/abs/2010.02573

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


Hello


add inquisitive

Adding inquisitive qg dataset

More info: https://github.com/wjko2/INQUISITIVE


Add Turku NLP Corpus for Finnish NER


Add DART

- **Name:** *DART*
- **Description:** *DART is a large dataset for open-domain structured data record to text generation.*
- **Paper:** *https://arxiv.org/abs/2007.02871*
- **Data:** *https://github.com/Yale-LILY/dart#leaderboard*

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


Add CC-100 dataset

wontfix

This will not be worked on

Add CC-100.

Close #773


Add XOR QA Dataset

Added XOR Question Answering Dataset. The link to the dataset can be found [here](https://nlp.cs.washington.edu/xorqa/)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


add dream dataset

Adding Dream: a Dataset and for Dialogue-Based Reading Comprehension

More details:
https://dataset.org/dream/
https://github.com/nlpdata/dream


wrong length with datasets

Hi
I have a MRPC dataset which I convert it to seq2seq format, then this is of this format:

`Dataset(features: {'src_texts': Value(dtype='string', id=None), 'tgt_texts': Value(dtype='string', id=None)}, num_rows: 10)
`

I feed it to a dataloader:
```
dataloader = DataLoader(
            train_dataset,
            batch_size=self.args.train_batch_size,
            sampler=train_sampler,
            collate_fn=self.data_collator,
            drop_last=self.args.dataloader_drop_last,
            num_workers=self.args.dataloader_num_workers,
        )
```

now if I type len(dataloader) this is 1, which is wrong, and this needs to be 10. could you assist me please? thanks


Add conll2002

Adding the Conll2002 dataset for NER.

More info here : https://www.clips.uantwerpen.be/conll2002/ner/

### Checkbox

- [x] Create the dataset script `/datasets/my_dataset/my_dataset.py` using the template
- [x] Fill the `_DESCRIPTION` and `_CITATION` variables
- [x] Implement `_infos()`, `_split_generators()` and `_generate_examples()`
- [x] Make sure that the `BUILDER_CONFIGS` class attribute is filled with the different configurations of the dataset and that the `BUILDER_CONFIG_CLASS` is specified if there is a custom config class.
- [x] Generate the metadata file `dataset_infos.json` for all configurations
- [x] Generate the dummy data `dummy_data.zip` files to have the dataset script tested and that they don't weigh too much (<50KB)
- [x] Add the dataset card `README.md` using the template : fill the tags and the various paragraphs
- [x] Both tests for the real data and the dummy data pass.


Addition of Concode Dataset

##Overview
Concode Dataset contains pairs of Nl Queries and the corresponding Code.(Contextual Code Generation)

Reference Links
Paper Link = https://arxiv.org/pdf/1904.09086.pdf
Github Link =https://github.com/microsoft/CodeXGLUE/tree/main/Text-Code/text-to-code


Add Swedish NER Corpus


Shall we change the hashing to encoding to reduce potential replicated cache files?

enhancement

New feature or request

generic discussion

Generic discussion on the library

Hi there. For now, we are using `xxhash` to hash the transformations to fingerprint and we will save a copy of the processed dataset to disk if there is a new hash value. However, there are some transformations that are idempotent or commutative to each other. I think that encoding the transformation chain as the fingerprint may help in those cases, for example, use `base64.urlsafe_b64encode`. In this way, before we want to save a new copy, we can decode the transformation chain and normalize it to prevent omit potential reuse. As the main targets of this project are the really large datasets that cannot be loaded entirely in memory, I believe it would save a lot of time if we can avoid some write.

If you have interest in this, I'd love to help :).


Add list_github_datasets api for retrieving dataset name list in github repo

Thank you for your great effort on unifying data processing for NLP!

This pr is trying to add a new api `list_github_datasets` in the `inspect` module. The reason for it is that the current `list_datasets` api need to access https://huggingface.co/api/datasets to get a large json. However, this connection can be really slow... (I was visiting from China) and from my own experience, most of the time `requests.get` failed to download the whole json after a long wait and will trigger fault in `r.json()`.
I also noticed that the current implementation will first try to download from github, which makes me be able to smoothly run `load_dataset('squad')` in the example.
Therefore, I think it would be better if we can have an api to get the list of datasets that are available on github, and it will also improve newcomers' experience (it is a little frustrating if one cannot successfully run the first function in the README example.) before we have faster source for huggingface.co.

As for the implementation, I've added a `dataset_infos.json` file under the `datasets` folder, and it has the following structure:
```json
    {
        "id": "aeslc",
        "folder": "datasets/aeslc",
        "dataset_infos": "datasets/aeslc/dataset_infos.json"
    },
    ...
    {
        "id": "json",
        "folder": "datasets/json"
    },
   ...
```
The script I used to get this file is:
```python
import json
import os

DATASETS_BASE_DIR = "/root/datasets"
DATASET_INFOS_JSON = "dataset_infos.json"

datasets = []
for item in os.listdir(os.path.join(DATASETS_BASE_DIR, "datasets")):
    if os.path.isdir(os.path.join(DATASETS_BASE_DIR, "datasets", item)):
        datasets.append(item)

datasets.sort()

total_ds_info = []
for ds in datasets:
    ds_dir = os.path.join("datasets", ds)
    ds_info_dir = os.path.join(ds_dir, DATASET_INFOS_JSON)
    if os.path.isfile(os.path.join(DATASETS_BASE_DIR, ds_info_dir)):
        total_ds_info.append({"id": ds,
                              "folder": ds_dir,
                              "dataset_infos": ds_info_dir})
    else:
        total_ds_info.append({"id": ds,
                              "folder": ds_dir})

with open(DATASET_INFOS_JSON, "w") as f:
    json.dump(total_ds_info, f)
```
The new `dataset_infos.json` was saved as a formated json so that it will be easy to add new dataset.

When calling `list_github_datasets`, the user will get the list of dataset names in this github repo and if `with_details` is set to be `True`, they can get the url of specific dataset info.

Thank you for your time on reviewing this pr :).


My new dataset PEC

A new dataset PEC published in EMNLP 2020.


datasets module not found

Currently, running `from datasets import load_dataset` will throw a `ModuleNotFoundError: No module named 'datasets'` error.


Grindr meeting app web.Grindr

## Adding a Dataset
- **Name:** *name of the dataset*
- **Description:** *short description of the dataset (or link to social media or blog post)*
- **Paper:** *link to the dataset paper if available*
- **Data:** *link to the Github repository or current dataset location*
- **Motivation:** *what are some good reasons to have this dataset*

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


Add FiNER dataset

Hi,

this PR adds "A Finnish News Corpus for Named Entity Recognition" as new `finer` dataset.

The dataset is described in [this paper](https://arxiv.org/abs/1908.04212). The data is publicly available in [this GitHub](https://github.com/mpsilfve/finer-data).

Notice: they provide two testsets. The additional test dataset taken from Wikipedia is named as "test_wikipedia" split.


Add dependency on black for tests

Add package 'black' as an installation requirement for tests.


Remove os.path.join from all URLs

Remove `os.path.join` from all URLs in dataset scripts.


Fix url with backslash in windows for blimp and pg19

Following #903 I also fixed blimp and pg19 which were using the `os.path.join` to create urls

cc @albertvillanova


Disallow backslash in urls

Following #903 @albertvillanova noticed that there are sometimes bad usage of `os.path.join` in datasets scripts to create URLS. However this should be avoided since it doesn't work on windows.

I'm suggesting a test to make sure we that all the urls don't have backslashes in them in the datasets scripts.
The tests works by adding a callback feature to the MockDownloadManager used to test the dataset scripts. In a download callback I just make sure that the url is valid.


Very detailed step-by-step on how to add a dataset

Add very detailed step-by-step instructions to add a new dataset to the library.


Fix URL with backslash in Windows

In Windows, `os.path.join` generates URLs containing backslashes, when the first "path" does not end with a slash.

In general, `os.path.join` should be avoided to generate URLs.


Follow cache_dir parameter to gcs downloader

As noticed in #900 the cache_dir parameter was not followed to the downloader in the case of an already processed dataset hosted on our google storage (one of them is natural questions).

Fix #900


Addition of Nl2Bash Dataset

## Overview
The NL2Bash data contains over 10,000 instances of linux shell commands and their corresponding natural language descriptions provided by experts, from the Tellina system. The dataset features 100+ commonly used shell utilities.
## Footnotes
The following dataset marks the first ML on source code related Dataset in datasets module. It'll be really useful as a lot of the research direction involves Transformer Based Model.
Thanks.
### Reference Links

> Paper Link = https://arxiv.org/pdf/1802.08979.pdf
> Github Link = https://github.com/TellinaTool/nl2bash


datasets.load_dataset() custom chaching directory bug

Hello,
I'm having issue with loading a dataset with a custom `cache_dir`. Despite specifying the output dir, it is still downloaded to
 `~/.cache`.

## Environment info
- `datasets` version: 1.1.3
- Platform: Linux-4.19.129-aufs-1-x86_64-with-debian-10.1
- Python version: 3.7.3

## The code I'm running:
```python
import datasets
from pathlib import Path

validation_dataset = datasets.load_dataset("natural_questions", split="validation[:5%]", cache_dir=Path("./data"))
```

## The output:

* The dataset is downloaded to my home directory's `.cache`
* A new empty directory named "`natural_questions` is created in the specified directory `.data`
* `tree data` in the shell outputs:
```
data
└── natural_questions
    └── default
        └── 0.0.2
3 directories, 0 files
```

The output:
```
Downloading: 8.61kB [00:00, 5.11MB/s]
Downloading: 13.6kB [00:00, 7.89MB/s]
Using custom data configuration default
Downloading and preparing dataset natural_questions/default (download: 41.97 GiB, generated: 92.95 GiB, post-processed: Unknown size, total: 134.92 GiB) to ./data/natural_questions/default/0.0.2/867dbbaf9137c1b8
3ecb19f5eb80559e1002ea26e702c6b919cfa81a17a8c531...
Downloading: 100%|██████████████████████████████████████████████████| 13.6k/13.6k [00:00<00:00, 1.51MB/s]
Downloading:   7%|███▎                                            | 6.70G/97.4G [03:46<1:37:05, 15.6MB/s]
```

## Expected behaviour:
The dataset "Natural Questions" should be downloaded to the directory "./data"


Allow arrow based builder in auto dummy data generation

Following #898 I added support for arrow based builder for the auto dummy data generator


Adding SQA dataset

As discussed in #880

Seems like automatic dummy-data generation doesn't work if the builder is a `ArrowBasedBuilder`, do you think you could take a look @lhoestq ?


Dataset viewer issues

I was looking through the dataset viewer and I like it a lot. Version numbers, citation information, everything's there! I've spotted a few issues/bugs though:

- the URL is still under `nlp`, perhaps an alias for `datasets` can be made
- when I remove a **feature** (and the feature list is empty), I get an error. This is probably expected, but perhaps a better error message can be shown to the user

```bash
IndexError: list index out of range
Traceback:
File "/home/sasha/streamlit/lib/streamlit/ScriptRunner.py", line 322, in _run_script
    exec(code, module.__dict__)
File "/home/sasha/nlp-viewer/run.py", line 316, in <module>
    st.table(style)
File "/home/sasha/streamlit/lib/streamlit/DeltaGenerator.py", line 122, in wrapped_method
    return dg._enqueue_new_element_delta(marshall_element, delta_type, last_index)
File "/home/sasha/streamlit/lib/streamlit/DeltaGenerator.py", line 367, in _enqueue_new_element_delta
    rv = marshall_element(msg.delta.new_element)
File "/home/sasha/streamlit/lib/streamlit/DeltaGenerator.py", line 120, in marshall_element
    return method(dg, element, *args, **kwargs)
File "/home/sasha/streamlit/lib/streamlit/DeltaGenerator.py", line 2944, in table
    data_frame_proto.marshall_data_frame(data, element.table)
File "/home/sasha/streamlit/lib/streamlit/elements/data_frame_proto.py", line 54, in marshall_data_frame
    _marshall_styles(proto_df.style, df, styler)
File "/home/sasha/streamlit/lib/streamlit/elements/data_frame_proto.py", line 73, in _marshall_styles
    translated_style = styler._translate()
File "/home/sasha/.local/share/virtualenvs/lib-ogGKnCK_/lib/python3.7/site-packages/pandas/io/formats/style.py", line 351, in _translate
    * (len(clabels[0]) - len(hidden_columns))
```

- there seems to be **an encoding issue** in the default view, the dataset examples are shown as raw monospace text, without a decent encoding. That makes it hard to read for languages that use a lot of special characters. Take for instance the [cs-en WMT19 set](https://huggingface.co/nlp/viewer/?dataset=wmt19&config=cs-en). This problem goes away when you enable "List view", because then some syntax highlighteris used, and the special characters are coded correctly.


Add template and documentation for dataset card

This PR adds a template for dataset cards, as well as a guide to filling out the template and a completed example for the ELI5 dataset, building on the work of @mcmillanmajora

New pull requests adding datasets should now have a README.md file which serves both to hold the tags we will have to index the datasets and as a data statement.

The template is designed to be pretty extensive. The idea is that the person who uploads the dataset should put in all the basic information (at least the Dataset Description section) and whatever else they feel comfortable adding and leave the `[More Information Needed]` annotation everywhere else as a placeholder.

We will then work with @mcmillanmajora to involve the data authors more directly in filling out the remaining information.

Direct links to:
- [Documentation](https://github.com/yjernite/datasets/blob/add_dataset_card_doc/templates/README_guide.md)
- [Empty template](https://github.com/yjernite/datasets/blob/add_dataset_card_doc/templates/README.md)
- [ELI5 example](https://github.com/yjernite/datasets/blob/add_dataset_card_doc/datasets/eli5/README.md)


Better messages regarding split naming

I made explicit the error message when a bad split name is used.

Also I wanted to allow the `-` symbol for split names but actually this symbol is used to name the arrow files `{dataset_name}-{dataset_split}.arrow` so we should probably keep it this way, i.e. not allowing the `-` symbol in split names. Moreover in the future we might want to use `{dataset_name}-{dataset_split}-{shard_id}_of_{n_shards}.arrow` and reuse the `-` symbol.


Allow several tags sets

Hi !

Currently we have three dataset cards : snli, cnn_dailymail and allocine.
For each one of those datasets a set of tag is defined. The set of tags contains fields like `multilinguality`, `task_ids`, `licenses` etc.

For certain datasets like `glue` for example, there exist several configurations: `sst2`, `mnli` etc. Therefore we should define one set of tags per configuration. However the current format used for tags only supports one set of tags per dataset.

In this PR I propose a simple change in the yaml format used for tags to allow for several sets of tags.

Let me know what you think, especially @julien-c let me know if it's good for you since it's going to be parsed by moon-landing


add metrec: arabic poetry dataset


Add a few datasets of reference in the documentation

I started making a small list of various datasets of reference in the documentation.
Since many datasets share a lot in common I think it's good to have a list of datasets scripts to get some inspiration from.

Let me know what you think, and if you have ideas of other datasets that we may add to this list, please let me know.


gitignore .python-version

ignore `.python-version` added by `pyenv`


Add LER


Optional per-dataset default config name

This PR adds a `DEFAULT_CONFIG_NAME` class attribute to `DatasetBuilder`. This allows a dataset to have a specified default config name when a dataset has more than one config but the user does not specify it. For example, after defining `DEFAULT_CONFIG_NAME = "combined"` in PolyglotNER, a user can now do the following:

```python
ds = load_dataset("polyglot_ner")
```
which is equivalent to,
```python
ds = load_dataset("polyglot_ner", "combined")
```
In effect (for this particular dataset configuration), this means that if the user doesn't specify a language, they are given the combined dataset including all languages.

Since it doesn't always make sense to have a default config, this feature is opt-in. If `DEFAULT_CONFIG_NAME` is not defined and a user does not pass a config for a dataset with multiple configs available, a ValueError is raised like usual.

Let me know what you think about this approach @lhoestq @thomwolf and I'll add some documentation and define a default for some of our existing datasets.


Nested lists are zipped unexpectedly

I might misunderstand something, but I expect that if I define:
```python
"top": datasets.features.Sequence({
  "middle": datasets.features.Sequence({
    "bottom": datasets.Value("int32")
  })
})
```

And I then create an example:
```python
yield 1, {
  "top": [{
    "middle": [
      {"bottom": 1},
      {"bottom": 2}
    ]
  }]
}
```

I then load my dataset:
```python
train = load_dataset("my dataset")["train"]
```

and expect to be able to access `data[0]["top"][0]["middle"][0]`.

That is not the case. Here is `data[0]` as JSON:

```json
{"top": {"middle": [{"bottom": [1, 2]}]}}
```

Clearly different than the thing I inputted.
```json
{"top": [{"middle": [{"bottom": 1},{"bottom": 2}]}]}
```


bug

Something isn't working

I set up a new dataset, with a sequence of arrays (really, I want to have an array of (None, 137, 2), and the first dimension is dynamic)

```python
    def _info(self):
        return datasets.DatasetInfo(
            description=_DESCRIPTION,
            # This defines the different columns of the dataset and their types
            features=datasets.Features(
                {
                    "pose": datasets.features.Sequence(datasets.features.Array2D(shape=(137, 2), dtype="float32"))
                }
            ),
            homepage=_HOMEPAGE,
            citation=_CITATION,
        )
    def _generate_examples(self):
        """ Yields examples. """

        yield 1, {
            "pose": [np.zeros(shape=(137, 2), dtype=np.float32)]
        }
```

But this doesn't work -
> pyarrow.lib.ArrowNotImplementedError: MakeBuilder: cannot construct builder for type extension<arrow.py_extension_type>


Fix wikipedia custom config

It should be possible to use the wikipedia dataset with any `language` and `date`.
However it was not working as noticed in #784 . Indeed the custom wikipedia configurations were not enabled for some reason.

I fixed that and was able to run
```python
from datasets import load_dataset
load_dataset("./datasets/wikipedia", language="zh", date="20201120", beam_runner='DirectRunner')
```

cc @stvhuang @SamuelCahyawijaya

Fix #784


Very slow cold-start

dataset request

Requesting to add a new dataset

Hi,
I expect when importing `datasets` that nothing major happens in the background, and so the import should be insignificant.
When I load a metric, or a dataset, its fine that it takes time.

The following ranges from 3 to 9 seconds:
```
python -m timeit -n 1 -r 1 'from datasets import load_dataset'
```

edit:
sorry for the mis-tag, not sure how I added it.


Auto generate dummy data

When adding a new dataset to the library, dummy data creation can take some time.
To make things easier I added a command line tool that automatically generates dummy data when possible.

The tool only supports certain data files types: txt, csv, tsv, jsonl, json and xml.

Here are some examples:
```
python datasets-cli dummy_data ./datasets/snli --auto_generate
python datasets-cli dummy_data ./datasets/squad --auto_generate --json_field data
python datasets-cli dummy_data ./datasets/iwslt2017 --auto_generate --xml_tag seg --match_text_files "train*" --n_lines 15
# --xml_tag seg => each sample corresponds to a "seg" tag in the xml tree
# --match_text_files "train*" =>  also match text files that don't have a proper text file extension (no suffix like ".txt" for example)
# --n_lines 15 => some text files have headers so we have to use at least 15 lines
```

and here is the command usage:

```
usage: datasets-cli <command> [<args>] dummy_data [-h] [--auto_generate]
                                  [--n_lines N_LINES]
                                  [--json_field JSON_FIELD]
                                  [--xml_tag XML_TAG]
                                  [--match_text_files MATCH_TEXT_FILES]
                                  [--keep_uncompressed]
                                  [--cache_dir CACHE_DIR]
                                  path_to_dataset

positional arguments:
path_to_dataset       Path to the dataset (example: ./datasets/squad)

optional arguments:
-h, --help            show this help message and exit
--auto_generate       Try to automatically generate dummy data
--n_lines N_LINES     Number of lines or samples to keep when auto-
                                                generating dummy data
--json_field JSON_FIELD
                                                Optional, json field to read the data from when auto-
                                                generating dummy data. In the json data files, this
                                                field must point to a list of samples as json objects
                                                (ex: the 'data' field for squad-like files)
--xml_tag XML_TAG     Optional, xml tag name of the samples inside the xml
                                                files when auto-generating dummy data.
--match_text_files MATCH_TEXT_FILES
                                                Optional, a comma separated list of file patterns that
                                                looks for line-by-line text files other than *.txt or
                                                *.csv. Example: --match_text_files *.label
--keep_uncompressed   Don't compress the dummy data folders when auto-
                                                generating dummy data. Useful for debugging for to do
                                                manual adjustements before compressing.
--cache_dir CACHE_DIR
                                                Cache directory to download and cache files when auto-
                                                generating dummy data
```

The command generates all the necessary `dummy_data.zip` files (one per config).

How it works:
- it runs the split_generators() method of the dataset script to download the original data files
- when downloading it records a mapping between the downloaded files and the corresponding expected dummy data files paths
- then for each data file it creates the dummy data file keeping only the first samples (the strategy depends on the type of file)
- finally it compresses the dummy data folders into dummy_zip files ready for dataset tests

Let me know if that makes sense or if you have ideas to improve this tool !

I also added a unit test.


Downloading/caching only a part of a datasets' dataset.

enhancement

New feature or request

question

Further information is requested

Hi,
I want to use the validation data *only* (of natural question).
I don't want to have the whole dataset cached in my machine, just the dev set.
Is this possible? I can't find a way to do it in the docs.

Thank you,
Sapir


Update README.md

"no label" is "-" in the original dataset but "-1" in Huggingface distribution.


Use GCP download url instead of tensorflow custom download for boolq

BoolQ is a dataset that used tf.io.gfile.copy to download the file from a GCP bucket.
It prevented the dataset to be downloaded twice because of a FileAlreadyExistsError.
Even though the error could be fixed by providing `overwrite=True` to the tf.io.gfile.copy call, I changed the script to use GCP download urls and use regular downloads instead and remove the tensorflow dependency.

Fix #875


Add SQA

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** SQA (Sequential Question Answering) by Microsoft.
- **Description:** The SQA dataset was created to explore the task of answering sequences of inter-related questions on HTML tables. It has 6,066 sequences with 17,553 questions in total.
- **Paper:** https://www.microsoft.com/en-us/research/publication/search-based-neural-structured-learning-sequential-question-answering/
- **Data:** https://www.microsoft.com/en-us/download/details.aspx?id=54253
- **Motivation:** currently, the [Tapas](https://ai.googleblog.com/2020/04/using-neural-networks-to-find-answers.html) algorithm by Google AI is being added to the Transformers library (see https://github.com/huggingface/transformers/pull/8113). It would be great to use that model in combination with this dataset, on which it achieves SOTA results (average question accuracy of 0.71).

Note 1: this dataset actually consists of 2 types of files:
1) TSV files, containing the questions, answer coordinates and answer texts (for training, dev and test)
2) a folder of csv files, which contain the actual tabular data

Note 2: if you download the dataset straight from the download link above, then you will see that the `answer_coordinates` and `answer_text` columns are string lists of string tuples and strings respectively, which is not ideal. It would be better to make them true Python lists of tuples and strings respectively (using `ast.literal_eval`), before uploading them to the HuggingFace hub.

Adding this would be great! Then we could possibly also add [WTQ (WikiTable Questions)](https://github.com/ppasupat/WikiTableQuestions) and [TabFact (Tabular Fact Checking)](https://github.com/wenhuchen/Table-Fact-Checking) on which TAPAS also achieves state-of-the-art results. Note that the TAPAS algorithm requires these datasets to first be converted into the SQA format.

Instructions to add a new dataset can be found [here](https://huggingface.co/docs/datasets/share_dataset.html).


boolq does not load

dataset bug

A bug in a dataset script provided in the library

Hi
I am getting these errors trying to load boolq thanks

Traceback (most recent call last):
  File "test.py", line 5, in <module>
    data = AutoTask().get("boolq").get_dataset("train", n_obs=10)
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/tasks/tasks.py", line 42, in get_dataset
    dataset = self.load_dataset(split=split)
  File "/remote/idiap.svm/user.active/rkarimi/dev/internship/seq2seq/tasks/tasks.py", line 38, in load_dataset
    return datasets.load_dataset(self.task.name, split=split)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/boolq/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11/boolq.py", line 74, in _split_generators
    downloaded_files = dl_manager.download_custom(urls_to_download, tf.io.gfile.copy)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 150, in download_custom
    get_from_cache(url, cache_dir=cache_dir, local_files_only=True, use_etag=False)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 472, in get_from_cache
    f"Cannot find the requested files in the cached path at {cache_path} and outgoing traffic has been"
FileNotFoundError: Cannot find the requested files in the cached path at /idiap/home/rkarimi/.cache/huggingface/datasets/eaee069e38f6ceaa84de02ad088c34e63ec97671f2cd1910ddb16b10dc60808c and outgoing traffic has been disabled. To enable file online look-ups, set 'local_files_only' to False.


Loading Data From S3 Path in Sagemaker

enhancement

New feature or request

question

Further information is requested

In Sagemaker Im tring to load the data set from S3 path as follows

`train_path = 's3://xxxxxxxxxx/xxxxxxxxxx/train.csv'
    valid_path = 's3://xxxxxxxxxx/xxxxxxxxxx/validation.csv'
    test_path = 's3://xxxxxxxxxx/xxxxxxxxxx/test.csv'

    data_files = {}
    data_files["train"] = train_path
    data_files["validation"] = valid_path
    data_files["test"] = test_path
    extension = train_path.split(".")[-1]
    datasets = load_dataset(extension, data_files=data_files, s3_enabled=True)
    print(datasets)`

I getting an error of

`algo-1-7plil_1  |   File "main.py", line 21, in <module>
algo-1-7plil_1  |     datasets = load_dataset(extension, data_files=data_files)
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/load.py", line 603, in load_dataset
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 155, in __init__
algo-1-7plil_1  |     **config_kwargs,
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/site-packages/datasets/builder.py", line 305, in _create_builder_config
algo-1-7plil_1  |     m.update(str(os.path.getmtime(data_file)))
algo-1-7plil_1  |   File "/opt/conda/lib/python3.6/genericpath.py", line 55, in getmtime
algo-1-7plil_1  |     return os.stat(filename).st_mtime
algo-1-7plil_1  | FileNotFoundError: [Errno 2] No such file or directory: 's3://lsmv-sagemaker/pubmedbert/test.csv`

But when im trying with pandas , it is able to load from S3

Does the datasets library support S3 path to load


DataLoader(datasets) become more and more slowly within iterations

Hello, when I for loop my dataloader, the loading speed is becoming more and more slowly!
```
dataset = load_from_disk(dataset_path)  # around 21,000,000 lines

lineloader = tqdm(DataLoader(dataset, batch_size=1))
for idx, line in enumerate(lineloader):
     # do some thing for each line
```
In the begining, the loading speed is around 2000it/s, but after 1 minutes later, the speed is much slower, just around 800it/s.

And when I set `num_workers=4` in DataLoader, the loading speed is much lower, just 130it/s.

Could you please help me with this problem?
Thanks a lot!


imdb dataset cannot be loaded

Hi
I am trying to load the imdb train dataset

`dataset = datasets.load_dataset("imdb", split="train")`

getting following errors, thanks for your help
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 558, in _download_and_prepare
    verify_splits(self.info.splits, split_dict)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/info_utils.py", line 73, in verify_splits
    raise NonMatchingSplitsSizesError(str(bad_splits))
datasets.utils.info_utils.NonMatchingSplitsSizesError: [{'expected': SplitInfo(name='test', num_bytes=32660064, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='test', num_bytes=26476338, num_examples=20316, dataset_name='imdb')}, {'expected': SplitInfo(name='train', num_bytes=33442202, num_examples=25000, dataset_name='imdb'), 'recorded': SplitInfo(name='train', num_bytes=0, num_examples=0, dataset_name='imdb')}, {'expected': SplitInfo(name='unsupervised', num_bytes=67125548, num_examples=50000, dataset_name='imdb'), 'recorded': SplitInfo(name='unsupervised', num_bytes=0, num_examples=0, dataset_name='imdb')}]
>>> dataset = datasets.load_dataset("imdb", split="train")

```


bug in boolq dataset loading

Hi
I am trying to load boolq dataset:

```
import datasets
datasets.load_dataset("boolq")
```

I am getting the following errors, thanks for your help

```
>>> import datasets
2020-11-22 09:16:30.070332: W tensorflow/stream_executor/platform/default/dso_loader.cc:60] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2020-11-22 09:16:30.070389: I tensorflow/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
>>> datasets.load_dataset("boolq")
cahce dir  /idiap/temp/rkarimi/cache_home/datasets
cahce dir  /idiap/temp/rkarimi/cache_home/datasets
Using custom data configuration default
Downloading and preparing dataset boolq/default (download: 8.36 MiB, generated: 7.47 MiB, post-processed: Unknown size, total: 15.83 MiB) to /idiap/temp/rkarimi/cache_home/datasets/boolq/default/0.1.0/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11...
cahce dir  /idiap/temp/rkarimi/cache_home/datasets
cahce dir  /idiap/temp/rkarimi/cache_home/datasets/downloads
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/boolq/2987db1f15deaa19500ae24de560eabeaf1f8ef51df88c0470beeec72943bf11/boolq.py", line 74, in _split_generators
    downloaded_files = dl_manager.download_custom(urls_to_download, tf.io.gfile.copy)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 149, in download_custom
    custom_download(url, path)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/tensorflow/python/lib/io/file_io.py", line 516, in copy_v2
    compat.path_to_bytes(src), compat.path_to_bytes(dst), overwrite)
tensorflow.python.framework.errors_impl.AlreadyExistsError: file already exists

```


trec dataset unavailable

Hi
when I try to load the trec dataset I am getting these errors, thanks for your help

`datasets.load_dataset("trec",  split="train")
`
```
  File "<stdin>", line 1, in <module>
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/load.py", line 611, in load_dataset
    ignore_verifications=ignore_verifications,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 476, in download_and_prepare
    dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/builder.py", line 531, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File " /idiap/home/rkarimi/.cache/huggingface/modules/datasets_modules/datasets/trec/ca4248481ad244f235f4cf277186cad2ee8769f975119a2bbfc41b8932b88bd7/trec.py", line 140, in _split_generators
    dl_files = dl_manager.download_and_extract(_URLs)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 254, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/download_manager.py", line 179, in download
    num_proc=download_config.num_proc,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 225, in map_nested
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 225, in <listcomp>
    _single_map_nested((function, obj, types, None, True)) for obj in tqdm(iterable, disable=disable_tqdm)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/py_utils.py", line 163, in _single_map_nested
    return function(data_struct)
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 308, in cached_path
    use_etag=download_config.use_etag,
  File "/idiap/user/rkarimi/libs/anaconda3/envs/internship/lib/python3.7/site-packages/datasets/utils/file_utils.py", line 477, in get_from_cache
    raise ConnectionError("Couldn't reach {}".format(url))
ConnectionError: Couldn't reach http://cogcomp.org/Data/QA/QC/train_5500.label
```


load_dataset('cnn_dalymail', '3.0.0') gives a 'Not a directory' error

```
from datasets import load_dataset
dataset = load_dataset('cnn_dailymail', '3.0.0')
```
Stack trace:
```
---------------------------------------------------------------------------

NotADirectoryError                        Traceback (most recent call last)

<ipython-input-6-2e06a8332652> in <module>()
      1 from datasets import load_dataset
----> 2 dataset = load_dataset('cnn_dailymail', '3.0.0')

5 frames

/usr/local/lib/python3.6/dist-packages/datasets/load.py in load_dataset(path, name, data_dir, data_files, split, cache_dir, features, download_config, download_mode, ignore_verifications, save_infos, script_version, **config_kwargs)
    608         download_config=download_config,
    609         download_mode=download_mode,
--> 610         ignore_verifications=ignore_verifications,
    611     )
    612

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, try_from_hf_gcs, dl_manager, **download_and_prepare_kwargs)
    513                     if not downloaded_from_gcs:
    514                         self._download_and_prepare(
--> 515                             dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    516                         )
    517                     # Sync info

/usr/local/lib/python3.6/dist-packages/datasets/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    568         split_dict = SplitDict(dataset_name=self.name)
    569         split_generators_kwargs = self._make_split_generators_kwargs(prepare_split_kwargs)
--> 570         split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
    571
    572         # Checksums verification

/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/0128610a44e10f25b4af6689441c72af86205282d26399642f7db38fa7535602/cnn_dailymail.py in _split_generators(self, dl_manager)
    252     def _split_generators(self, dl_manager):
    253         dl_paths = dl_manager.download_and_extract(_DL_URLS)
--> 254         train_files = _subset_filenames(dl_paths, datasets.Split.TRAIN)
    255         # Generate shared vocabulary
    256

/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/0128610a44e10f25b4af6689441c72af86205282d26399642f7db38fa7535602/cnn_dailymail.py in _subset_filenames(dl_paths, split)
    153     else:
    154         logging.fatal("Unsupported split: %s", split)
--> 155     cnn = _find_files(dl_paths, "cnn", urls)
    156     dm = _find_files(dl_paths, "dm", urls)
    157     return cnn + dm

/root/.cache/huggingface/modules/datasets_modules/datasets/cnn_dailymail/0128610a44e10f25b4af6689441c72af86205282d26399642f7db38fa7535602/cnn_dailymail.py in _find_files(dl_paths, publisher, url_dict)
    132     else:
    133         logging.fatal("Unsupported publisher: %s", publisher)
--> 134     files = sorted(os.listdir(top_dir))
    135
    136     ret_files = []

NotADirectoryError: [Errno 20] Not a directory: '/root/.cache/huggingface/datasets/downloads/1bc05d24fa6dda2468e83a73cf6dc207226e01e3c48a507ea716dc0421da583b/cnn/stories'
```
I have ran the code on Google Colab


Add IndicGLUE dataset and Metrics

Added IndicGLUE benchmark for evaluating models on 11 Indian Languages. The descriptions of the tasks and the corresponding paper can be found [here](https://indicnlp.ai4bharat.org/indic-glue/)

- [x] Followed the instructions in CONTRIBUTING.md
- [x] Ran the tests successfully
- [x] Created the dummy data


Hi
I am using the dataset "iwslt2017-en-nl", and after downloading it I am getting this error when trying to evaluate it on T5-base with seq2seq_trainer.py in the huggingface repo could you assist me please? thanks

100%|████████████████████████████████████████████████████████████████████████████████████████████████████| 63/63 [02:47<00:00,  2.18s/it][libprotobuf FATAL /sentencepiece/src/../third_party/protobuf-lite/google/protobuf/repeated_field.h:1505] CHECK failed: (index) >= (0):
terminate called after throwing an instance of 'google::protobuf::FatalException'
  what():  CHECK failed: (index) >= (0):
run_t5_base_eval.sh: line 19:  5795 Aborted


[Feature Request] Add optional parameter in text loading script to preserve linebreaks

enhancement

New feature or request

I'm working on a project about rhyming verse using phonetic poetry and song lyrics, and line breaks are a vital part of the data.

I recently switched over to use the datasets library when my various corpora grew larger than my computer's memory. And so far, it is SO great.

But the first time I processed all of my data into a dataset, I hadn't realized the text loader script was processing the source files line-by-line and stripping off the newlines.

Once I caught the issue, I made my own data loader by modifying one line in the default text loader (changing `batch = batch.splitlines()` to `batch = batch.splitlines(True)` inside `_generate_tables`). And so I'm all set as far as my project is concerned.

But if my use case is more general, it seems like it'd be pretty trivial to add a kwarg to the default text loader called keeplinebreaks or something, which would default to False and get passed to `splitlines()`.


Update ner datasets infos

Update the dataset_infos.json files for changes made in #850 regarding the ner datasets feature types (and the change to ClassLabel)
I also fixed the ner types of conll2003


Consistent metric outputs

To automate the use of metrics, they should return consistent outputs.
In particular I'm working on adding a conversion of metrics to keras metrics.
To achieve this we need two things:
- have each metric return dictionaries of string -> floats since each keras metrics should return one float
- define in the metric info the different fields of the output dictionary

In this PR I'm adding these two features.
I also fixed a few bugs in some metrics

#867 needs to be merged first


Fix some metrics feature types

Replace `int` feature type to `int32` since `int` is not a pyarrow dtype in those metrics:
- accuracy
- precision
- recall
- f1
I also added the sklearn citation and used keyword arguments to remove future warnings


OSCAR from Inria group

dataset request

Requesting to add a new dataset

## Adding a Dataset
- **Name:** *OSCAR* (Open Super-large Crawled ALMAnaCH coRpus), multilingual parsing of Common Crawl (separate crawls for many different languages), [here](https://oscar-corpus.com/).
- **Description:** *OSCAR or Open Super-large Crawled ALMAnaCH coRpus is a huge multilingual corpus obtained by language classification and filtering of the Common Crawl corpus using the goclassy architecture.*
- **Paper:** *[here](https://hal.inria.fr/hal-02148693)*
- **Data:** *[here](https://oscar-corpus.com/)*
- **Motivation:** *useful for unsupervised tasks in separate languages. In an ideal world, your team would be able to obtain the unshuffled version, that could be used to train GPT-2-like models (the shuffled version, I suppose, could be used for translation).*

I am aware that you do offer the "colossal" Common Crawl dataset already, but this has the advantage to be available in many subcorpora for different languages.