Some error inside nlp.load_dataset()

First of all, nice work!

I am going through [this overview notebook](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb)

In simple step `dataset = nlp.load_dataset('squad', split='validation[:10%]')`

I get an error, which is connected with some inner code, I think:
```
---------------------------------------------------------------------------

TypeError                                 Traceback (most recent call last)

<ipython-input-8-d848d3a99b8c> in <module>()
      1 # Downloading and loading a dataset
      2
----> 3 dataset = nlp.load_dataset('squad', split='validation[:10%]')

8 frames

/usr/local/lib/python3.6/dist-packages/nlp/load.py in load_dataset(path, name, version, data_dir, data_files, split, cache_dir, download_config, download_mode, ignore_verifications, save_infos, **config_kwargs)
    515         download_mode=download_mode,
    516         ignore_verifications=ignore_verifications,
--> 517         save_infos=save_infos,
    518     )
    519

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in download_and_prepare(self, download_config, download_mode, ignore_verifications, save_infos, dl_manager, **download_and_prepare_kwargs)
    361                 verify_infos = not save_infos and not ignore_verifications
    362                 self._download_and_prepare(
--> 363                     dl_manager=dl_manager, verify_infos=verify_infos, **download_and_prepare_kwargs
    364                 )
    365                 # Sync info

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _download_and_prepare(self, dl_manager, verify_infos, **prepare_split_kwargs)
    414             try:
    415                 # Prepare split will record examples associated to the split
--> 416                 self._prepare_split(split_generator, **prepare_split_kwargs)
    417             except OSError:
    418                 raise OSError("Cannot find data file. " + (self.MANUAL_DOWNLOAD_INSTRUCTIONS or ""))

/usr/local/lib/python3.6/dist-packages/nlp/builder.py in _prepare_split(self, split_generator)
    585         fname = "{}-{}.arrow".format(self.name, split_generator.name)
    586         fpath = os.path.join(self._cache_dir, fname)
--> 587         examples_type = self.info.features.type
    588         writer = ArrowWriter(data_type=examples_type, path=fpath, writer_batch_size=self._writer_batch_size)
    589

/usr/local/lib/python3.6/dist-packages/nlp/features.py in type(self)
    460     @property
    461     def type(self):
--> 462         return get_nested_type(self)
    463
    464     @classmethod

/usr/local/lib/python3.6/dist-packages/nlp/features.py in get_nested_type(schema)
    370     # Nested structures: we allow dict, list/tuples, sequences
    371     if isinstance(schema, dict):
--> 372         return pa.struct({key: get_nested_type(value) for key, value in schema.items()})
    373     elif isinstance(schema, (list, tuple)):
    374         assert len(schema) == 1, "We defining list feature, you should just provide one example of the inner type"

/usr/local/lib/python3.6/dist-packages/nlp/features.py in <dictcomp>(.0)
    370     # Nested structures: we allow dict, list/tuples, sequences
    371     if isinstance(schema, dict):
--> 372         return pa.struct({key: get_nested_type(value) for key, value in schema.items()})
    373     elif isinstance(schema, (list, tuple)):
    374         assert len(schema) == 1, "We defining list feature, you should just provide one example of the inner type"

/usr/local/lib/python3.6/dist-packages/nlp/features.py in get_nested_type(schema)
    379         # We allow to reverse list of dict => dict of list for compatiblity with tfds
    380         if isinstance(inner_type, pa.StructType):
--> 381             return pa.struct(dict((f.name, pa.list_(f.type, schema.length)) for f in inner_type))
    382         return pa.list_(inner_type, schema.length)
    383

/usr/local/lib/python3.6/dist-packages/nlp/features.py in <genexpr>(.0)
    379         # We allow to reverse list of dict => dict of list for compatiblity with tfds
    380         if isinstance(inner_type, pa.StructType):
--> 381             return pa.struct(dict((f.name, pa.list_(f.type, schema.length)) for f in inner_type))
    382         return pa.list_(inner_type, schema.length)
    383

TypeError: list_() takes exactly one argument (2 given)
```


Update Overview.ipynb

Hi @lewtun, thanks for reporting.

It seems that our library fails at inferring the dtype of the columns:
- `milestone`
- `performed_via_github_app`

(and assigns them `null` dtype).

update notebook


remove webis

Remove webis from dataset folder.

Our first dataset script that only lives on AWS :-) https://s3.console.aws.amazon.com/s3/buckets/datasets.huggingface.co/nlp/datasets/webis/tl_dr/?region=us-east-1 @julien-c @jplu


[Newsroom] add newsroom

I checked it with the data link of the mail you forwarded @thomwolf => works well!


Xsum, require manual download of some files


[Tests] Local => aws

## Change default Test from local => aws

As a default we set` aws=True`, `Local=False`, `slow=False`

### 1. RUN_AWS=1 (default)
This runs 4 tests per dataset script.

a) Does the dataset script have a valid etag / Can it be reached on AWS?
b) Can we load its `builder_class`?
c) Can we load **all** dataset configs?
d) _Most importantly_: Can we load the dataset?

Important - we currently only test the first config of each dataset to reduce test time. Total test time is around 1min20s.

### 2. RUN_LOCAL=1 RUN_AWS=0

***This should be done when debugging dataset scripts of the ./datasets folder***

This only runs 1 test per dataset test, which is equivalent to aws d) - Can we load the dataset from the local `datasets` directory?

### 3. RUN_SLOW=1

We should set up to run these tests maybe 1 time per week ? @thomwolf

The `slow` tests include two more important tests.

e) Can we load the dataset with all possible configs? This test will probably fail at the moment because a lot of dummy data is missing. We should add the dummy data step by step to be sure that all configs work.

f) Test that the actual dataset can be loaded. This will take quite some time to run, but is important to make sure that the "real" data can be loaded. It will also test whether the dataset script has the correct checksums file which is currently not tested with `aws=True`. @lhoestq - is there an easy way to check cheaply whether the `dataset_info.json` is correct for each dataset script?


Final cleanup of readme and metrics


make style


🐛 `map` not working

I'm trying to run a basic example (mapping function to add a prefix).
[Here is the colab notebook I'm using.](https://colab.research.google.com/drive/1YH4JCAy0R1MMSc-k_Vlik_s1LEzP_t1h?usp=sharing)

```python
import nlp

dataset = nlp.load_dataset('squad', split='validation[:10%]')

def test(sample):
    sample['title'] = "test prefix @@@ " + sample["title"]
    return sample

print(dataset[0]['title'])
dataset.map(test)
print(dataset[0]['title'])
```
Output :
> Super_Bowl_50
Super_Bowl_50

Expected output :
> Super_Bowl_50
test prefix @@@ Super_Bowl_50


🐛 Colab : type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'

I'm trying to load CNN/DM dataset on Colab.

[Colab notebook](https://colab.research.google.com/drive/11Mf7iNhIyt6GpgA1dBEtg3cyMHmMhtZS?usp=sharing)

But I meet this error :

> AttributeError: type object 'pyarrow.lib.RecordBatch' has no attribute 'from_struct_array'


❓ How to apply a map to all subsets ?

I'm working with CNN/DM dataset, where I have 3 subsets : `train`, `test`, `validation`.

Should I apply my map function on the subsets one by one ?

```python
import nlp

cnn_dm = nlp.load_dataset('cnn_dailymail')
for corpus in ['train', 'test', 'validation']:
         cnn_dm[corpus] = cnn_dm[corpus].map(my_func)
```

Or is there a better way to do this ?


❓ How to remove specific rows of a dataset ?

I saw on the [example notebook](https://colab.research.google.com/github/huggingface/nlp/blob/master/notebooks/Overview.ipynb#scrollTo=efFhDWhlvSVC) how to remove a specific column :

```python
dataset.drop('id')
```

But I didn't find how to remove a specific row.

**For example, how can I remove all sample with `id` < 10 ?**


🐛 Trying to use ROUGE metric : pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323

metric bug

A bug in a metric script

I'm trying to use rouge metric.

I have to files : `test.pred.tokenized` and `test.gold.tokenized` with each line containing a sentence.
I tried :

```python
import nlp

rouge = nlp.load_metric('rouge')
with open("test.pred.tokenized") as p, open("test.gold.tokenized") as g:
    for lp, lg in zip(p, g):
            rouge.add(lp, lg)
```

But I meet following error :

> pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323

---

Full stack-trace :

```
Traceback (most recent call last):
  File "<stdin>", line 3, in <module>
  File "/home/me/.venv/transformers/lib/python3.6/site-packages/nlp/metric.py", line 224, in add
    self.writer.write_batch(batch)
  File "/home/me/.venv/transformers/lib/python3.6/site-packages/nlp/arrow_writer.py", line 148, in write_batch
    pa_table: pa.Table = pa.Table.from_pydict(batch_examples, schema=self._schema)
  File "pyarrow/table.pxi", line 1550, in pyarrow.lib.Table.from_pydict
  File "pyarrow/table.pxi", line 1503, in pyarrow.lib.Table.from_arrays
  File "pyarrow/public-api.pxi", line 390, in pyarrow.lib.pyarrow_wrap_table
  File "pyarrow/error.pxi", line 85, in pyarrow.lib.check_status
pyarrow.lib.ArrowInvalid: Column 1 named references expected length 534 but got length 323
```

(`nlp` installed from source)


I'm trying to access the information of CNN/DM dataset :

```python
cnn_dm = nlp.load_dataset('cnn_dailymail')
print(cnn_dm.info)
```

returns :

> AttributeError: 'dict' object has no attribute 'info'


Couldn't reach CNN/DM dataset

I can't get CNN / DailyMail dataset.

```python
import nlp

assert "cnn_dailymail" in [dataset.id for dataset in nlp.list_datasets()]
cnn_dm = nlp.load_dataset('cnn_dailymail')
```

[Colab notebook](https://colab.research.google.com/drive/1zQ3bYAVzm1h0mw0yWPqKAg_4EUlSx5Ex?usp=sharing)

gives following error :

```
ConnectionError: Couldn't reach https://s3.amazonaws.com/datasets.huggingface.co/nlp/cnn_dailymail/cnn_dailymail.py
```


Adding docstrings and some doc

Some doc


Qa4mre - add dataset

Added dummy data test only for the first config. Will do the rest later.
I had to do add some minor hacks to an important function to make it work.
There might be a cleaner way to handle it - can you take a look @thomwolf ?


[Clean-up] remove under construction datastes


fix reddit tifu dummy data


[Reclor] fix reclor

- That's probably one me. Could have made the manual data test more flexible. @mariamabarham


convert can use manual dir as second argument

@mariamabarham


add writer_batch_size to GeneratorBasedBuilder

You can now specify `writer_batch_size` in the builder arguments or directly in `load_dataset`


Add data dir test command


[New structure on AWS] Adapt paths

Some small changes so that we have the correct paths. @julien-c


Add trivia_q

Currently tested only for one config to pass tests. Needs to add more dummy data later.


[Manual downloads] add logic proposal for manual downloads and add wikihow

Wikihow is an example that needs to manually download two files as stated in: https://github.com/mahnazkoupaee/WikiHow-Dataset.

The user can then store these files under a hard-coded name: `wikihowAll.csv` and `wikihowSep.csv` in this case in a directory of his choice, e.g. `~/wikihow/manual_dir`.

The dataset can then be loaded via:

```python
import nlp
nlp.load_dataset("wikihow", data_dir="~/wikihow/manual_dir")
```

I added/changed so that there are explicit error messages when using manually downloaded files.


Run save infos

I replaced the old checksum file with the new `dataset_infos.json` by running the script on almost all the datasets we have. The only one that is still running on my side is the cornell dialog


[Reddit] add reddit

- Everything worked fine @mariamabarham. Made my computer nearly crash, but all seems to be working :-)


Add per type scores in seqeval metric

This PR add a bit more detail in the seqeval metric. Now the usage and output are:

```python
import nlp
met = nlp.load_metric('metrics/seqeval')
references = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
predictions =  [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
met.compute(predictions, references)

#Output: {'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.8}
```

It is also possible to compute scores for non IOB notations, POS tagging for example hasn't this kind of notation. Add `suffix` parameter:

```python
import nlp
met = nlp.load_metric('metrics/seqeval')
references = [['O', 'O', 'O', 'MISC', 'MISC', 'MISC', 'O'], ['PER', 'PER', 'O']]
predictions =  [['O', 'O', 'MISC', 'MISC', 'MISC', 'MISC', 'O'], ['PER', 'PER', 'O']]
met.compute(predictions, references, metrics_kwargs={"suffix": True})

#Output: {'PER': {'precision': 1.0, 'recall': 1.0, 'f1': 1.0, 'number': 1}, 'MISC': {'precision': 0.0, 'recall': 0.0, 'f1': 0, 'number': 1}, 'overall_precision': 0.5, 'overall_recall': 0.5, 'overall_f1': 0.5, 'overall_accuracy': 0.9}
```


[Cmrc 2018] fix cmrc2018


Webis tl-dr

Add the Webid TL:DR dataset.


[Csv] add tests for csv dataset script

Adds dummy data tests for csv.


lm1b

Add lm1b dataset.


Replace checksums files by Dataset infos json

### Better verifications when loading a dataset

I replaced the `urls_checksums` directory that used to contain `checksums.txt` and `cached_sizes.txt`, by a single file `dataset_infos.json`. It's just a dict `config_name` -> `DatasetInfo`.

It simplifies and improves how verifications of checksums and splits sizes are done, as they're all stored in `DatasetInfo` (one per config). Also, having already access to `DatasetInfo` enables to check disk space before running `download_and_prepare` for a given config.

The dataset infos json file is user readable, you can take a look at the squad one that I generated in this PR.

### Renaming

According to these changes, I did some renaming:
`save_checksums` -> `save_infos`
`ignore_checksums` -> `ignore_verifications`

for example, when you are creating a dataset you have to run
```nlp-cli test path/to/my/dataset --save_infos --all_configs```
instead of
```nlp-cli test path/to/my/dataset --save_checksums --all_configs```

### And now, the fun part

We'll have to rerun the `nlp-cli test ... --save_infos --all_configs` for all the datasets

-----------------

feedback appreciated !


Librispeech

Add librispeech dataset and remove some useless content.


Cleanup notebooks and various fixes

Fixes on dataset (more flexible) metrics (fix) and general clean ups


[WIP] add wmt14

WMT14 takes forever to download :-/

- WMT is the first dataset that uses an abstract class IMO, so I had to modify the `load_dataset_module` a bit.


[Paracrawl] add paracrawl

- Huge dataset - took ~1h to download
- Also this PR reformats all dataset scripts and adds `datasets` to `make style`


Add download gg drive

We can now add datasets that download from google drive


Add list and inspect methods - cleanup hf_api

Add a bunch of methods to easily list and inspect the processing scripts up-loaded on S3:
```python
nlp.list_datasets()
nlp.list_metrics()
# Copy and prepare the scripts at `local_path` for easy inspection/modification.
nlp.inspect_dataset(path, local_path)
# Copy and prepare the scripts at `local_path` for easy inspection/modification.
nlp.inspect_metric(path, local_path)
```

Also clean up the `HfAPI` to use `dataclasses` for better user-experience


Add wiki40b

This one is a beam dataset that downloads files using tensorflow.
I tested it on a small config and it works fine


Add Flores

Beautiful language for sure!


[Load => load_dataset] change naming

Rename leftovers @thomwolf


Add boolq

I just added the dummy data for this dataset.
This one was uses `tf.io.gfile.copy` to download the data but I added the support for custom download in the mock_download_manager. I also had to add a `tensorflow` dependency for tests.


[TedHrLr] add left dummy data


New datasets


[Datasets] add ted_hrlr

@thomwolf - After looking at `xnli` I think it's better to leave the translation features and add a `translation` key to make them work in our framework.

The result looks like this:
![Screenshot from 2020-05-12 18-34-43](https://user-images.githubusercontent.com/23423619/81721933-ee1faf00-9480-11ea-9e95-d6557cbd0ce0.png)

you can see that each split has a `translation` key which value is the nlp.features.Translation object.

That's a simple change. If it's ok for you, I will add dummy data for the other configs and treat the other translation scripts in the same way.


add tests

Tests for py_utils functions and for the BaseReader used to read from arrow and parquet.
I also removed unused utils functions.


Add nbytes + nexamples check

### Save size and number of examples
Now when you do `save_checksums`, it also create `cached_sizes.txt` right next to the checksum file.
This new file stores the bytes sizes and the number of examples of each split that has been prepared and stored in the cache. Example:

```
# Cached sizes: <full_config_name> <num_bytes> <num_examples>
hansards/house/1.0.0/test 22906629 122290
hansards/house/1.0.0/train 191459584 947969
hansards/senate/1.0.0/test 5711686 25553
hansards/senate/1.0.0/train 40324278 182135
```

### Check processing output

If there is a `caches_sizes.txt`, then each time we run `download_and_prepare` it will make sure that the sizes match. You can set `ignore_checksums=True` if you don't want that to happen.

### Fill Dataset Info

All the split infos and the checksums are now stored correctly in DatasetInfo after `download_and_prepare`

### Check space on disk before running `download_and_prepare`

Check if the space is lower than the sum of the sizes of the files in `checksums.txt` and `cached_files.txt`. This is not ideal though as it considers the files for all configs.

TODO:
A better way to do it would be to have save the `DatasetInfo` instead of the `checksums.txt` and `cached_sizes.txt`, in order to have one file per dataset config (and therefore consider only the sizes of the files for one config and not all of them). It can also be the occasion to factorize all the `download_and_prepare` verifications. Maybe next PR ?


[Convert] add new pattern


[Tests] skip beam dataset tests for now

For now we will skip tests for Beam Datasets


pin flake 8

Flake 8's new version does not like our format. Pinning the version for now.


WIP adding metrics

Adding the following metrics as identified by @mariamabarham:

1. BLEU:  BiLingual Evaluation Understudy: https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py,  https://github.com/chakki-works/sumeval/blob/master/sumeval/metrics/bleu.py (multilingual)
2. GLEU:  Google-BLEU:  https://github.com/cnap/gec-ranking/blob/master/scripts/compute_gleu
3. Sacrebleu: https://pypi.org/project/sacrebleu/1.4.8/ (pypi package), https://github.com/mjpost/sacrebleu (github implementation)
4. ROUGE: Recall-Oriented Understudy for Gisting Evaluation: https://github.com/google-research/google-research/tree/master/rouge, https://github.com/chakki-works/sumeval/blob/master/sumeval/metrics/rouge.py (multilingual)
5. Seqeval: https://github.com/chakki-works/seqeval (github implementation), https://pypi.org/project/seqeval/0.0.12/ (pypi package)
6. Coval:  coreference evaluation package for the CoNLL and ARRAU datasets https://github.com/ns-moosavi/coval
7. SQuAD v1 evaluation script
8. SQuAD V2 evaluation script: https://worksheets.codalab.org/rest/bundles/0x6b567e1cf2e041ec80d7098f031c5c9e/contents/blob/
9. GLUE
10. XNLI

Not now:
1. Perplexity: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/perplexity.py
2. Spearman: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/spearman_correlation.py
3. F1_measure: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/f1_measure.py
4. Pearson_corelation: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/pearson_correlation.py
5. AUC: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/auc.py
6. Entropy: https://github.com/allenai/allennlp/blob/master/allennlp/training/metrics/entropy.py


fix overflow check

I did some tests and unfortunately the test
```
pa_array.nbytes > MAX_BATCH_BYTES
```
doesn't work. Indeed for a StructArray, `nbytes` can be less 2GB even if there is an overflow (it loops...).

I don't think we can do a proper overflow test for the limit of 2GB...

For now I replaced it with a sanity check on the first element.


JSON script

Add a JSONS script to read JSON datasets from files.


[README dummy data tests] README to better understand how the dummy data structure works

In this PR a README.md is added to tests to shine more light on how the dummy data structure works. I try to explain the different possible cases. IMO the best way to understand the logic is to checkout the dummy data structure of the different datasets I mention in the README.md since those are the "edge cases".

@mariamabarham @thomwolf @lhoestq @jplu - I'd be happy to checkout the dummy data structure and get some feedback on possible improvements.


Fix arrow writer for big datasets using writer_batch_size

This PR fixes Yacine's bug.
According to [this](https://github.com/apache/arrow/blob/master/docs/source/cpp/arrays.rst#size-limitations-and-recommendations), it is not recommended to have pyarrow arrays bigger than 2Go.

Therefore I set a default batch size of 100 000 examples per batch. In general it shouldn't exceed 2Go. If it does, I reduce the batch_size on the fly, and I notify the user with a warning.


adding RACE, QASC, Super_glue and Tiny_shakespear datasets


fix cache dir in builder tests

minor fix


[CSV] re-add csv

Re-adding csv under the datasets under construction to keep circle ci happy - will have to see how to include it in the tests.

@lhoestq noticed that I accidently deleted it in https://github.com/huggingface/nlp/pull/63#discussion_r422263729.


[Tests] Test files locally

This PR adds a `aws` and a `local` decorator to the tests so that tests now run on the local datasets.

By default, the `aws` is deactivated and `local` is activated and `slow` is deactivated, so that only 1 test per dataset runs on circle ci.

**When local is activated all folders in `./datasets` are tested.**

**Important** When adding a dataset, we should no longer upload it to AWS. The steps are:
1. Open a PR
2. Add a dataset as described in `datasets/README.md`
3. If all tests pass, push to master

Currently we have 49 functional datasets in our code base.

We have 6 datasets "under-construction" that don't pass the tests - so I put them in a folder "datasets_under_construction" - it would be nice to open a PR to fix them and put them in the `datasets` folder.

**Important** when running tests locally, the datasets are cached so to rerun them delete your local cache via:
`rm -r ~/.cache/huggingface/datasets/*`

@thomwolf @mariamabarham @lhoestq


[Datasets] ReadME


fix math dataset and xcopa

- fixes math dataset and xcopa, uploaded both of the to S3


[Datasets] Make master ready for datasets adding

Add all relevant files so that datasets can now be added on master


[Dataset scripts] add all datasets scripts

As mentioned, we can have the canonical datasets in the master. For now I also want to include all the data as present on S3 to make the synchronization easier when uploading new datastes.

@mariamabarham @lhoestq @thomwolf - what do you think?

If this is ok for you, I can sync up the master with the `add_dataset` branch: https://github.com/huggingface/nlp/pull/37 so that master is up to date.


[Cached Path] Better error message

IMO returning `None` in this function only leads to confusion and is never helpful.


[Load] rename setup_module to prepare_module

rename setup_module to prepare_module due to issues with pytests `setup_module` function.
See: PR #59.


Update to simplify some datasets conversion

This PR updates the encoding of `Values` like `integers`, `boolean` and `float` to use python casting and avoid having to cast in the dataset scripts, as mentioned here: https://github.com/huggingface/nlp/pull/37#discussion_r420176626

We could also change (not included in this PR yet):
- `supervized_keys` to make them a NamedTuple instead of a dataclass, and
- handle specifically the `Translation` features.
as mentioned here: https://github.com/huggingface/nlp/pull/37#discussion_r421740236

@patrickvonplaten @mariamabarham tell me if you want these two last changes as well.


Fix tests

@patrickvonplaten I've broken a bit the tests with #25 while simplifying and re-organizing the `load.py` and `download_manager.py` scripts.

I'm trying to fix them here but I have a weird error, do you think you can have a look?
```bash
(datasets) MacBook-Pro-de-Thomas:datasets thomwolf$ python -m pytest -sv ./tests/test_dataset_common.py::DatasetTest::test_builder_class_snli
============================================================================= test session starts =============================================================================
platform darwin -- Python 3.7.7, pytest-5.4.1, py-1.8.1, pluggy-0.13.1 -- /Users/thomwolf/miniconda2/envs/datasets/bin/python
cachedir: .pytest_cache
rootdir: /Users/thomwolf/Documents/GitHub/datasets
plugins: xdist-1.31.0, forked-1.1.3
collected 1 item

tests/test_dataset_common.py::DatasetTest::test_builder_class_snli ERROR

=================================================================================== ERRORS ====================================================================================
____________________________________________________________ ERROR at setup of DatasetTest.test_builder_class_snli ____________________________________________________________

file_path = <module 'tests.test_dataset_common' from '/Users/thomwolf/Documents/GitHub/datasets/tests/test_dataset_common.py'>
download_config = DownloadConfig(cache_dir=None, force_download=False, resume_download=False, local_files_only=False, proxies=None, user_agent=None, extract_compressed_file=True, force_extract=True)
download_kwargs = {}

    def setup_module(file_path: str, download_config: Optional[DownloadConfig] = None, **download_kwargs,) -> DatasetBuilder:
        r"""
            Download/extract/cache a dataset to add to the lib from a path or url which can be:
                - a path to a local directory containing the dataset processing python script
                - an url to a S3 directory with a dataset processing python script

            Dataset codes are cached inside the lib to allow easy import (avoid ugly sys.path tweaks)
            and using cloudpickle (among other things).

            Return: tuple of
                the unique id associated to the dataset
                the local path to the dataset
        """
        if download_config is None:
            download_config = DownloadConfig(**download_kwargs)
        download_config.extract_compressed_file = True
        download_config.force_extract = True

>       name = list(filter(lambda x: x, file_path.split("/")))[-1] + ".py"
E       AttributeError: module 'tests.test_dataset_common' has no attribute 'split'

src/nlp/load.py:169: AttributeError
============================================================================== warnings summary ===============================================================================
/Users/thomwolf/miniconda2/envs/datasets/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15
  /Users/thomwolf/miniconda2/envs/datasets/lib/python3.7/site-packages/tensorflow_core/python/pywrap_tensorflow_internal.py:15: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

-- Docs: https://docs.pytest.org/en/latest/warnings.html
=========================================================================== short test summary info ===========================================================================
ERROR tests/test_dataset_common.py::DatasetTest::test_builder_class_snli - AttributeError: module 'tests.test_dataset_common' has no attribute 'split'
========================================================================= 1 warning, 1 error in 3.63s =========================================================================
```


Aborted PR - Fix tests


Better cached path

### Changes:
- The `cached_path` no longer returns None if the file is missing/the url doesn't work. Instead, it can raise `FileNotFoundError` (missing file), `ConnectionError` (no cache and unreachable url) or `ValueError` (parsing error)
- Fix requests to firebase API that doesn't handle HEAD requests...
- Allow custom download in datasets script: it allows to use `tf.io.gfile.copy` for example, to download from google storage. I added an example: the `boolq` script


[Dataset] Tester add mock function

need to add an empty `extract()` function to make `hansard` dataset test work.


Beam datasets

# Beam datasets

## Intro

Beam Datasets are using beam pipelines for preprocessing (basically lots of `.map` over objects called PCollections).
The advantage of apache beam is that you can choose which type of runner you want to use to preprocess your data. The main runners are:
- the `DirectRunner` to run the pipeline locally (default). However I encountered memory issues for big datasets (like the french or english wikipedia). Small dataset work fine
- Google Dataflow. I didn't play with it.
- Spark or Flink, two well known data processing frameworks. I tried to use the Spark/Flink local runners provided by apache beam for python and wasn't able to make them work properly though...

## From tfds beam datasets to our own beam datasets

Tensorflow datasets used beam and a complicated pipeline to shard the TFRecords files.
To allow users to download beam datasets and not having to preprocess them, they also allow to download the already preprocessed datasets from their google storage (the beam pipeline doesn't run in that case).

On our side, we replace TFRecords by something else. Arrow or Parquet do the job but I chose Parquet as: 1) there is a builtin apache beam parquet writer that is quite convenient, and 2) reading parquet from the pyarrow library is also simple and effective (there is a mmap option !)

Moreover we don't shard datasets in many many files like tfds (they were doing probably doing that mainly because of the limit of 2Gb per TFRecord file). Therefore we have a simpler pipeline that saves each split into one parquet file. We also removed the utilities to use their google storage (for now maybe ? we'll have to discuss it).

## Main changes

- Added a BeamWriter to save the output of beam pipelines into parquet files and fill dataset infos
- Create a ParquetReader and refactor a bit the arrow_reader.py

\> **With this, we can now try to add beam datasets from tfds**

I already added the wikipedia one, and I will also try to add the Wiki40b dataset

## Test the wikipedia script

You can download and run the beam pipeline for wikipedia (using the `DirectRunner` by default) like this:

```
>>> import nlp
>>> nlp.load("datasets/nlp/wikipedia", dataset_config="20200501.frr")
```

This wikipedia dataset (lang: frr, North Frisian) is a small one (~10Mb), but feel free to try bigger ones (and fill 20Gb of swap memory if you try the english one lol)

## Next

Should we allow to download preprocessed datasets from the tfds google storage ?
Should we try to optimize the beam pipelines to run locally without memory issues ?
Should we try other data processing frameworks for big datasets, like spark ?

## About this PR

It should be merged after #25

-----------------

I'd be happy to have your feedback and your ideas to improve the processing of big datasets like wikipedia :)


[Tests] Improved Error message for dummy folder structure

Improved Error message


[Features] Typo in generate_from_dict

Change `isinstance` test in features when generating features from dict.


allow dummy folder structure to handle dict of lists

`esnli.py` needs that extension of the dummy data testing.


[Testing] Improved testing structure

This PR refactors the test design a bit and puts the mock download manager in the `utils` files as it is just a test helper class.

as @mariamabarham pointed out, creating a dummy folder structure can be quite hard to grasp.
This PR tries to change that to some extent.

It follows the following logic for the `dummy` folder structure now:
1.) The data bulider has no config -> the  `dummy` folder structure is:
`dummy/<version>/dummy_data.zip`
2) The data builder has >= 1 configs -> the `dummy` folder structure is:
`dummy/<config_name_1>/<version>/dummy_data.zip`
`dummy/<config_name_2>/<version>/dummy_data.zip`

Now, the difficult part is how to create the `dummy_data.zip` file. There are two cases:
A) The `data_urs` parameter inserted into the `download_and_extract` fn is a **string**:
-> the `dummy_data.zip` file zips the folder:
`dummy_data/<relative_path_of_folder_structure_of_url>`
B) The `data_urs` parameter inserted into the `download_and_extract` fn is a **dict**:
-> the `dummy_data.zip` file zips the folder:
`dummy_data/<relative_path_of_folder_structure_of_url_behind _key_1>`
`dummy_data/<relative_path_of_folder_structure_of_url_behind _key_2>`

By relative folder structure I mean `url_path.split('./')[-1]`. As an example the dataset **xquad** by deepmind has the following url path behind the key `de`: `https://github.com/deepmind/xquad/blob/master/xquad.de.json`
-> This means that the relative url path should be `xquad.de.json`.

@mariamabarham B) is a change from how is was before and I think is makes more sense.
While before the `dummy_data.zip` file for xquad with config `de` looked like:
`dummy_data/de` it would now look like `dummy_data/xquad.de.json`. I think this is better and easier to understand.

Therefore there are currently 6 tests that would have to have changed their dummy folder structure, but which can easily be done (30min).

I also added a function: `print_dummy_data_folder_structure` that prints out the expected structures when testing which should be quite helpful.


[Tests] test only for fast test as a default

Test only for one config on circle ci to speed up testing. Add all config test as a slow test.
@mariamabarham @thomwolf


fix flatten nested


[Command Convert] remove tensorflow import

Remove all tensorflow import statements.


[PyArrow Feature] fix py arrow bool

To me it seems that `bool` can only be accessed with `bool_` when looking at the pyarrow types: https://arrow.apache.org/docs/python/api/datatypes.html.


[Features] Strip str key before dict look-up

The dataset `anli.py` currently fails because it tries to look up a key `1\n` in a dict that only has the key `1`. Added an if statement to strip key if it cannot be found in dict.


[Load] Separate Module kwargs and builder kwargs.

Kwargs for the `load_module` fn should be passed with `module_xxxx` to `builder_kwargs` of `load` fn.

This is a follow-up PR of: https://github.com/huggingface/nlp/pull/41


[Tests] Fix tests for datasets with no config

Forgot to fix `None` problem for datasets that have no config this in PR: https://github.com/huggingface/nlp/pull/42


[Checksums] If no configs exist prevent to run over empty list

`movie_rationales` e.g. has no configs.


[Tests] allow tests for builders without config

Some dataset scripts have no configs - the tests have to be adapted for this case.
In this case the dummy data will be saved as:
- natural_questions
  -> dummy
  -> -> 1.0.0 (version num)
  -> -> -> dummy_data.zip


[Load module] allow kwargs into load module

Currenly it is not possible to force a re-download of the dataset script.

This simple change allows to pass ``force_reload=True`` as ``builder_kwargs`` in the ``load.py`` function.


Update remote checksums instead of overwrite

When the user uploads a dataset on S3, checksums are also uploaded with the `--upload_checksums` parameter.

If the user uploads the dataset in several steps, then the remote checksums file was previously overwritten. Now it's going to be updated with the new checksums.


[Test] improve slow testing


[Checksums] Error for some datasets

The checksums command works very nicely for `squad`. But for `crime_and_punish` and `xnli`,
the same bug happens:

When running:
```
python nlp-cli nlp-cli test xnli --save_checksums
```

leads to:

```
  File "nlp-cli", line 33, in <module>
    service.run()
  File "/home/patrick/python_bin/nlp/commands/test.py", line 61, in run
    ignore_checksums=self._ignore_checksums,
  File "/home/patrick/python_bin/nlp/builder.py", line 383, in download_and_prepare
    self._download_and_prepare(dl_manager=dl_manager, download_config=download_config)
  File "/home/patrick/python_bin/nlp/builder.py", line 627, in _download_and_prepare
    dl_manager=dl_manager, max_examples_per_split=download_config.max_examples_per_split,
  File "/home/patrick/python_bin/nlp/builder.py", line 431, in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
  File "/home/patrick/python_bin/nlp/datasets/xnli/8bf4185a2da1ef2a523186dd660d9adcf0946189e7fa5942ea31c63c07b68a7f/xnli.py", line 95, in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
  File "/home/patrick/python_bin/nlp/utils/download_manager.py", line 246, in download_and_extract
    return self.extract(self.download(url_or_urls))
  File "/home/patrick/python_bin/nlp/utils/download_manager.py", line 186, in download
    self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
  File "/home/patrick/python_bin/nlp/utils/download_manager.py", line 166, in _record_sizes_checksums
    self._recorded_sizes_checksums[url] = get_size_checksum(path)
  File "/home/patrick/python_bin/nlp/utils/checksums_utils.py", line 81, in get_size_checksum
    with open(path, "rb") as f:
TypeError: expected str, bytes or os.PathLike object, not tuple
```


[Datasets ToDo-List] add datasets

## Description

This PR acts as a dashboard to see which datasets are added to the library and work.

Cicle-ci should always be green so that we can be sure that newly added datasets are functional.
This PR should not be merged.

## Progress

**For the following datasets the test commands**:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_<your-dataset-name>
```
and
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_dataset_all_configs_<your-dataset-name>
```

**passes**.

- [x] Squad
- [x] Sentiment140
- [x] XNLI
- [x] Crime_and_Punish
- [x] movie_rationales
- [x] ai2_arc
- [x] anli
- [x] event2Mind
- [x] Fquad
- [x] blimp
- [x] empathetic_dialogues
- [x] cosmos_qa
- [x] xquad
- [x] blog_authorship_corpus
- [x] SNLI
- [x] break_data
- [x] SQuAD v2
- [x] cfq
- [x] eraser_multi_rc
- [x] Glue
- [x] Tydiqa
- [x] wiki_qa
- [x] wikitext
- [x] winogrande
- [x] wiqa
- [x] esnli
- [x] civil_comments
- [x] commonsense_qa
- [x] com_qa
- [x] coqa
- [x] wiki_split
- [x] cos_e
- [x] xcopa
- [x] quarel
- [x] quartz
- [x] squad_it
- [x] quoref
- [x] squad_pt
- [x] cornell_movie_dialog
- [x] SciQ
- [x] Scifact
- [x] hellaswag
- [x] ted_multi (in translate)
- [x] Aeslc (summarization)
- [x] drop
- [x] gap
- [x] hansard
- [x] opinosis
- [x] MLQA
- [x] math_dataset

## How-To-Add a dataset

**Before adding a dataset make sure that your branch is up to date**:
1. `git checkout add_datasets`
2. `git pull`

**Add a dataset via the `convert_dataset.sh` bash script:**

Running `bash convert_dataset.sh <file/to/tfds/datascript.py>` (*e.g.* `bash convert_dataset.sh ../tensorflow-datasets/tensorflow_datasets/text/movie_rationales.py`) will automatically run all the steps mentioned in **Add a dataset manually** below.

Make sure that you run `convert_dataset.sh` from the root folder of `nlp`.

The conversion script should work almost always for step 1): "convert dataset script from tfds to nlp format" and 2) "create checksum file" and step 3) "make style".

It can also sometimes automatically run step 4) "create the correct dummy data from tfds", but this will only work if a) there is either no config name or only one config name and b) the `tfds testing/test_data/fake_example` is in the correct form.

Nevertheless, the script should always be run in the beginning until an error occurs to be more efficient.

If the conversion script does not work or fails at some step, then you can run the steps manually as follows:

**Add a dataset manually**

Make sure you run all of the following commands from the root of your `nlp` git clone.
Also make sure that you changed to this branch:
```
git checkout add_datasets
```

1) the tfds datascript file should be converted to `nlp` style:

```
python nlp-cli convert --tfds_path <path/to/tensorflow_datasets/text/your_dataset_name>.py --nlp_directory datasets/nlp
```

This will convert the tdfs script and create a folder with the correct name.

2) the checksum file should be added. Use the command:
```
python nlp-cli test datasets/nlp/<your-dataset-folder> --save_checksums --all_configs
```

A checksums.txt file should be created in your folder and the structure should look as follows:

squad/
├── squad.py/
└── urls_checksums/
...........└── checksums.txt

Delete the created `*.lock` file afterward - it should not be uploaded to AWS.

3) run black and isort on your newly added datascript files so that they look nice:

```
make style
```

4) the dummy data should be added. For this it might be useful to take a look into the structure of other examples as shown in the PR here and at `<path/to/tensorflow_datasets/testing/test_data/test_data/fake_examples>` whether the same  data can be used.

5)  the data can be uploaded to AWS using the command
```
aws s3 cp datasets/nlp/<your-dataset-folder> s3://datasets.huggingface.co/nlp/<your-dataset-folder> --recursive
```

6) check whether all works as expected using:
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_<your-dataset-name>
```
and
```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_dataset_all_configs_<your-dataset-name>
```

7) push to this PR and rerun the circle ci workflow to check whether circle ci stays green.

8) Edit this commend and tick off your newly added dataset :-)

## TODO-list

Maybe we can add a TODO-list here for everybody that feels like adding new datasets so that we will not add the same datasets.

Here a link to available datasets: https://docs.google.com/spreadsheets/d/1zOtEqOrnVQwdgkC4nJrTY6d-Av02u0XFzeKAtBM2fUI/edit#gid=0

Patrick:

- [ ] boolq - *weird download link*
- [ ] c4 - *beam dataset*


Metrics - refactoring, adding support for download and distributed metrics

Refactoring metrics to have a similar loading API than the datasets and improving the import system.

# Import system
The import system has ben upgraded. There are now three types of imports allowed:
1. `library` imports (identified as "absolute imports")
```python
import seqeval
```
=> we'll test all the imports before running the scripts and if one cannot be imported we'll display an error message like this one:
`ImportError: To be able to use this metric/dataset, you need to install the following dependencies ['seqeval'] using 'pip install seqeval' for instance'`

2. `internal` imports (identified as "relative imports")
```python
import .c4_utils
```
=> we'll assume this point to a file in the same directory/S3-directory as the main script and download this file.

2. `external` imports (identified as "relative imports" with a comment starting with `# From:`)
```python
from .nmt_bleu import compute_bleu  # From: https://github.com/tensorflow/nmt/blob/master/nmt/scripts/bleu.py
```
=> we'll assume this point to the URL of a python script (if it's a link to a github file, we'll take the raw file automatically).
=> the script is downloaded and renamed to the import name (here above renamed from `bleu.py` to `nmt_bleu.py`). Renaming the file can be necessary if the distant file has the same name as the dataset/metric processing script. If you forgot to rename the distant script and it has the same name as the dataset/metric, you'll have an explicit error message asking to rename the import anyway.

# Hosting metrics

Metrics are hosted on a S3 bucket like the dataset processing scripts.

# Metrics scripts

Metrics scripts have a lot in common with datasets processing scripts. They also have a `metric.info` including citations, descriptions and links to relevant pages.

Metrics have more documentation to supply to ensure they are used well.

Four examples are already included for reference in [./metrics](./metrics): BLEU, ROUGE, SacreBLEU and SeqEVAL.

# Automatic support for distributed/multi-processing metric computation

We've also added support for automatic distributed/multi-processing metric computation (e.g. when using DistributedDataParallel). We leverage our own dataset format for smart caching in this case.

Here is a quick gist of a standard use of metrics (the simplest usage):
```python
import nlp
bleu_metric = nlp.load_metric('bleu')

# If you only have a single iteration, you can easily compute the score like this
predictions = model(inputs)
score = bleu_metric.compute(predictions, references)

# If you have a loop, you can "add" your predictions and references at each iteration instead of having to save them yourself (the metric object store them efficiently for you)
for batch in dataloader:
    model_input, targets = batch
    predictions = model(model_inputs)
    bleu.add(predictions, targets)
score = bleu_metric.compute()  # Compute the score from all the stored predictions/references
```

Here is a quick gist of a use in a distributed torch setup (should work for any python multi-process setup actually). It's pretty much identical to the second example above:
```python
import nlp
# You need to give the total number of parallel python processes (num_process) and the id of each process (process_id)
bleu = nlp.load_metric('bleu', process_id=torch.distributed.get_rank(),b num_process=torch.distributed.get_world_size())

for batch in dataloader:
    model_input, targets = batch
    predictions = model(model_inputs)
    bleu.add(predictions, targets)
score = bleu_metric.compute()  # Compute the score on the first node by default (can be set to compute on each node as well)
```


[Tests] fix typo

@lhoestq - currently the slow test fail with:

```
_____________________________________________________________________________________ DatasetTest.test_load_real_dataset_xnli _____________________________________________________________________________________

self = <tests.test_dataset_common.DatasetTest testMethod=test_load_real_dataset_xnli>, dataset_name = 'xnli'

    @slow
    def test_load_real_dataset(self, dataset_name):
        with tempfile.TemporaryDirectory() as temp_data_dir:
>           dataset = load(dataset_name, data_dir=temp_data_dir)

tests/test_dataset_common.py:153:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
../../python_bin/nlp/load.py:497: in load
    dbuilder.download_and_prepare(**download_and_prepare_kwargs)
../../python_bin/nlp/builder.py:383: in download_and_prepare
    self._download_and_prepare(dl_manager=dl_manager, download_config=download_config)
../../python_bin/nlp/builder.py:627: in _download_and_prepare
    dl_manager=dl_manager, max_examples_per_split=download_config.max_examples_per_split,
../../python_bin/nlp/builder.py:431: in _download_and_prepare
    split_generators = self._split_generators(dl_manager, **split_generators_kwargs)
../../python_bin/nlp/datasets/xnli/8bf4185a2da1ef2a523186dd660d9adcf0946189e7fa5942ea31c63c07b68a7f/xnli.py:95: in _split_generators
    dl_dir = dl_manager.download_and_extract(_DATA_URL)
../../python_bin/nlp/utils/download_manager.py:246: in download_and_extract
    return self.extract(self.download(url_or_urls))
../../python_bin/nlp/utils/download_manager.py:186: in download
    self._record_sizes_checksums(url_or_urls, downloaded_path_or_paths)
../../python_bin/nlp/utils/download_manager.py:166: in _record_sizes_checksums
    self._recorded_sizes_checksums[url] = get_size_checksum(path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

path = ('', '/tmp/tmpkajlg9yc/downloads/c0f7773c480a3f2d85639d777e0e17e65527460310d80760fd3fc2b2f2960556.c952a63cb17d3d46e412ceb7dbcd656ce2b15cc9ef17f50c28f81c48a7c853b5')

    def get_size_checksum(path: str) -> Tuple[int, str]:
        """Compute the file size and the sha256 checksum of a file"""
        m = sha256()
>       with open(path, "rb") as f:
E       TypeError: expected str, bytes or os.PathLike object, not tuple

../../python_bin/nlp/utils/checksums_utils.py:81: TypeError
```

- the checksums probably need to be updated no? And we should also think about how to write a test for the checksums.


[Tests] add slow tests

This PR adds a slow test that downloads the "real" dataset. The test is decorated as "slow" so that it will not automatically run on circle ci.

Before uploading a dataset, one should test that this test passes, manually by running

```
RUN_SLOW=1 pytest tests/test_dataset_common.py::DatasetTest::test_load_real_dataset_<your-dataset-script-name>
```

This PR should be merged after PR: #33


Big cleanup/refactoring for clean serialization

This PR cleans many base classes to re-build them as `dataclasses`. We can thus use a simple serialization workflow for `DatasetInfo`, including it's `Features` and `SplitDict` based on `dataclasses` `asdict()`.

The resulting code is a lot shorter, can be easily serialized/deserialized, dataset info are human-readable and we can get rid of the `dataclass_json` dependency.

The scripts have breaking changes and the conversion tool is updated.

Example of dataset info in SQuAD script now:
```python
  def _info(self):
    return nlp.DatasetInfo(
        description=_DESCRIPTION,
        features=nlp.Features({
            "id":
                nlp.Value('string'),
            "title":
                nlp.Value('string'),
            "context":
                nlp.Value('string'),
            "question":
                nlp.Value('string'),
            "answers":
                nlp.Sequence({
                    "text": nlp.Value('string'),
                    "answer_start": nlp.Value('int32'),
                }),
        }),
        # No default supervised_keys (as we have to pass both question
        # and context as input).
        supervised_keys=None,
        homepage="https://rajpurkar.github.io/SQuAD-explorer/",
        citation=_CITATION,
    )
```

Example of serialized dataset info:
```bash
{
    "description": "Stanford Question Answering Dataset (SQuAD) is a reading comprehension dataset, consisting of questions posed by crowdworkers on a set of Wikipedia articles, where the answer to every question is a segment of text, or span, from the corresponding reading passage, or the question might be unanswerable.\n",
    "citation": "@article{2016arXiv160605250R,\n             author = {{Rajpurkar}, Pranav and {Zhang}, Jian and {Lopyrev},\n                                 Konstantin and {Liang}, Percy},\n                title = \"{SQuAD: 100,000+ Questions for Machine Comprehension of Text}\",\n            journal = {arXiv e-prints},\n                 year = 2016,\n                    eid = {arXiv:1606.05250},\n                pages = {arXiv:1606.05250},\narchivePrefix = {arXiv},\n             eprint = {1606.05250},\n}\n",
    "homepage": "https://rajpurkar.github.io/SQuAD-explorer/",
    "license": "",
    "features": {
        "id": {
            "dtype": "string",
            "_type": "Value"
        },
        "title": {
            "dtype": "string",
            "_type": "Value"
        },
        "context": {
            "dtype": "string",
            "_type": "Value"
        },
        "question": {
            "dtype": "string",
            "_type": "Value"
        },
        "answers": {
            "feature": {
                "text": {
                    "dtype": "string",
                    "_type": "Value"
                },
                "answer_start": {
                    "dtype": "int32",
                    "_type": "Value"
                }
            },
            "length": -1,
            "_type": "Sequence"
        }
    },
    "supervised_keys": null,
    "name": "squad",
    "version": {
        "version_str": "1.0.0",
        "description": "New split API (https://tensorflow.org/datasets/splits)",
        "nlp_version_to_prepare": null,
        "major": 1,
        "minor": 0,
        "patch": 0
    },
    "splits": {
        "train": {
            "name": "train",
            "num_bytes": 79426386,
            "num_examples": 87599,
            "dataset_name": "squad"
        },
        "validation": {
            "name": "validation",
            "num_bytes": 10491883,
            "num_examples": 10570,
            "dataset_name": "squad"
        }
    },
    "size_in_bytes": 0,
    "download_size": 35142551,
    "download_checksums": []
}
```


Fix map caching notebooks

Previously, caching results with `.map()` didn't work in notebooks.
To reuse a result, `.map()` serializes the functions with `dill.dumps` and then it hashes it.

The problem is that when using `dill.dumps` to serialize a function, it also saves its origin (filename + line no.) and the origin of all the `globals` this function needs. However for notebooks and shells, the filename looks like \<ipython-input-13-9ed2afe61d25\> and the line no. changes often.

To fix the problem, I added a new dispatch function for code objects that ignore the origin of the code if it comes from a notebook or a python shell.

I tested these cases in a notebook:
- lambda functions
- named functions
- methods
- classmethods
- staticmethods
- classes that implement `__call__`

The caching now works as expected for all of them :)
I also tested the caching in the demo notebook and it works fine !


[Circle ci] Install a virtual env before running tests

Install a virtual env before running tests to not running into sudo issues when dynamically downloading files.

Same number of tests now pass / fail as on my local computer:
![Screenshot from 2020-05-01 12-14-44](https://user-images.githubusercontent.com/23423619/80798814-8a0a0a80-8ba5-11ea-8db8-599d33bbfccd.png)


add metrics which require download files from github

To download files from github, I copied the `load_dataset_module` and its dependencies (without the builder) in `load.py` to `metrics/metric_utils.py`. I made the following changes:

- copy the needed files in a folder`metric_name`
- delete all other files that are not needed

For metrics that require an external import, I first create a `<metric_name>_imports.py` file which contains all external urls. Then I create a `<metric_name>.py` in which I will load the external files using `<metric_name>_imports.py`


Hf_api small changes

From Patrick:
```python
from nlp import hf_api
api = hf_api.HfApi()
api.dataset_list()
```

works :-)


[Circle ci] Adds circle ci config

@thomwolf can you take a look and set up circle ci on:
https://app.circleci.com/projects/project-dashboard/github/huggingface

I think for `nlp` only admins can set it up, which I guess is you :-)


[Cleanup] Removes all files in testing except test_dataset_common

As far as I know, all files in `tests` were old `tfds test files` so I removed them. We can still look them up on the other library.


[Tests] Clean tests

the abseil testing library (https://abseil.io/docs/python/quickstart.html) is better than the one I had before, so I decided to switch to that and changed the `setup.py` config file.
Abseil has more support and a cleaner API for parametrized testing I think.

I added a list of all dataset scripts that are currently on AWS, but will replace that once the
API is integrated into this lib.

One can now easily test for just a single function for a single dataset with:
`tests/test_dataset_common.py::DatasetTest::test_load_dataset_wikipedia`

NOTE: This PR is rebased on PR #29 so should be merged after.


Add script csv datasets

This is a PR allowing to create datasets from local CSV files. A usage might be:

```python
import nlp
ds = nlp.load(
    path="csv",
    name="bbc",
    dataset_files={
        nlp.Split.TRAIN: ["datasets/dummy_data/csv/train.csv"],
        nlp.Split.TEST: [""datasets/dummy_data/csv/test.csv""]
    },
    csv_kwargs={
        "skip_rows": 0,
        "delimiter": ",",
        "quote_char": "\"",
        "header_as_column_names": True
    }
)
```

```
Downloading and preparing dataset bbc/1.0.0 (download: Unknown size, generated: Unknown size, total: Unknown size) to /home/jplu/.cache/huggingface/datasets/bbc/1.0.0...
Dataset bbc downloaded and prepared to /home/jplu/.cache/huggingface/datasets/bbc/1.0.0. Subsequent calls will reuse this data.
{'test': Dataset(schema: {'category': 'string', 'text': 'string'}, num_rows: 49), 'train': Dataset(schema: {'category': 'string', 'text': 'string'}, num_rows: 99), 'validation': Dataset(schema: {'category': 'string', 'text': 'string'}, num_rows: 0)}
```

How it is read:

- `path`: the `csv` word means "I want to create a CSV dataset"
- `name`: the name of this dataset is `bbc`
- `dataset_files`: this is a dictionary where each key is the list of files corresponding to the key split.
- `csv_kwargs`: this is the keywords arguments to "explain" how to read the CSV files
  * `skip_rows`: number of rows have to be skipped, starting from the beginning of the file
  * `delimiter`: which delimiter is used to separate the columns
  * `quote_char`: which quote char is used to represent a column where the delimiter appears in one of them
  * `header_as_column_names`: will use the first row (header) of the file as name for the features. Otherwise the names will be automatically generated as `f1`, `f2`, etc... Will be applied after the `skip_rows` parameter.

**TODO**:  for now the `csv.py` is copied each time we create a new dataset as `ds_name.py`, this behavior will be modified to have only the `csv.py` script copied only once and not for all the CSV datasets.


Add checksums

### Checksums files

They are stored next to the dataset script in urls_checksums/checksums.txt.
They are used to check the integrity of the datasets downloaded files.
I kept the same format as tensorflow-datasets.
There is one checksums file for all configs.

### Load a dataset

When you do `load("squad")`, it will also download the checksums file and put it next to the script in nlp/datasets/hash/urls_checksums/checksums.txt.
It also verifies that the downloaded files checksums match the expected ones.

You can ignore checksum tests with `load("squad", ignore_checksums=True)` (under the hood it just adds `ignore_checksums=True` in the `DownloadConfig`)

### Test a dataset

There is a new command `nlp-cli test squad` that runs `download_and_prepare` to see if it runs ok, and that verifies that all the checksums match. Allowed arguments are `--name`, `--all_configs`, `--ignore_checksums` and `--register_checksums`.

### Register checksums

1. If the dataset has external dataset files

The command `nlp-cli test squad --register_checksums --all_configs` runs `download_and_prepare` on all configs to see if it runs ok, and it creates the checksums file.
You can also register one config at a time using `--name` instead ; the checksums file will be completed and not overwritten.

If the script is a local script, the checksum file is moved to urls_checksums/checksums.txt next to the local script, to enable the user to upload both the script and the checksums file afterwards with `nlp-cli upload squad`.

2. If the dataset files are all inside the directory of the dataset script

The user can directly do `nlp-cli upload squad --register_checksums`, as there is no need to download anything.
In this case however, all the dataset must be uploaded at once.

--

PS : it doesn't allow to register checksums for canonical datasets, the file has to be added manually on S3 for now (I guess ?)

Also I feel like we must be sure that this processes would not constrain too much any user from uploading its dataset.
Let me know what you think :)


Add metrics

This PR is a draft for adding metrics (sacrebleu and seqeval are added)

use case examples:
`import nlp`
**sacrebleu:**
```
refs = [['The dog bit the man.', 'It was not unexpected.', 'The man bit him first.'],
        ['The dog had bit the man.', 'No one was surprised.', 'The man had bitten the dog.']]
sys = ['The dog bit the man.', "It wasn't surprising.", 'The man had just bitten him.']
sacrebleu = nlp.load_metrics('sacrebleu')
print(sacrebleu.score)
```

**seqeval:**
```
y_true = [['O', 'O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
y_pred = [['O', 'O', 'B-MISC', 'I-MISC', 'I-MISC', 'I-MISC', 'O'], ['B-PER', 'I-PER', 'O']]
seqeval = nlp.load_metrics('seqeval')
print(seqeval.accuracy_score(y_true, y_pred)
print(seqeval.f1_score(y_true, y_pred)
```
_examples are taken from the corresponding web page_

your comments and suggestions are more than welcomed


adding bleu score code

this PR add the BLEU score metric to the lib. It can be tested by running the following code.

` from nlp.metrics import bleu

hyp1 = "It is a guide to action which ensures that the military always obeys the commands of the party"
ref1a = "It is a guide to action that ensures that the military  forces always being under the commands of the party "
 ref1b = "It is the guiding principle which guarantees the military force always being under the command of the Party"
ref1c = "It is the practical guide for the army always to heed the directions of the party"

list_of_references = [[ref1a, ref1b, ref1c]]
hypotheses = [hyp1]
bleu = bleu.bleu_score(list_of_references, hypotheses,4, smooth=True)
print(bleu) `


Cleanup Features - Updating convert command - Fix Download manager

This PR makes a number of changes:

# Updating `Features`

Features are a complex mechanism provided in `tfds` to be able to modify a dataset on-the-fly when serializing to disk and when loading from disk.

We don't really need this because (1) it hides too much from the user and (2) our datatype can be directly mapped to Arrow tables on drive so we usually don't need to change the format before/after serialization.

This PR extracts and refactors these features in a single `features.py` files. It still keep a number of features classes for easy compatibility with tfds, namely the `Sequence`, `Tensor`, `ClassLabel` and `Translation` features.

Some more complex features involving a pre-processing on-the-fly during serialization are kept:
- `ClassLabel` which are able to convert from label strings to integers,
- `Translation`which does some check on the languages.

# Updating the `convert` command

We do a few updates here
- following the simplification of the `features` (cf above), conversion are updated
- we also makes it simpler to convert a single file
- some code need to be fixed manually after conversion (e.g. to remove some encoding processing in former tfds `Text` features. We highlight this code with a "git merge conflict" style syntax for easy manual fixing.

# Fix download manager iterator

You kept me up quite late on Tuesday night with this `os.scandir` change @lhoestq ;-)


remove boto3 and promise dependencies

With the new download manager, we don't need `promise` anymore.
I also removed `boto3` as in [this pr](https://github.com/huggingface/transformers/pull/3968)


Replace tf.constant for TF

Replace simple tf.constant type of Tensor to tf.ragged.constant which allows to have examples of different size in a tf.data.Dataset.

Now the training works with TF. Here the same example than for the PT in collab:

```python
import tensorflow as tf
import nlp
from transformers import BertTokenizerFast, TFBertForQuestionAnswering

# Load our training dataset and tokenizer
train_dataset = nlp.load('squad', split="train[:1%]")
tokenizer = BertTokenizerFast.from_pretrained('bert-base-cased')

def get_correct_alignement(context, answer):
    start_idx = answer['answer_start'][0]
    text = answer['text'][0]
    end_idx = start_idx + len(text)
    if context[start_idx:end_idx] == text:
        return start_idx, end_idx       # When the gold label position is good
    elif context[start_idx-1:end_idx-1] == text:
        return start_idx-1, end_idx-1   # When the gold label is off by one character
    elif context[start_idx-2:end_idx-2] == text:
        return start_idx-2, end_idx-2   # When the gold label is off by two character
    else:
        raise ValueError()

# Tokenize our training dataset
def convert_to_features(example_batch):
    # Tokenize contexts and questions (as pairs of inputs)
    input_pairs = list(zip(example_batch['context'], example_batch['question']))
    encodings = tokenizer.batch_encode_plus(input_pairs, pad_to_max_length=True)

    # Compute start and end tokens for labels using Transformers's fast tokenizers alignement methods.
    start_positions, end_positions = [], []
    for i, (context, answer) in enumerate(zip(example_batch['context'], example_batch['answers'])):
        start_idx, end_idx = get_correct_alignement(context, answer)
        start_positions.append([encodings.char_to_token(i, start_idx)])
        end_positions.append([encodings.char_to_token(i, end_idx-1)])

    if start_positions and end_positions:
      encodings.update({'start_positions': start_positions,
                        'end_positions': end_positions})
    return encodings

train_dataset = train_dataset.map(convert_to_features, batched=True)

columns = ['input_ids', 'token_type_ids', 'attention_mask', 'start_positions', 'end_positions']
train_dataset.set_format(type='tensorflow', columns=columns)
features = {x: train_dataset[x] for x in columns[:3]}
labels = {"output_1": train_dataset["start_positions"]}
labels["output_2"] = train_dataset["end_positions"]
tfdataset = tf.data.Dataset.from_tensor_slices((features, labels)).batch(8)
model = TFBertForQuestionAnswering.from_pretrained("bert-base-cased")
loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(reduction=tf.keras.losses.Reduction.NONE, from_logits=True)
opt = tf.keras.optimizers.Adam(learning_rate=3e-5)
model.compile(optimizer=opt,
              loss={'output_1': loss_fn, 'output_2': loss_fn},
              loss_weights={'output_1': 1., 'output_2': 1.},
              metrics=['accuracy'])
model.fit(tfdataset, epochs=1, steps_per_epoch=3)
```


Updating caching mechanism - Allow dependency in dataset processing scripts - Fix style and quality in the repo

This PR has a lot of content (might be hard to review, sorry, in particular because I fixed the style in the repo at the same time).

# Style & quality:
You can now install the style and quality tools with `pip install -e .[quality]`. This will install black, the compatible version of sort and flake8.
You can then clean the style and check the quality before merging your PR with:
```bash
make style
make quality
```

# Allow dependencies in dataset processing scripts
We can now allow (some level) of imports in dataset processing scripts (in addition to PyPi imports).
Namely, you can do the two following things:

Import from a relative path to a file in the same folder as the dataset processing script:
```python
import .c4_utils
```

Or import from a relative path to a file in a folder/archive/github repo to which you provide an URL after the import state with `# From: [URL]`:
```python
import .clicr.dataset_code.build_json_dataset  # From: https://github.com/clips/clicr
```

In both these cases, after downloading the main dataset processing script, we will identify the location of these dependencies, download them and copy them in the dataset processing script folder.

Note that only direct import in the dataset processing script will be handled.
We don't recursively explore the additional import to download further files.
Also, when we download from an additional directory (in the second case above), we recursively add `__init__.py` to all the sub-folder so you can import from them.

This part is still tested for now. If you've seen datasets which required external utilities, tell me and I can test it.

# Update the cache to have a better local structure

The local structure in the `src/datasets` folder is now: `src/datasets/DATASET_NAME/DATASET_HASH/*`

The hash is computed from the full code of the dataset processing script as well as all the local and downloaded dependencies as mentioned above. This way if you change some code in a utility related to your dataset, a new hash should be computed.


Add Pandas as format type

As detailed in the title ^^


create our own DownloadManager

I tried to create our own - and way simpler - download manager, by replacing all the complicated stuff with our own `cached_path` solution.
With this implementation, I tried `dataset = nlp.load('squad')` and it seems to work fine.

For the implementation, what I did exactly:
- I copied the old download manager
- I removed all the dependences to the old `download` files
- I replaced all the download + extract calls by calls to `cached_path`
- I removed unused parameters (extract_dir, compute_stats) (maybe compute_stats could be re-added later if we want to compute stats...)
- I left some functions unimplemented for now. We will probably have to implement them because they are used by some datasets scripts (download_kaggle_data, iter_archive) or because we may need them at some point (download_checksums, _record_sizes_checksums)

Let me know if you think that this is going the right direction or if you have remarks.
Note: I didn't write any test yet as I wanted to read your remarks first


[Tests] General Test Design for all dataset scripts

The general idea is similar to how testing is done in `transformers`. There is one general `test_dataset_common.py` file which has a `DatasetTesterMixin` class. This class implements all of the logic that can be used in a generic way for all dataset classes. The idea is to keep each individual dataset test file as minimal as possible.

In order to test whether the specific data set class can download the data and generate the examples **without** downloading the actual data all the time, a MockDataLoaderManager class is used which receives a `mock_folder_structure_fn` function from each individual dataset test file that create "fake" data and which returns the same folder structure that would have been created when using the real data downloader.


[Download] Only create dir if not already exist

This was quite annoying to find out :D.
Some datasets have save in the same directory. So we should only create a new directory if it doesn't already exist.


[Make style]

Added Makefile and applied make style to all.
make style runs the following code:

```
style:
          black --line-length 119 --target-version py35 src
          isort --recursive src
```

It's the same code that is run in `transformers`.


[Map Function] add assert statement if map function does not return dict or None

IMO, if a function is provided that is not a print statement (-> returns variable of type `None`) or a function that updates the datasets (-> returns variable of type `dict`), then a `TypeError` should be raised.

Not sure whether you had cases in mind where the user should do something else @thomwolf , but I think a lot of silent errors can be avoided with this assert statement.


[Convert TFDS to HFDS] Extend script to also allow just converting a single file

Adds another argument to be able to convert only a single file


Name json file "squad.json" instead of "squad.py.json"


[Clean up] Datasets

Clean up `nlp/datasets` folder.

As I understood, eventually the `nlp/datasets` shall not exist anymore at all.

The folder `nlp/datasets/nlp` is kept for the moment, but won't be needed in the future, since it will live on S3 (actually it already does) at: `https://s3.console.aws.amazon.com/s3/buckets/datasets.huggingface.co/nlp/?region=us-east-1` and the different `dataset downloader scripts will be added to `nlp/src/nlp` when downloaded by the user.

The folder `nlp/datasets/checksums` is kept for now, but won't be needed anymore in the future.

The remaining folders/ files are leftovers from tensorflow-datasets and are not needed. The can be looked up in the private tensorflow-dataset repo.


Fix issue 6: error when the citation is missing in the DatasetInfo


Fix issue 5: allow empty datasets


Error when citation is not given in the DatasetInfo

The following error is raised when the `citation` parameter is missing when we instantiate a `DatasetInfo`:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/dev/jplu/datasets/src/nlp/info.py", line 338, in __repr__
    citation_pprint = _indent('"""{}"""'.format(self.citation.strip()))
AttributeError: 'NoneType' object has no attribute 'strip'
```

I propose to do the following change in the `info.py` file. The method:
```python
def __repr__(self):
        splits_pprint = _indent("\n".join(["{"] + [
                "    '{}': {},".format(k, split.num_examples)
                for k, split in sorted(self.splits.items())
        ] + ["}"]))
        features_pprint = _indent(repr(self.features))
        citation_pprint = _indent('"""{}"""'.format(self.citation.strip()))
        return INFO_STR.format(
                name=self.name,
                version=self.version,
                description=self.description,
                total_num_examples=self.splits.total_num_examples,
                features=features_pprint,
                splits=splits_pprint,
                citation=citation_pprint,
                homepage=self.homepage,
                supervised_keys=self.supervised_keys,
                # Proto add a \n that we strip.
                license=str(self.license).strip())
```
Becomes:
```python
def __repr__(self):
        splits_pprint = _indent("\n".join(["{"] + [
                "    '{}': {},".format(k, split.num_examples)
                for k, split in sorted(self.splits.items())
        ] + ["}"]))
        features_pprint = _indent(repr(self.features))
        ## the strip is done only is the citation is given
        citation_pprint = self.citation

        if self.citation:
            citation_pprint = _indent('"""{}"""'.format(self.citation.strip()))
        return INFO_STR.format(
                name=self.name,
                version=self.version,
                description=self.description,
                total_num_examples=self.splits.total_num_examples,
                features=features_pprint,
                splits=splits_pprint,
                citation=citation_pprint,
                homepage=self.homepage,
                supervised_keys=self.supervised_keys,
                # Proto add a \n that we strip.
                license=str(self.license).strip())
```
And now it is ok. @thomwolf are you ok with this fix?


ValueError when a split is empty

When a split is empty either TEST, VALIDATION or TRAIN I get the following error:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/jplu/dev/jplu/datasets/src/nlp/load.py", line 295, in load
    ds = dbuilder.as_dataset(**as_dataset_kwargs)
  File "/home/jplu/dev/jplu/datasets/src/nlp/builder.py", line 587, in as_dataset
    datasets = utils.map_nested(build_single_dataset, split, map_tuple=True)
  File "/home/jplu/dev/jplu/datasets/src/nlp/utils/py_utils.py", line 158, in map_nested
    for k, v in data_struct.items()
  File "/home/jplu/dev/jplu/datasets/src/nlp/utils/py_utils.py", line 158, in <dictcomp>
    for k, v in data_struct.items()
  File "/home/jplu/dev/jplu/datasets/src/nlp/utils/py_utils.py", line 172, in map_nested
    return function(data_struct)
  File "/home/jplu/dev/jplu/datasets/src/nlp/builder.py", line 601, in _build_single_dataset
    split=split,
  File "/home/jplu/dev/jplu/datasets/src/nlp/builder.py", line 625, in _as_dataset
    split_infos=self.info.splits.values(),
  File "/home/jplu/dev/jplu/datasets/src/nlp/arrow_reader.py", line 200, in read
    return py_utils.map_nested(_read_instruction_to_ds, instructions)
  File "/home/jplu/dev/jplu/datasets/src/nlp/utils/py_utils.py", line 172, in map_nested
    return function(data_struct)
  File "/home/jplu/dev/jplu/datasets/src/nlp/arrow_reader.py", line 191, in _read_instruction_to_ds
    file_instructions = make_file_instructions(name, split_infos, instruction)
  File "/home/jplu/dev/jplu/datasets/src/nlp/arrow_reader.py", line 104, in make_file_instructions
    absolute_instructions=absolute_instructions,
  File "/home/jplu/dev/jplu/datasets/src/nlp/arrow_reader.py", line 122, in _make_file_instructions_from_absolutes
    'Split empty. This might means that dataset hasn\'t been generated '
ValueError: Split empty. This might means that dataset hasn't been generated yet and info not restored from GCS, or that legacy dataset is used.
```

How to reproduce:
```python
import csv

import nlp

class Bbc(nlp.GeneratorBasedBuilder):
    VERSION = nlp.Version("1.0.0")

    def __init__(self, **config):
        self.train = config.pop("train", None)
        self.validation = config.pop("validation", None)
        super(Bbc, self).__init__(**config)

    def _info(self):
        return nlp.DatasetInfo(builder=self, description="bla", features=nlp.features.FeaturesDict({"id": nlp.int32, "text": nlp.string, "label": nlp.string}))

    def _split_generators(self, dl_manager):
        return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"filepath": self.train}),
                nlp.SplitGenerator(name=nlp.Split.VALIDATION, gen_kwargs={"filepath": self.validation}),
                nlp.SplitGenerator(name=nlp.Split.TEST, gen_kwargs={"filepath": None})]

    def _generate_examples(self, filepath):
        if not filepath:
            return None, {}

        with open(filepath) as f:
            reader = csv.reader(f, delimiter=',', quotechar="\"")
            lines = list(reader)[1:]

            for idx, line in enumerate(lines):
                yield idx, {"id": idx, "text": line[1], "label": line[0]}
```

```python
import nlp
dataset = nlp.load("bbc", builder_kwargs={"train": "bbc/data/train.csv", "validation": "bbc/data/test.csv"})
```


[Feature] Keep the list of labels of a dataset as metadata

It would be useful to keep the list of the labels of a dataset as metadata. Either directly in the `DatasetInfo` or in the Arrow metadata.


[Feature] More dataset outputs

Add the following dataset outputs:

- Spark
- Pandas


Issue to read a local dataset

Hello,

As proposed by @thomwolf, I open an issue to explain what I'm trying to do without success. What I want to do is to create and load a local dataset, the script I have done is the following:
```python
import os
import csv

import nlp

class BbcConfig(nlp.BuilderConfig):
    def __init__(self, **kwargs):
        super(BbcConfig, self).__init__(**kwargs)

class Bbc(nlp.GeneratorBasedBuilder):
    _DIR = "./data"
    _DEV_FILE = "test.csv"
    _TRAINING_FILE = "train.csv"

    BUILDER_CONFIGS = [BbcConfig(name="bbc", version=nlp.Version("1.0.0"))]

    def _info(self):
        return nlp.DatasetInfo(builder=self, features=nlp.features.FeaturesDict({"id": nlp.string, "text": nlp.string, "label": nlp.string}))

    def _split_generators(self, dl_manager):
        files = {"train": os.path.join(self._DIR, self._TRAINING_FILE), "dev": os.path.join(self._DIR, self._DEV_FILE)}

        return [nlp.SplitGenerator(name=nlp.Split.TRAIN, gen_kwargs={"filepath": files["train"]}),
                nlp.SplitGenerator(name=nlp.Split.VALIDATION, gen_kwargs={"filepath": files["dev"]})]

    def _generate_examples(self, filepath):
        with open(filepath) as f:
            reader = csv.reader(f, delimiter=',', quotechar="\"")
            lines = list(reader)[1:]

            for idx, line in enumerate(lines):
                yield idx, {"idx": idx, "text": line[1], "label": line[0]}

```

The dataset is attached to this issue as well:
[data.zip](https://github.com/huggingface/datasets/files/4476928/data.zip)

Now the steps to reproduce what I would like to do:
1. unzip data locally (I know the nlp lib can detect and extract archives but I want to reduce and facilitate the reproduction as much as possible)
2. create the `bbc.py` script as above at the same location than the unziped `data` folder.

Now I try to load the dataset in three different ways and none works, the first one with the name of the dataset like I would do with TFDS:
```python
import nlp
from bbc import Bbc
dataset = nlp.load("bbc")
```

I get:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 280, in load
    dbuilder: DatasetBuilder = builder(path, name, data_dir=data_dir, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 166, in builder
    builder_cls = load_dataset(path, name=name, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 88, in load_dataset
    local_files_only=local_files_only,
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/utils/file_utils.py", line 214, in cached_path
    if not is_zipfile(output_path) and not tarfile.is_tarfile(output_path):
  File "/opt/anaconda3/envs/transformers/lib/python3.7/zipfile.py", line 203, in is_zipfile
    with open(filename, "rb") as fp:
TypeError: expected str, bytes or os.PathLike object, not NoneType
```

But @thomwolf told me that no need to import the script, just put the path of it, then I tried three different way to do:
```python
import nlp
dataset = nlp.load("bbc.py")
```
And
```python
import nlp
dataset = nlp.load("./bbc.py")
```
And
```python
import nlp
dataset = nlp.load("/absolute/path/to/bbc.py")
```

These three ways gives me:
```
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 280, in load
    dbuilder: DatasetBuilder = builder(path, name, data_dir=data_dir, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 166, in builder
    builder_cls = load_dataset(path, name=name, **builder_kwargs)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/site-packages/nlp/load.py", line 124, in load_dataset
    dataset_module = importlib.import_module(module_path)
  File "/opt/anaconda3/envs/transformers/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 965, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'nlp.datasets.2fd72627d92c328b3e9c4a3bf7ec932c48083caca09230cebe4c618da6e93688.bbc'
```
Any idea of what I'm missing? or I might have spot a bug :)


changing nlp.bool to nlp.bool_