How can I see summary statistics (e.g. number of samples; type of data) of HuggingFace datasets?
I'm looking for suitable datasets to test a few new machine learning ideas. Is there any way to see summary statistics (e.g. number of samples; type of data) of HuggingFace datasets?
They provide descriptions here https://huggingface.co/datasets , but it's a bit hard to filter them.

Not sure if I have missed the obvious, but I think you have to code it by yourself. When you use list_datasets, you only get general information for each dataset:
from datasets import list_datasets
list_datasets(with_details=True)[1].__dict__

Output:
{'id': 'ag_news',
 'key': 'datasets/datasets/ag_news/ag_news.py',
 'lastModified': '2020-09-15T08:26:31.000Z',
 'description': &quot;AG is a collection of more than 1 million news articles. News articles have been\ngathered from more than 2000 news sources by ComeToMyHead in more than 1 year of\nactivity. ComeToMyHead is an academic news search engine which has been running\nsince July, 2004. The dataset is provided by the academic comunity for research\npurposes in data mining (clustering, classification, etc), information retrieval\n(ranking, search, etc), xml, data compression, data streaming, and any other\nnon-commercial activity. For more information, please refer to the link\nhttp://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html .\n\nThe AG's news topic classification dataset is constructed by Xiang Zhang\n(xiang.zhang@nyu.edu) from the dataset above. It is used as a text\nclassification benchmark in the following paper: Xiang Zhang, Junbo Zhao, Yann\nLeCun. Character-level Convolutional Networks for Text Classification. Advances\nin Neural Information Processing Systems 28 (NIPS 2015).&quot;,
 'citation': '@inproceedings{Zhang2015CharacterlevelCN,\n  title={Character-level Convolutional Networks for Text Classification},\n  author={Xiang Zhang and Junbo Jake Zhao and Yann LeCun},\n  booktitle={NIPS},\n  year={2015}\n}',
 'size': 3991,
 'etag': '&quot;560ac59ac8cb6f76ac4180562a7f9342&quot;',
 'siblings': [datasets.S3Object('ag_news.py'),
  datasets.S3Object('dataset_infos.json'),
  datasets.S3Object('dummy/0.0.0/dummy_data.zip')],
 'author': None,
 'numModels': 1}

What you are actually looking for is the information that is provided by load_dataset:
from datasets import load_dataset
squad = load_dataset('squad')
squad

Output:
 DatasetDict({'train': Dataset(features: {'id': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'context': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'answers': Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None)}, num_rows: 87599), 'validation': Dataset(features: {'id': Value(dtype='string', id=None), 'title': Value(dtype='string', id=None), 'context': Value(dtype='string', id=None), 'question': Value(dtype='string', id=None), 'answers': Sequence(feature={'text': Value(dtype='string', id=None), 'answer_start': Value(dtype='int32', id=None)}, length=-1, id=None)}, num_rows: 10570)})

Here you get the number of samples for each split (num_rows) and the datatype of each feature. But load_dataset will load the whole dataset which can be an undesired behavior and therefore should be rejected for performance reasons.
An alternative is the following as far as I have not overlooked a parameter that allows to only load the dataset_infos.json of each dataset:
import datasets
import requests
from datasets import list_datasets
from datasets.utils.file_utils import REPO_DATASETS_URL

sets = list_datasets()
version = datasets.__version__
name = 'dataset_infos.json'
summary =[]

for d in sets:
     print('loading {}'.format(d))
     try:
         r = requests.get(REPO_DATASETS_URL.format(version=version, path=d, name=name))
         summary.append(r.json())
     except:
         print('Could not load {}'.format(d))

#the features and splits values are probably interesting for you
print(summary[0]['default']['features'])
print(summary[0]['default']['splits'])

Output:
{'email_body': {'dtype': 'string', 'id': None, '_type': 'Value'}, 'subject_line': {'dtype': 'string', 'id': None, '_type': 'Value'}}
{'test': {'name': 'test', 'num_bytes': 1384177, 'num_examples': 1906, 'dataset_name': 'aeslc'}, 'train': {'name': 'train', 'num_bytes': 11902668, 'num_examples': 14436, 'dataset_name': 'aeslc'}, 'validation': {'name': 'validation', 'num_bytes': 1660730, 'num_examples': 1960, 'dataset_name': 'aeslc'}}

P.S.: I haven't check the dataset_infos.json of the datasets that weren't loaded. They have probably a more complex structure or errors inside.
