accessing_the_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Accessing the Data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In order to be able to access the data on Hugging Face Hub, we must import the\n",
    "necessary libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the load_dataset function from the datasets module\n",
    "from datasets import load_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After importing the modules, we set a few variables that will be used throughout\n",
    "this demo."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# path to the dataset repository on the Hugging Face Hub\n",
    "path = \"molssiai-hub/pubchemqc-b3lyp\"\n",
    "\n",
    "# set the dataset configuration/subset name\n",
    "name = \"b3lyp_pm6\"\n",
    "\n",
    "# set the dataset split\n",
    "split = \"train\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this tutorial, we are going to work with the\n",
    "[PubChemQC-B3LYP/6-31G*//PM6\n",
    "Dataset](https://huggingface.co/datasets/molssiai-hub/pubchemqc-b3lyp)\n",
    "(PubChemQC-B3LYP for short) from the [PubChemQC dataset\n",
    "collection](https://huggingface.co/collections/molssiai-hub/pubchemqc-datasets-669e5482260861ba7cce3d1c).\n",
    "Let us load the dataset as shown below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# load the dataset\n",
    "hub_dataset = load_dataset(path=path,\n",
    "                           name=name,\n",
    "                           split=split,\n",
    "                           streaming=True,\n",
    "                           trust_remote_code=True)\n",
    "\n",
    "hub_dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "PubChemQC datasets are very large in size and downloading them to your local\n",
    "machine can be a heavy lift for your internet network and disk storage.\n",
    "Therefore, we set the `streaming` parameter to `True` to avoid downloading the\n",
    "dataset on disk and ensure streaming the data from the hub. In the `streaming`\n",
    "mode, the `load_dataset` function returns an `IterableDataset` object which\n",
    "allows iterative access to the data. The `trust_remote_code` argument is also\n",
    "set to `True` to allow the usage of a custom [load\n",
    "script](https://huggingface.co/datasets/molssiai-hub/pubchemqc-b3lyp/blob/main/pubchemqc-b3lyp.py)\n",
    "which is in charge of preprocessing the data.\n",
    "\n",
    "The PubChemQC-B3LYP dataset is made of several files called `shards` that enable\n",
    "multiprocessing and parallelization of the data loading process. Multiprocessing\n",
    "can speed up the loading process significantly if you are absolutely sure that\n",
    "you have enough CPU cores to handle the load and enough memory and storage space\n",
    "to download and store the data.\n",
    "\n",
    "You can choose the number of processes to use for loading the data by setting\n",
    "the `num_proc` parameter in the `load_dataset` function.\n",
    "\n",
    "```python\n",
    "\n",
    "    >>> dataset = load_dataset(path=path,\n",
    "                               split=split,\n",
    "                               streaming=False,\n",
    "                               trust_remote_code=True,\n",
    "                               num_proc=4)\n",
    "```\n",
    "\n",
    "Note that we have set the `streaming` parameter to `False` in the code snippet\n",
    "above. This allows the `load_dataset` function to download the dataset on disk\n",
    "and load it into memory as a `Dataset` object which can access the data using\n",
    "fancy indexing and slicing."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once we create our `Dataset` or `IterableDataset` instance, we can access its\n",
    "features or column names using the `features` or `column_names` attributes of\n",
    "the dataset object.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# print the column names\n",
    "hub_dataset.column_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use `take(n)` to fetch the first $n$ examples from the dataset.\n",
    "For demonstration purposes, we set $n$ to 2 which yields a list of\n",
    "two dictionaries, each containing the features of the corresponding\n",
    "data point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "list(hub_dataset.take(2))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "hface",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}