add tutorial to docs
MODIFY
docs/conf.py
docs/conf.py
@@ -11,14 +11,14 @@ author = "Pangeo Community"
# -- General configuration ---------------------------------------------------
extensions = [
- "myst_parser",
+ "myst_nb",
"sphinx.ext.autodoc",
# "numpydoc",
"sphinx_autodoc_typehints",
]
templates_path = ["_templates"]
-exclude_patterns = []
+exclude_patterns = ["_build", "**.ipynb_checkpoints"]
master_doc = "index"
# -- Options for HTML output -------------------------------------------------

MODIFY
docs/index.md
docs/index.md
@@ -35,6 +35,7 @@ For more information, see {doc}`bakeries`.
:caption: Contents
recipes
+recipe_tutorial
execution
bakeries
contribute

ADD
docs/recipe_tutorial.ipynb
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Recipe Tutorial\n",
+ "\n",
+ "This tutorial describes how to create a recipe from scratch.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Step 1: Get to know your source data\n",
+ "\n",
+ "If you are developing a new recipe, you are probably starting from an existing\n",
+ "datset. The first step is to just get to know the dataset. For this tutorial,\n",
+ "our example will be the _NOAA Optimum Interpolation Sea Surface Temperature\n",
+ "(OISST) v2.1_. The authoritative website describing the data is\n",
+ "<https://www.ncdc.noaa.gov/oisst/optimum-interpolation-sea-surface-temperature-oisst-v21>.\n",
+ "This website contains links to the actual data files on the\n",
+ "[data access](https://www.ncdc.noaa.gov/oisst/data-access) page. We will use the\n",
+ "_AVHRR-Only_ version of the data and follow the corresponding link to the\n",
+ "[Gridded netCDF Data](https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/).\n",
+ "Browsing through the directories, we can see that there is one file per day. The\n",
+ "very first day of the dataset is stored at the following URL:\n",
+ "\n",
+ "```text\n",
+ "https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198109/oisst-avhrr-v02r01.19810901.nc\n",
+ "```\n",
+ "\n",
+ "From this example, we can work out the pattern of the file naming conventions.\n",
+ "But first, let's just download one of the files and open it up.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "! curl -O https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198109/oisst-avhrr-v02r01.19810901.nc "
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import xarray as xr\n",
+ "\n",
+ "ds = xr.open_dataset(\"oisst-avhrr-v02r01.19810901.nc\")\n",
+ "ds"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "We can see there are four data variables, all with dimension\n",
+ "`(time, zlev, lat, lon)`. There is a _dimension coordinate_ for each dimension,\n",
+ "and no _non-dimension coordinates_. Each file in the sequence presumably has the\n",
+ "same `zlev`, `lat`, and `lon`, but we expect `time` to be different in each one.\n",
+ "\n",
+ "Let's also check the total size of the dataset in the file.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "print(f\"File size is {ds.nbytes/1e6} MB\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "The file size is important because it will help us define the _chunk size_\n",
+ "Pangeo Forge will use to build up the target dataset.\n"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Step 2: Pick a Recipe class\n",
+ "\n",
+ "For our first recipe, we will want to use a pre-defined Recipe class from Pangeo\n",
+ "Forge.\n",
+ "\n",
+ "By examining the {doc}`recipes` documentation page, we see that our scenario is\n",
+ "a good case for the {class}`pangeo_forge.recipe.NetCDFtoZarrSequentialRecipe`\n",
+ "class. Let's examine its documentation string in our notebook.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "from pangeo_forge.recipe import NetCDFtoZarrSequentialRecipe\n",
+ "NetCDFtoZarrSequentialRecipe?"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Step 3: Define Recipe parameters\n",
+ "\n",
+ "Our chosen class has only two required parameters: `input_urls` and\n",
+ "`sequence_dim`.\n",
+ "\n",
+ "`input_urls` is a list of URLs pointing to the data. To populate this, we need\n",
+ "to explicitly create this list based on what we know about the file naming\n",
+ "conventions. Let's look again at the first URL\n",
+ "\n",
+ "```text\n",
+ "https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation/v2.1/access/avhrr/198109/oisst-avhrr-v02r01.19810901.nc\n",
+ "```\n",
+ "\n",
+ "From this we deduce the following format string.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "input_url_pattern = (\n",
+ " \"https://www.ncei.noaa.gov/data/sea-surface-temperature-optimum-interpolation\"\n",
+ " \"/v2.1/access/avhrr/{yyyymm}/oisst-avhrr-v02r01.{yyyymmdd}.nc\"\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "Now we need a sequence of datetimes. Pandas is the easiest way to get this. At\n",
+ "the time of writing, the latest available data is from 2021-01-05.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "import pandas as pd\n",
+ "\n",
+ "dates = pd.date_range(\"1981-09-01\", \"2021-01-05\", freq=\"D\")\n",
+ "input_urls = [\n",
+ " input_url_pattern.format(\n",
+ " yyyymm=day.strftime(\"%Y%m\"), yyyymmdd=day.strftime(\"%Y%m%d\")\n",
+ " )\n",
+ " for day in dates\n",
+ "]\n",
+ "print(f\"Found {len(input_urls)} files!\")\n",
+ "input_urls[-1]"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "That's a lot of files!\n",
+ "\n",
+ "The other remaining parameter is `sequence_dim`. It's just `\"time\"`. We can now\n",
+ "instantiate our recipe.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "recipe = NetCDFtoZarrSequentialRecipe(\n",
+ " input_urls=input_urls, sequence_dim=\"time\"\n",
+ ")\n",
+ "recipe"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.8.6"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}

MODIFY
pangeo_forge/recipe.py
pangeo_forge/recipe.py
@@ -117,8 +117,13 @@ class BaseRecipe(ABC):
@dataclass
class NetCDFtoZarrSequentialRecipe(BaseRecipe):
- """There are many inputs (a.k.a. files, granules), arranged in a sequence
- along the dimension `sequence_dim`. Each file may contain multiple variables.
+ """This class represents a dataset composed of many individual NetCDF files.
+ The files are arraged in a sequence along a single dimension, called the
+ `sequence_dim`. Each file may contain multiple variables.
+
+ The dataset is assembled by concatenating all of these files along `sequence_dim`.
+ The target is written in Zarr format.
+
This class uses Xarray to read and write data.
:param input_urls: The inputs used to generate the dataset.
@@ -133,7 +138,7 @@ class NetCDFtoZarrSequentialRecipe(BaseRecipe):
the inputs to form a chunk.
"""
- input_urls: Iterable[str]
+ input_urls: Iterable[str] = field(repr=False)
sequence_dim: str
inputs_per_chunk: int = 1
nitems_per_input: int = 1

add tutorial to docs