last commit of the night
ADD
docs/_static/custom.css
+/* Put your custom CSS here */
+@import url('http://fonts.cdnfonts.com/css/panton-black-caps');
+
+h1 {
+ font-family: "Panton Black Caps", sans-serif;
+ color: #003B71 !important;
+}
+
+h2 {
+ font-family: "Panton Light Caps", sans-serif;
+ color: #003B71 !important;
+}
+
+a {
+ color: #5eb130 !important;
+}
+
+
+
+/* Fixing up some pygments and code-styling CSS for accessibility */
+code { font-size: 100%; color: #e50051; }
+pre { font-family: monospace; }
+
+/* .highlight { font-size: 125%; } */
+.highlight .c1 { color: #e50051; }
+.highlight .si { color: #e50051; }
+.highlight .nn { color: #e50051; }

MODIFY
docs/api.md
docs/api.md
:members:
```
+```{eval-rst}
+.. autoclass:: pangeo_forge.storage.FlatFSSpecTarget
+ :members:
+ :show-inheritance:
+```
+
```{eval-rst}
.. autoclass:: pangeo_forge.storage.CacheFSSpecTarget
:members:
+ :show-inheritance:
```
## Recipes
.. autoclass:: pangeo_forge.recipe.NetCDFtoZarrSequentialRecipe
:show-inheritance:
```
+
+## Excutors
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.executors.PythonPipelineExecutor
+ :members:
+```
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.executors.DaskPipelineExecutor
+ :members:
+```
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.executors.PrefectPipelineExecutor
+ :members:
+```

MODIFY
docs/execution.md
docs/execution.md
# Recipe Execution
+
+There are many different types of Pangeo Forge recipes.
+However, **all recipes are executed the same way**!
+This is a key part of the Pangeo Forge design.
+
+Once you have created a recipe object (see {doc}`recipes`) you have two
+options for executing it. In the subsequent code, we will assume that a
+recipe has already been initialized in the variable `recipe`.
+
+## Manual Execution
+
+A recipe can be executed manually, step by step, in serial, from a notebook
+or interactive interpreter. The ability to manually step through a recipe
+is very important for developing and debugging complex recipes.
+There are four stages of recipe execution.
+
+### Stage 1: Cache Inputs
+
+Recipes may define files that have to be cached locally before the subsequent
+steps can proceed. The common use case here is for files that have to be
+extracted from a slow FTP server. Here is how to cache the inputs.
+
+```{code-block} python
+for input_name in recipe.iter_inputs():
+ recipe.cache_input(input_name)
+```
+
+If the recipe doesn't do input caching, nothing will happen here.
+
+### Stage 2: Prepare Target
+
+Once the inputs have been cached, we can get the target ready.
+Preparing the target for writing is done as follows:
+
+```{code-block} python
+recipe.prepare_target()
+```
+
+For example, for Zarr targets, this sets up the Zarr group with the necessary
+arrays and metadata.
+
+### Stage 3: Store Chunks
+
+This is the step where the bulk of the work happens.
+
+```{code-block} python
+for chunk in recipe.iter_chunks():
+ recipe.store_chunk(chunk)
+```
+
+### Stage 4: Finalize Target
+
+If there is any cleanup or consolidation to be done, it happens here.
+
+```{code-block} python
+recipe.finalize_target()
+```
+
+For example, consolidating Zarr metadta happens in the finalize step.
+
+## Execution by Executors
+
+Very large recipes cannot feasibly be executed this way.
+To support distributed parallel execution, Pangeo Forge borrows the
+[Executors framework from Rechunker](https://rechunker.readthedocs.io/en/latest/executors.html).
+
+There are currently three executors implemented.
+- {class}`pangeo_forge.executors.PythonPipelineExecutor`: a reference executor
+ using simple python
+- {class}`pangeo_forge.executors.DaskPipelineExecutor`: distributed executor using Dask
+- {class}`pangeo_forge.executors.PrefectPipelineExecutor`: distributed executor using Prefect
+
+To use an executor, the recipe must first be transformed into a `Pipeline` object.
+The full process looks like this:
+
+```{code-block} python
+pipeline = recipe.to_pipelines()
+executor = PrefectPipelineExecutor()
+plan = executor.pipelines_to_plan(pipeline)
+executor.execute_plan(plan) # actually runs the recipe
+```

MODIFY
docs/index.md
docs/index.md
@@ -14,7 +14,8 @@ The most important concept in Pangeo Forge is a ``recipe``.
A recipe defines how to transform data in one format / location into another format / location.
The primary way people contribute to Pangeo Forge is by writing / maintaining recipes.
Recipes developed by the community are stored in GitHub repositories.
-For information about how to write a recipe, see {doc}`recipes`.
+For information about how recipes work see {doc}`recipes`.
+The {doc}`tutorials/index` provide deep dives into how to develop and debug Pangeo Forge recipes.
## Recipe Execution
@@ -35,7 +36,7 @@ For more information, see {doc}`bakeries`.
:caption: Contents
recipes
-recipe_tutorial
+tutorials/index
execution
bakeries
contribute

MODIFY
docs/recipes.md
docs/recipes.md
@@ -32,20 +32,43 @@ target = FSSpecTarget(fs=fs, root_path=target_path)
Temporary data is recommended to use a {class}`pangeo_forge.storage.CacheFSSpecTarget` object.
-## The Base Recipe Class
+## The Recipe Object
+
+You define a recipe by instantiating a class that inherits from {class}`pangeo_forge.recipe.BaseRecipe`.
+The `pangeo_forge` package includes several pre-defined Recipe classes which
+cover common scenarios. You can also define your own Recipe class.
+
+For a the common scenario of assembling many NetCDF files into a single Zarr
+group, we use {class}`pangeo_forge.recipe.NetCDFtoZarrSequentialRecipe`.
+Initializing a recipe looks something like this.
-A recipe is initialized from a recipe class.
```{code-block} python
-recipe = Recipe(option1='foo', option2=)
+from pangeo_forge.recipes import NetCDFtoZarrSequentialRecipe
+input_urls = [...] # build a list of inputs
+recipe = NetCDFtoZarrSequentialRecipe(
+ input_urls=input_urls,
+ sequence_dim="time"
+)
```
-All recipes follow the same basic steps.
+There are many other options we can pass, all covered in the [API documentation](api).
+For a deeper dive on how to pick these options and what they mean, check out the
+tutorial: {doc}`tutorials/netcdf_zarr_sequential`.
+Your recipe will also need storage.
+If you have already defined a `Target` object (as in the the [Storage section](#storage)),
+then you can either assign it when you initialize the recipe or later, e.g.
-## Specific Recipe Classes
+```{code-block} python
+recipe.target = FSSpecTarget(fs=fs, root_path=target_path)
+```
-```{eval-rst}
-.. autoclass:: pangeo_forge.recipe.NetCDFtoZarrSequentialRecipe
- :show-inheritance:
- :noindex:
+This particular class of recipe also requires a cache, a place to store temporary
+files. We can create one as follows.
+
+```{code-block} python
+recipe.input_cache = CacheFSSpecTarget(fs=fs, root_path=cache_path)
```
+
+Once your recipe is defined and has its targets assigned, you're ready to
+move on to {doc}`execution`.

ADD
docs/tutorials/index.md
+# Recipe Tutorials
+
+These tutorials are deep dives into how to develop and debug Pangeo Forge recipes.
+
+```{toctree}
+:maxdepth: 1
+
+netcdf_zarr_sequential
+```

RENAME
docs/recipe_tutorial.ipynb
docs/tutorials/netcdf_zarr_sequential.ipynb
"cell_type": "markdown",
"metadata": {},
"source": [
- "# Recipe Tutorial\n",
+ "# NetCDF Zarr Sequential Recipe\n",
"\n",
- "This tutorial describes how to create a recipe from scratch.\n"
+ "This tutorial describes how to create a recipe from scratch.\n",
+ "The source data is a sequence of NetCDF files accessed via HTTP.\n",
+ "The target is a Zarr store."
]
},
{
"For our first recipe, we will want to use a pre-defined Recipe class from Pangeo\n",
"Forge.\n",
"\n",
- "By examining the {doc}`recipes` documentation page, we see that our scenario is\n",
+ "By examining the {doc}`../recipes` documentation page, we see that our scenario is\n",
"a good case for the {class}`pangeo_forge.recipe.NetCDFtoZarrSequentialRecipe`\n",
"class. Let's examine its documentation string in our notebook.\n"
]
"\n",
"We are now confident that our recipe works as we expect.\n",
"At this point we could either:\n",
- "- Execute it all ourselves (see {doc}`execution`)\n",
+ "- Execute it all ourselves (see {doc}`../execution`)\n",
"- Create a new recipe feedstock on Pangeo Forge\n",
"\n",
"If we wanted to execute it ourselves, one way would be to simply run the following code\n",
"\n",
"But hopefully now you have a better understanding of how Pangeo Forge recipes work."
]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
}
],
"metadata": {

ADD
pangeo_forge/executors.py
+from rechunker.executors import DaskPipelineExecutor # noqa: F401
+from rechunker.executors import PrefectPipelineExecutor # noqa: F401
+from rechunker.executors import PythonPipelineExecutor # noqa: F401

MODIFY
tests/test_executors.py
tests/test_executors.py
import pytest
import xarray as xr
-from rechunker.executors import (
+
+from pangeo_forge.executors import (
DaskPipelineExecutor,
PrefectPipelineExecutor,
PythonPipelineExecutor,

last commit of the night