start working on docs
ADD
docs/bakeries.md
+# Bakeries

DELETE
docs/concepts.md
-# Concepts
-
-pangeo-forge is modeled after [conda-forge], a community-led collection of recipes
-for building conda packages.
-
-## Recipes
-
-The most important concept in Pangeo Forge is a ``Recipe``.
-A recipe defines how to transform data in one format / location into another format / location.
-
-
-## Storage
-
-
-
-
-## Components
-
-The high-level components of pangeo-forge are:
-
-* https://github.com/pangeo-forge/staged-recipes: The birthplace for new recipes.
- Anyone interested in adding a dataset to pangeo-forge can submit a new recipe
- through a pull request, using the examples there as a starting point.
-* Recipes: Metadata describing a dataset and code for transforming it from
- raw input to analysis-ready dataset.
-* pangeo-forge: A Python library containing useful pipeline helpers and the
- [pangeo-forge command-line-interface][cli] for validating pipelines.
-* https://github.com/pangeo-forge/docker-images: A repository for building docker
- images for pangeo-forge.
-
-## Pipeline Structure
-
-pangeo-forge uses [Prefect] for
-
-1. Pipeline definitions: the code that expresses the transformation from raw to analysis-ready data.
-2. Orchestration: everything involved with taking a pipeline definition and actually running it.
-
-This imposes a few constraints which we build upon.
-
-All recipe definitions must be named `recipe/pipeline.py` and must contain an instances
-of a `prefect.Flow` at the top-level of the module.
-
-Additionally, pangeo-forge imposes additional strucutre. All pipelines modules should include a `Pipeline`
-class that inherits from `pangeo_forge.AbstractPipeline`. So most pipeline modules will have
-
-
-```
-class Pipeline(pangeo_forge.AbstractPipeline):
- @property
- def flow(self):
- with prefect.Flow(self.name) as flow:
- ...
- return flow
-
-
-pipeline = Pipeline()
-flow = pipeline.flow
-```
-
-## Lifecycle of a recipe
-
-A maintainer contributes a recipe to [staged-recipes] through a pull request. We
-use GitHub Actions to perform some initial validation. The actions are at https://github.com/pangeo-forge/staged-recipes/blob/master/.github/workflows/main.yaml and the logs are available at https://github.com/pangeo-forge/staged-recipes/actions.
-
-When a PR is merged into `staged-recipes` the [Create Repository](https://github.com/pangeo-forge/staged-recipes/blob/master/.github/workflows/create-repository.yaml) action is run, which
-
-1. Creates a new git repository with the contents of `staged-recipes/examples/<new-pipeline>`
- inserted, along with a few other files provided by pangeo-forge, including the `register_pipeline` action definition.
-2. Pushes that new repository to `https://pangeo-forge/<new-pipeline>`, for example https://github.com/pangeo-forge/example-pipeline/.
-3. Executes the [Register Pipeline](https://github.com/pangeo-forge/staged-recipes/blob/master/.github/workflows/scripts/register_pipeline.yaml) action, which registers the new pipeline with Prefect and executes it.
-
-[conda-forge]: https://conda-forge.github.io
-[cli]: https://github.com/pangeo-forge/pangeo-forge/blob/master/pangeo_forge/cli.py
-[Prefect]: https://docs.prefect.io
-[staged-recipes]: https://github.com/pangeo-forge/staged-recipes/

MODIFY
docs/conf.py
docs/conf.py
@@ -11,6 +11,9 @@ author = "Pangeo Community"
extensions = [
"myst_parser",
+ "sphinx.ext.autodoc",
+ # "numpydoc",
+ "sphinx_autodoc_typehints",
]
templates_path = ["_templates"]
@@ -21,5 +24,6 @@ master_doc = "index"
html_theme = "pangeo"
html_static_path = ["_static"]
+html_sidebars = {"index": [], "**": ["localtoc.html"]}
myst_heading_anchors = 2

ADD
docs/execution.md
+# Recipe Execution

MODIFY
docs/index.md
docs/index.md
-# pangeo-forge
+# Pangeo Forge
+
+Pangeo Forge is an open source tool for data Extraction, Transformation, and Loading (ETL).
+The goal of Pangeo Forge is to make it easy to extract data from traditional data
+repositories and deposit in cloud object storage in analysis-ready, cloud-optimize (ARCO) format.
+
+Pangeo Forge is inspired by [Conda Forge](https://conda-forge.org/), a
+community-led collection of recipes for building conda packages.
+We hope that Pangeo Forge can play the same role for datasets.
+
+## Recipes
+
+The most important concept in Pangeo Forge is a ``recipe``.
+A recipe defines how to transform data in one format / location into another format / location.
+The primary way people contribute to Pangeo Forge is by writing / maintaining recipes.
+Recipes developed by the community are stored in GitHub repositories.
+For information about how to write a recipe, see {doc}`recipes`.
+
+## Recipe Execution
+
+There are several different ways to execute recipes.
+See {doc}`execution` for details.
+
+## Bakeries
+
+Bakeries are cloud-based environments for executing recipes.
+Each Bakery is coupled to one or more cloud storage buckets where the ARCO data is stored.
+Bakeries use [Prefect](https://prefect.io/) to orchestrate the various steps
+of the recipe.
+For more information, see {doc}`bakeries`.
+
```{toctree}
:maxdepth: 2
:caption: Contents
-concepts
-contribte
+recipes
+execution
+bakeries
+contribute
```

ADD
docs/recipes.md
+# Recipes
+
+## The Base Recipe Class
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.recipe.BaseRecipe
+ :members:
+```
+
+## Specific Recipe Classes
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.recipe.NetCDFtoZarrSequentialRecipe
+ :show-inheritance:
+```

MODIFY
pangeo_forge/recipe.py
pangeo_forge/recipe.py
@@ -51,40 +51,53 @@ def input_opener(fname, **kwargs):
class BaseRecipe(ABC):
+ """Base recipe class from which all other Recipes inherit.
+ """
+
@property
@abstractmethod
def prepare(self) -> Callable[[], NoReturn]:
+ """Prepare the recipe for execution by initializing the target.
+ Attribute that returns a callable function.
+ """
pass
@abstractmethod
def iter_inputs(self) -> Iterable[Hashable]:
+ """Iterate over all inputs."""
pass
@property
@abstractmethod
def cache_input(self) -> Callable[[Hashable], NoReturn]:
+ """Copy an input from its source location to the cache.
+ Attribute that returns a callable function.
+ """
pass
@abstractmethod
def iter_chunks(self) -> Iterable[Hashable]:
+ """Iterate over all target chunks."""
pass
@property
@abstractmethod
def store_chunk(self) -> Callable[[Hashable], NoReturn]:
+ """Store a chunk of data in the target.
+ Attribute that returns a callable function.
+ """
pass
@property
@abstractmethod
def finalize(self) -> Callable[[], NoReturn]:
+ """Final step to finish the recipe after data has been written.
+ Attribute that returns a callable function.
+ """
pass
def to_pipelines(self) -> ParallelPipelines:
- """Translate recipe to pipelines
-
- Returns
- -------
- pipeline : ParallelPipelines
+ """Translate recipe to pipeline for execution.
"""
pipeline = [] # type: MultiStagePipeline
@@ -106,34 +119,28 @@ class BaseRecipe(ABC):
class NetCDFtoZarrSequentialRecipe(BaseRecipe):
"""There are many inputs (a.k.a. files, granules), arranged in a sequence
along the dimension `sequence_dim`. Each file may contain multiple variables.
+
+ :param input_urls: The inputs used to generate the dataset.
+ :param sequence_dim: The dimension name along which the inputs will be concatenated.
+ :param inputs_per_chunk: The number of inputs to use in each chunk.
+ :param nitems_per_input: The length of each input along the `sequence_dim` dimension.
+ :param target: A location in which to put the dataset. Can also be assigned at run time.
+ :param input_cache: A location in which to cache temporary data.
+ :param consolidate_zarr: Whether to consolidate the resulting Zarr dataset.
+ :param xarray_open_kwargs: Extra options for opening the inputs with Xarray.
+ :param xarray_concat_kwargs: Extra options to pass to Xarray when concatenating
+ the inputs to form a chunk.
"""
input_urls: Iterable[str]
- """The inputs used to generate the dataset."""
-
sequence_dim: str
- """The dimension name along which the inputs will be concatenated."""
-
inputs_per_chunk: int = 1
- """The number of inputs to use in each chunk."""
-
nitems_per_input: int = 1
- """The length of each input along the `sequence_dim` dimension."""
-
target: Optional[Target] = None
- """A location in which to put the dataset."""
-
input_cache: Optional[InputCache] = None
- """The length of each input along the `sequence_dim` dimension."""
-
consolidate_zarr: bool = True
- """Whether to consolidate the resulting Zarr dataset."""
-
xarray_open_kwargs: dict = field(default_factory=dict)
- """Extra options for opening the inputs with Xarray."""
-
xarray_concat_kwargs: dict = field(default_factory=dict)
- """Extra options to pass to Xarray when concatenating the inputs to form a chunk."""
def __post_init__(self):
self._chunks_inputs = {

start working on docs