big doc update
RENAME
docs/bakeries.md
docs/cloud_automation_user_guide/bakeries.md
# Bakeries
-As described in {doc}`execution`, you can define and execute recipes on your
+As described in {doc}`../recipe_user_guide/execution`, you can define and execute recipes on your
own computers using the executor of your choice.
However, you can also contribute your recipe to the Pangeo Forge {doc}`recipe_box`.
and have it executed automatically in the cloud.

RENAME
docs/recipe_box.md
docs/cloud_automation_user_guide/recipe_box.md
# The Recipe Box
-You are free to use Pangeo Forge Recipes in private, {doc}`executing <execution>`
+You are free to use Pangeo Forge Recipes in private, {doc}`executing <../recipe_user_guide/execution>`
them on your own computers / infrastructure however you wish.
However, the real point of Pangeo Forge is to create a crowdsourced database
of public recipes, which are executed automatically in the cloud via {doc}`bakeries`.
@@ -13,8 +13,8 @@ The process of contributing a recipe will likely evolve and change over time.
## Contributing a Recipe
-To contribute a recipe, you will need to understand the basics of Pangeo Forge {doc}`recipes`.
-Working through the {doc}`tutorials/index` is the best way to develop your understanding.
+To contribute a recipe, you will need to understand the basics of Pangeo Forge {doc}`../recipe_user_guide/recipes`.
+Working through the {doc}`../tutorials/index` is the best way to develop your understanding.
You will also need a thorough understanding of the source dataset you wish to ingest,
and a vision for the desired target dataset (desired format, chunk structure, etc.)
and any cleaning / processing steps needed along the way.

MODIFY
docs/development/index.md
docs/development/index.md
# Development
+TODO developers guide.
+
```{toctree}
:maxdepth: 1

MODIFY
docs/recipe_user_guide/index.md
docs/recipe_user_guide/index.md
-# Pangeo Forge
+# Recipe User Guide
-Pangeo Forge is an open source tool for data Extraction, Transformation, and Loading (ETL).
-The goal of Pangeo Forge is to make it easy to extract data from traditional data
-repositories and deposit in cloud object storage in analysis-ready, cloud-optimized (ARCO) format.
-
-Pangeo Forge is inspired by [Conda Forge](https://conda-forge.org/), a
-community-led collection of recipes for building conda packages.
-We hope that Pangeo Forge can play the same role for datasets.
+Welcome to the User Guide for the `pangeo_forge_recipes` Python package.
+This package is an important part of the broader Pangeo Forge infrastructure.
+The python package can be used on its own or as part of an automated recipe.
## File Patterns
@@ -15,42 +11,27 @@ they are the raw "ingredients" upon which the recipe will act.
The point of file patterns is to describe how many individual source files are
organized logically as part of a larger dataset.
-## Recipes
+## Recipe Object
-The most important concept in Pangeo Forge is a {doc}``Recipe <recipes>``.
-A recipe defines how to transform data in one format / location into another format / location.
-The primary way people contribute to Pangeo Forge is by writing / maintaining recipes.
+The central object in `pangeo_forge_recipes` is a {doc}``Recipe Object <recipes>``.
+A Recipe Object defines how to transform data in one format / location into another format / location.
+The primary way people contribute to Pangeo Forge is by writing / maintaining Recipes.
For information about how recipes work see {doc}`recipes`.
-The {doc}`tutorials/index` provide deep dives into how to develop and debug Pangeo Forge Recipes.
+The {doc}`../tutorials/index` provide deep dives into how to develop and debug Recipes.
## Recipe Execution
There are several different ways to execute Recipes.
See {doc}`execution` for details.
-## Recipe Box
-
-The {doc}`recipe_box` is a public database of Pangeo Forge Recipes.
-Recipes are stored in Github repos.
-
-## Bakeries
-
-Bakeries are cloud-based environments for executing recipes from the Recipe Box.
-Each Bakery is coupled to one or more cloud storage buckets where the ARCO data is stored.
-Bakeries use [Prefect](https://prefect.io/) to orchestrate the various steps
-of the recipe. For more information, see {doc}`bakeries`.
-
-## Site Contents
+## Index
```{toctree}
-:maxdepth: 2
+:maxdepth: 1
+:glob:
file_patterns
storage
recipes
execution
-tutorials/index
-recipe_box
-bakeries
-development/index
```

MODIFY
docs/recipe_user_guide/recipes.md
docs/recipe_user_guide/recipes.md
@@ -19,8 +19,8 @@ recipe is just the _description_ of the transformation.
To actually do the work, the recipe must be {doc}`executed <execution>`.
Recipe authors (i.e. data users or data managers) can either execute their recipes
on their own computers and infrastructure, in private, or contribute their recipe
-to the public Pangeo Forge {doc}`recipe_box`, where it can be executed in the cloud via
-{doc}`bakeries`.
+to the public Pangeo Forge {doc}`../cloud_automation_user_guide/recipe_box`, where it can be executed in the cloud via
+{doc}`../cloud_automation_user_guide/bakeries`.
## Recipe Classes
@@ -44,9 +44,9 @@ The target Zarr dataset will conform to the
The best way to really understand how recipes work is to go through the relevant
tutorials for this recipe class. These are, in order of increasing complexity
-- {doc}`tutorials/netcdf_zarr_sequential`
-- {doc}`tutorials/multi_variable_recipe`
-- {doc}`tutorials/terraclimate`
+- {doc}`../tutorials/netcdf_zarr_sequential`
+- {doc}`../tutorials/multi_variable_recipe`
+- {doc}`../tutorials/terraclimate`
Below we give a very basic overview of how recipes are used.
@@ -80,7 +80,7 @@ Once your recipe is defined and has its targets assigned, you're ready to
move on to {doc}`execution`.
The API documentation below explains all of the possible options for `XarrayZarrRecipe`.
-Many of these options are explored further in the {doc}`tutorials/index`.
+Many of these options are explored further in the {doc}`../tutorials/index`.
```{eval-rst}
.. autoclass:: pangeo_forge_recipes.recipes.XarrayZarrRecipe

MODIFY
docs/tutorials/netcdf_zarr_sequential.ipynb
docs/tutorials/netcdf_zarr_sequential.ipynb
"source": [
"## Step 2: Define File Pattern\n",
"\n",
- "The first step in developing a recipe is to define a {doc}`File Pattern <../file_patterns>`.\n",
+ "The first step in developing a recipe is to define a {doc}`File Pattern <../recipe_user_guide/file_patterns>`.\n",
"The file pattern describes how the source files (a.k.a. \"inputs\") are organized.\n",
"\n",
"In this case, we have a very simple sequence of files that we want to concatenate along a single dimension (time), so we can use the helper function {func}`pangeo_forge_recipes.patterns.pattern_from_file_sequence`. This allows us to simply pass a list of URLs, which we define explicitly."
"\n",
"We are now confident that our recipe works as we expect.\n",
"At this point we could either:\n",
- "- Execute it all ourselves (see {doc}`../execution`)\n",
+ "- Execute it all ourselves (see {doc}`../recipe_user_guide/execution`)\n",
"- Create a new recipe feedstock on Pangeo Forge\n",
"\n",
"If we wanted to execute it ourselves, one way would be to simply run the following code\n",
"\n",
"But hopefully now you have a better understanding of how Pangeo Forge recipes work."
]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
}
],
"metadata": {

big doc update