reorganize docs
DELETE
docs/api.md
-# API Reference
-
-
-## Storage
-
-```{eval-rst}
-.. autoclass:: pangeo_forge.storage.FSSpecTarget
- :members:
-```
-
-```{eval-rst}
-.. autoclass:: pangeo_forge.storage.FlatFSSpecTarget
- :members:
- :show-inheritance:
-```
-
-```{eval-rst}
-.. autoclass:: pangeo_forge.storage.CacheFSSpecTarget
- :members:
- :show-inheritance:
-```
-
-## Recipes
-
-```{eval-rst}
-.. autoclass:: pangeo_forge.recipe.BaseRecipe
- :members:
-```
-
-```{eval-rst}
-.. autoclass:: pangeo_forge.recipe.NetCDFtoZarrRecipe
- :members:
-```
-
-```{eval-rst}
-.. autoclass:: pangeo_forge.recipe.NetCDFtoZarrSequentialRecipe
- :show-inheritance:
-```
-
-```{eval-rst}
-.. autoclass:: pangeo_forge.recipe.NetCDFtoZarrMultiVarSequentialRecipe
- :show-inheritance:
-```
-
-## Patterns
-
-```{eval-rst}
-.. autoclass:: pangeo_forge.patterns.VariableSequencePattern
- :members:
-```
-
-## Executors
-
-```{eval-rst}
-.. autoclass:: pangeo_forge.executors.PythonPipelineExecutor
- :members:
-```
-
-```{eval-rst}
-.. autoclass:: pangeo_forge.executors.DaskPipelineExecutor
- :members:
-```
-
-```{eval-rst}
-.. autoclass:: pangeo_forge.executors.PrefectPipelineExecutor
- :members:
-```

MODIFY
docs/conf.py
docs/conf.py
@@ -23,7 +23,8 @@ exclude_patterns = ["_build", "**.ipynb_checkpoints"]
master_doc = "index"
# we always have to manually run the notebooks because they are slow / expensive
-jupyter_execute_notebooks = "off"
+jupyter_execute_notebooks = "auto"
+execution_excludepatterns = ["tutorials/*"]
# -- Options for HTML output -------------------------------------------------

MODIFY
docs/execution.md
docs/execution.md
@@ -80,3 +80,20 @@ executor = PrefectPipelineExecutor()
plan = executor.pipelines_to_plan(pipeline)
executor.execute_plan(plan) # actually runs the recipe
```
+
+## Executors
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.executors.PythonPipelineExecutor
+ :members:
+```
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.executors.DaskPipelineExecutor
+ :members:
+```
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.executors.PrefectPipelineExecutor
+ :members:
+```

ADD
docs/file_patterns.md
+---
+jupytext:
+ text_representation:
+ format_name: myst
+kernelspec:
+ display_name: Python 3
+ name: python3
+---
+
+# File Patterns
+
+File patterns are the starting point for any Pangeo Forge recipe.
+The point of file patterns is to describe how many individual source files are
+organized logically as part of a larger dataset.
+
+
+```{code-cell} ipython3
+
+```
+
+
+## File Patterns API
+
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.patterns.ConcatDim
+ :members:
+```
+
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.patterns.MergeDim
+ :members:
+```
+
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.patterns.FilePattern
+ :members:
+```

MODIFY
docs/index.md
docs/index.md
@@ -35,11 +35,12 @@ For more information, see {doc}`bakeries`.
:maxdepth: 2
:caption: Contents
+file_patterns
+storage
recipes
-tutorials/index
execution
+tutorials/index
bakeries
contribute
-api
```

MODIFY
docs/recipes.md
docs/recipes.md
@@ -8,57 +8,25 @@ The Recipe API is still in flux and may change. Make sure the version of the doc
you are reading matches your installed version of pangeo_forge.
```
-## Storage
-
-Recipes need a place to store data.
-The location where the final dataset produced by the recipe is stored is called the
-``Target``. Pangeo forge has a special class for this: {class}`pangeo_forge.storage.FSSpecTarget`
-
-Creating a Target requires two arguments:
-- The ``fs`` argument is an [fsspec](https://filesystem-spec.readthedocs.io/en/latest/)
- filesystem. Fsspec supports many different types of storage via its
- [built in](https://filesystem-spec.readthedocs.io/en/latest/api.html#built-in-implementations)
- and [third party](https://filesystem-spec.readthedocs.io/en/latest/api.html#other-known-implementations)
- implementations.
-- The `root_path` argument specifies the specific path where the data should be stored.
-
-For example, creating a storage target for AWS S3 might look like this:
-```{code-block} python
-import s3fs
-fs = s3fs.S3FileSystem(key="MY_AWS_KEY", secret="MY_AWS_SECRET")
-target_path = "pangeo-forge-bucket/my-dataset-v1.zarr"
-target = FSSpecTarget(fs=fs, root_path=target_path)
-```
-
-Temporary data is can be cached via a {class}`pangeo_forge.storage.CacheFSSpecTarget` object.
-Some recipes require separate caching of metadata, which is provided by a third {class}`pangeo_forge.storage.FSSpecTarget`.
-
-
## The Recipe Object
-You define a recipe by instantiating a class that inherits from {class}`pangeo_forge.recipe.BaseRecipe`.
-The `pangeo_forge` package includes several pre-defined Recipe classes which
-cover common scenarios. You can also define your own Recipe class.
-
+The job of a Recipe is to take a file pattern and turn into a single analysis-ready, cloud-optimized Dataset.
For the common scenario of assembling many NetCDF files into a single Zarr
-group, we use {class}`pangeo_forge.recipe.NetCDFtoZarrSequentialRecipe`.
+group, we use {class}`pangeo_forge.recipes.XarrayZarrRecipe`.
+
Initializing a recipe looks something like this.
```{code-block} python
-from pangeo_forge.recipes import NetCDFtoZarrSequentialRecipe
-input_urls = [...] # build a list of inputs
-recipe = NetCDFtoZarrSequentialRecipe(
- input_urls=input_urls,
- sequence_dim="time"
-)
+file_pattern = pattern_from_file_sequence(...)
+recipe = XarrayZarrRecipe(file_pattern)
```
-There are many other options we can pass, all covered in the [API documentation](api).
+There are many other options we can pass, all covered in the API documentation for {class}`pangeo_forge.recipes.XarrayZarrRecipe`.
For a deeper dive on how to pick these options and what they mean, check out the
tutorial: {doc}`tutorials/netcdf_zarr_sequential`.
-Your recipe will also need storage.
-If you have already defined a `Target` object (as in the the [Storage section](#storage)),
+Your recipe will also need {doc}`storage <storage>`.
+If you have already defined a `Target` object,
then you can either assign it when you initialize the recipe or later, e.g.
```{code-block} python
@@ -74,3 +42,9 @@ recipe.input_cache = CacheFSSpecTarget(fs=fs, root_path=cache_path)
Once your recipe is defined and has its targets assigned, you're ready to
move on to {doc}`execution`.
+
+## Recipe API
+
+```{eval-rst}
+.. autoclass:: pangeo_forge.recipes.XarrayZarrRecipe
+```

reorganize docs