codecov-commenter: # [Codecov](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=h1) Report
> Merging [#14](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=desc) into [master](https://codecov.io/gh/sdss/sdsstools/commit/c75fa6dd59ccd5c4bd58fd7c2a82765389d31eab&el=desc) will **increase** coverage by `1.01%`.
> The diff coverage is `100.00%`.

[![Impacted file tree graph](https://codecov.io/gh/sdss/sdsstools/pull/14/graphs/tree.svg?width=650&height=150&src=pr&token=RXDhA7SpXB)](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master      #14      +/-   ##
==========================================
+ Coverage   95.60%   96.61%   +1.01%     
==========================================
  Files           3        3              
  Lines         250      265      +15     
  Branches       48       53       +5     
==========================================
+ Hits          239      256      +17     
+ Misses          1        0       -1     
+ Partials       10        9       -1     
```


| [Impacted Files](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/sdsstools/configuration.py](https://codecov.io/gh/sdss/sdsstools/pull/14/diff?src=pr&el=tree#diff-c3JjL3Nkc3N0b29scy9jb25maWd1cmF0aW9uLnB5) | `100.00% <100.00%> (+2.36%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=footer). Last update [c75fa6d...3ff7487](https://codecov.io/gh/sdss/sdsstools/pull/14?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

diff --git a/CHANGELOG.md b/CHANGELOG.md
index 03b5541..63e6960 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -2,6 +2,7 @@
 
 ## [Next version](https://github.com/sdss/sdsstools/compare/0.2.1...HEAD)
 
+- Feature [#7](https://github.com/sdss/sdsstools/issues/7): add support for a tag `!extends` in YAML files when read with `read_yaml_file`.
 - Bug [#8](https://github.com/sdss/sdsstools/issues/8): log `StreamHandler` to `stderr` when the record level is `ERROR` or greater.
 - Support [#11](https://github.com/sdss/sdsstools/issues/11): replaced `colored_formatter` with a propper `Formatter` and added tests for logging.
 - Support [#12](https://github.com/sdss/sdsstools/issues/12): move `color_print` to `sdsstools._vendor`.
diff --git a/README.md b/README.md
index 1f7e05f..9647d13 100644
--- a/README.md
+++ b/README.md
@@ -70,6 +70,36 @@ In addition to the (recommended) location `~/.config/sdss/<NAME>.yaml`, `get_con
 
 `sdsstools.configuration` includes two other tools, `merge_config`, that allows to merge dictionaries recursively, and `read_yaml_file`, to read a YAML file.
 
+### Extending a YAML file
+
+`read_yaml_file` provides a non-standard feature that allows you to extend one YAML file with another. To achieve this you need to add the tag `!extends <base-file>` at the top of the file that you want to extend. For example, if you have a file `base.yaml`
+
+```yaml
+cat1:
+    key1: value2
+
+cat2:
+    key2: 1
+```
+
+that you want to use as a template for `extendable.yaml`
+
+```yaml
+#!extends base.yaml
+
+cat1:
+    key1: value1
+```
+
+you can use `read_yaml_file` to parse the result
+
+```python
+>>> read_yaml_file('extendable.yaml')
+{'cat1': {'key1': 'value2'}, 'cat2': {'key2': 1}}
+```
+
+The path to the base file must be absolute, or relative to the location of the file to be extended.
+
 ## Metadata
 
 sdsscore provides tools to locate and parse metadata files (`pyproject.toml`, `setup.cfg`, `setup.py`). `get_metadata_files` locates the path of the metadata file relative to a given `path`. `get_package_version` tries to find the version of the package by looking for a version string in the metadata file or in the egg/wheel metadata file, if the package has been installed. To use it
diff --git a/src/sdsstools/configuration.py b/src/sdsstools/configuration.py
index cd8fd40..3782a94 100644
--- a/src/sdsstools/configuration.py
+++ b/src/sdsstools/configuration.py
@@ -43,15 +43,34 @@ yaml.add_implicit_resolver('!env', env_matcher)
 yaml.add_constructor('!env', env_constructor)
 
 
-def read_yaml_file(path):
+def read_yaml_file(path, use_extends=True, loader=yaml.FullLoader):
     """Read a YAML file and returns a dictionary."""
 
     if isinstance(path, (str, pathlib.Path)):
-        with open(path, 'r') as fp:
-            config = yaml.load(fp, Loader=yaml.FullLoader)
+        fp = open(path, 'r')
     else:
-        # Assume it's an stream
-        config = yaml.load(path, Loader=yaml.FullLoader)
+        fp = path
+
+    fp.seek(0)
+    config = yaml.load(fp, Loader=loader)
+
+    if use_extends:
+        fp.seek(0)
+        while True:
+            line = fp.readline()
+            if line.strip().startswith('#!extends'):
+                base_file = line.strip().split()[1]
+                if not os.path.isabs(base_file) and hasattr(fp, 'buffer'):
+                    base_file = os.path.join(os.path.dirname(path), base_file)
+                if not os.path.exists(base_file):
+                    raise FileExistsError(f'cannot find !extends file {base_file}.')
+                return merge_config(read_yaml_file(base_file),
+                                    read_yaml_file(path, use_extends=False))
+            elif line.strip().startswith('#') or line.strip() == '':
+                continue
+            else:
+                fp.seek(0)
+                break
 
     return config
 

Add !extends constructor
Add a PyYAML constructor to base a YAML file on another. For example

```yaml
# file: a.yml
a: 1
b:
  c: 3
  d: 4
```

```yaml
# file: b.yaml
!extends a.yml
b:
  c: 42
```

would be parsed into `{a: 1, b: {c: 42, d: 4}}`.

This would be useful for example for APO/LCO configuration files in which on can be made the template and the other can just extend and modify it.
diff --git a/test/test_configuration.py b/test/test_configuration.py
index 31a0788..d84ab84 100644
--- a/test/test_configuration.py
+++ b/test/test_configuration.py
@@ -7,18 +7,38 @@
 # @License: BSD 3-clause (http://www.opensource.org/licenses/BSD-3-Clause)
 
 import inspect
+import io
 import os
 import unittest.mock
 
 import pytest
 
 from sdsstools import Configuration, get_config
-from sdsstools.configuration import DEFAULT_PATHS
+from sdsstools.configuration import DEFAULT_PATHS, read_yaml_file
 
 
 BASE_CONFIG_FILE = os.path.join(os.path.dirname(__file__), 'etc/test.yml')
 
 
+BASE = """
+cat1:
+    key1: base_value
+
+cat2:
+    key2: 1
+"""
+
+EXTENDABLE = """
+#
+
+#!extends {base_path}
+
+cat1:
+    # test
+    key1: value1
+"""
+
+
 @pytest.fixture(autouse=True)
 def cleanup():
 
@@ -64,6 +84,15 @@ def set_envvar():
     del os.environ['A_TEST_VARIABLE']
 
 
+@pytest.fixture
+def extendable(tmp_path):
+
+    base_path = tmp_path / 'base.yaml'
+    base_path.write_text(BASE)
+
+    yield io.StringIO(EXTENDABLE.format(base_path=str(base_path))), base_path
+
+
 def test_configuration(config_file):
 
     config = Configuration()
@@ -155,3 +184,47 @@ def test_get_config_bad_module(mock_func):
 
     config = get_config('test')
     assert config == {}
+
+
+def test_extends(extendable):
+
+    stream, __ = extendable
+    data = read_yaml_file(stream)
+
+    assert data['cat1']['key1'] == 'base_value'
+    assert 'cat2' in data
+
+
+def test_extends_file_not_found(extendable):
+
+    stream, base_path = extendable
+    base_path.unlink()
+
+    with pytest.raises(FileExistsError):
+        read_yaml_file(stream)
+
+
+def test_dont_extend(extendable):
+
+    stream, __ = extendable
+    data = read_yaml_file(stream, use_extends=False)
+
+    assert data['cat1']['key1'] == 'value1'
+    assert 'cat2' not in data
+
+
+def test_extends_from_file(tmp_path):
+
+    base_path = tmp_path / 'subdir' / 'base.yaml'
+    (tmp_path / 'subdir').mkdir()
+    base_path.touch()
+    base_path.write_text(BASE)
+
+    extendable_path = tmp_path / 'extendable.yaml'
+    extendable_relative = EXTENDABLE.format(base_path='subdir/base.yaml')
+    extendable_path.write_text(extendable_relative)
+
+    data = read_yaml_file(extendable_path)
+
+    assert data['cat1']['key1'] == 'base_value'
+    assert 'cat2' in data

pip install -e .[dev]
parse_log_pytest
requirements.txt
pytest pytest-cov
apt-get update apt-get install -y gcc
3.9
requirements/base.txt
pytest --no-header -rA --tb=line --color=no -p no:cacheprovider -W ignore::DeprecationWarning
backports.tarfile==1.2.0
certifi==2025.1.31
cffi==1.17.1
charset-normalizer==3.4.1
colorama==0.4.6
coverage==7.8.0
cryptography==44.0.2
docutils==0.16
exceptiongroup==1.2.2
idna==3.10
importlib_metadata==8.6.1
iniconfig==2.1.0
invoke==1.7.3
jaraco.classes==3.4.0
jaraco.context==6.0.1
jaraco.functools==4.1.0
jeepney==0.9.0
keyring==25.6.0
more-itertools==10.6.0
nh3==0.2.21
packaging==24.2
pkginfo==1.12.1.2
pluggy==1.5.0
pycparser==2.22
Pygments==2.19.1
pytest==8.3.5
pytest-cov==6.0.0
PyYAML==6.0.2
readme_renderer==43.0
requests==2.32.3
requests-toolbelt==1.0.0
rfc3986==2.0.0
-e git+https://github.com/sdss/sdsstools.git@c75fa6dd59ccd5c4bd58fd7c2a82765389d31eab#egg=sdsstools
SecretStorage==3.3.3
semantic-version==2.10.0
six==1.17.0
tomli==2.2.1
tqdm==4.67.1
twine==3.8.0
urllib3==2.3.0
zipp==3.21.0

name: sdsstools
channels:
  - defaults
  - https://repo.anaconda.com/pkgs/main
  - https://repo.anaconda.com/pkgs/r
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - ca-certificates=2025.2.25=h06a4308_0
  - ld_impl_linux-64=2.40=h12ee557_0
  - libffi=3.4.4=h6a678d5_1
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - ncurses=6.4=h6a678d5_0
  - openssl=3.0.16=h5eee18b_0
  - pip=25.0=py39h06a4308_0
  - python=3.9.21=he870216_1
  - readline=8.2=h5eee18b_0
  - setuptools=75.8.0=py39h06a4308_0
  - sqlite=3.45.3=h5eee18b_0
  - tk=8.6.14=h39e8969_0
  - tzdata=2025a=h04d1e81_0
  - xz=5.6.4=h5eee18b_1
  - zlib=1.2.13=h5eee18b_1
  - pip:
      - backports-tarfile==1.2.0
      - certifi==2025.1.31
      - cffi==1.17.1
      - charset-normalizer==3.4.1
      - colorama==0.4.6
      - coverage==7.8.0
      - cryptography==44.0.2
      - docutils==0.16
      - exceptiongroup==1.2.2
      - idna==3.10
      - importlib-metadata==8.6.1
      - iniconfig==2.1.0
      - invoke==1.7.3
      - jaraco-classes==3.4.0
      - jaraco-context==6.0.1
      - jaraco-functools==4.1.0
      - jeepney==0.9.0
      - keyring==25.6.0
      - more-itertools==10.6.0
      - nh3==0.2.21
      - packaging==24.2
      - pkginfo==1.12.1.2
      - pluggy==1.5.0
      - pycparser==2.22
      - pygments==2.19.1
      - pytest==8.3.5
      - pytest-cov==6.0.0
      - pyyaml==6.0.2
      - readme-renderer==43.0
      - requests==2.32.3
      - requests-toolbelt==1.0.0
      - rfc3986==2.0.0
      - sdsstools==0.2.2a0
      - secretstorage==3.3.3
      - semantic-version==2.10.0
      - six==1.17.0
      - tomli==2.2.1
      - tqdm==4.67.1
      - twine==3.8.0
      - urllib3==2.3.0
      - wheel==0.33.6
      - zipp==3.21.0
prefix: /opt/conda/envs/sdsstools

test/test_configuration.py::test_extends test/test_configuration.py::test_extends_file_not_found test/test_configuration.py::test_dont_extend test/test_configuration.py::test_extends_from_file
test/test_configuration.py::test_configuration test/test_configuration.py::test_configuration_user test/test_configuration.py::test_configuration_envvar test/test_configuration.py::test_configuration_envvar_defaults test/test_configuration.py::test_configurations_bad_value test/test_configuration.py::test_get_config_etc test/test_configuration.py::test_get_config_etc_with_user test/test_configuration.py::test_get_config_etc_with_user_str test/test_configuration.py::test_get_config_default_path test/test_configuration.py::test_get_config_envvar_path test/test_configuration.py::test_get_config_no_update test/test_configuration.py::test_get_config_bad_module
src/sdsstools/configuration.py README.md CHANGELOG.md
src/sdsstools/configuration.py README.md CHANGELOG.md