xrotwang: @tresoldi I added tests, so backwards compat should be provided for. Merging now.
codecov-io: # [Codecov](https://codecov.io/gh/lexibank/pylexibank/pull/152?src=pr&el=h1) Report
> Merging [#152](https://codecov.io/gh/lexibank/pylexibank/pull/152?src=pr&el=desc) into [master](https://codecov.io/gh/lexibank/pylexibank/commit/7a360022dbae0311095feb903bb6924e1b659c3a?src=pr&el=desc) will **increase** coverage by `0.02%`.
> The diff coverage is `100%`.

[![Impacted file tree graph](https://codecov.io/gh/lexibank/pylexibank/pull/152/graphs/tree.svg?width=650&token=v0oYsvytpU&height=150&src=pr)](https://codecov.io/gh/lexibank/pylexibank/pull/152?src=pr&el=tree)

```diff
@@            Coverage Diff             @@
##           master     #152      +/-   ##
==========================================
+ Coverage   97.34%   97.36%   +0.02%     
==========================================
  Files          26       26              
  Lines        1467     1481      +14     
==========================================
+ Hits         1428     1442      +14     
  Misses         39       39
```


| [Impacted Files](https://codecov.io/gh/lexibank/pylexibank/pull/152?src=pr&el=tree) | Coverage Δ | |
|---|---|---|
| [src/pylexibank/dataset.py](https://codecov.io/gh/lexibank/pylexibank/pull/152/diff?src=pr&el=tree#diff-c3JjL3B5bGV4aWJhbmsvZGF0YXNldC5weQ==) | `98.13% <100%> (+0.17%)` | :arrow_up: |

------

[Continue to review full report at Codecov](https://codecov.io/gh/lexibank/pylexibank/pull/152?src=pr&el=continue).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/lexibank/pylexibank/pull/152?src=pr&el=footer). Last update [7a36002...070af60](https://codecov.io/gh/lexibank/pylexibank/pull/152?src=pr&el=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).

diff --git a/src/pylexibank/dataset.py b/src/pylexibank/dataset.py
index c2c596b..078b376 100644
--- a/src/pylexibank/dataset.py
+++ b/src/pylexibank/dataset.py
@@ -139,19 +139,37 @@ class Dataset(BaseDataset):
         - `kw` may be used to pass any context info to the tokenizer, when called
           explicitly.
         """
-        profile = self.dir / 'etc' / 'orthography.tsv'
-        if profile.exists():
-            profile = Profile.from_file(str(profile), form='NFC')
-            default_spec = list(next(iter(profile.graphemes.values())).keys())
+        def get_tokenizer(profile_):
+            profile_ = Profile.from_file(str(profile_), form='NFC')
+            default_spec = list(next(iter(profile_.graphemes.values())).keys())
             for grapheme in ['^', '$']:
-                if grapheme not in profile.graphemes:
-                    profile.graphemes[grapheme] = {k: None for k in default_spec}
-            profile.tree = Tree(list(profile.graphemes.keys()))
-            tokenizer = Tokenizer(profile=profile, errors_replace=lambda c: '<{0}>'.format(c))
+                if grapheme not in profile_.graphemes:
+                    profile_.graphemes[grapheme] = {k: None for k in default_spec}
+            profile_.tree = Tree(list(profile_.graphemes.keys()))
+            return Tokenizer(profile=profile_, errors_replace=lambda c: '<{0}>'.format(c))
+
+        tokenizers = {}
+        profile = self.etc_dir / 'orthography.tsv'
+        profile_dir = self.etc_dir / 'orthography'
+        if profile.exists():
+            tokenizers[None] = get_tokenizer(profile)
+        if profile_dir.exists() and profile_dir.is_dir():
+            for profile in profile_dir.glob('*.tsv'):
+                tokenizers[profile.stem] = get_tokenizer(profile)
 
+        if tokenizers:
             def _tokenizer(item, string, **kw):
                 kw.setdefault("column", "IPA")
                 kw.setdefault("separator", " + ")
+                profile = kw.pop('profile', None)
+                if profile:
+                    tokenizer = tokenizers[profile]
+                elif isinstance(item, dict) \
+                        and 'Language_ID' in item \
+                        and item['Language_ID'] in tokenizers:
+                    tokenizer = tokenizers[item['Language_ID']]
+                else:
+                    tokenizer = tokenizers[None]
                 return tokenizer(unicodedata.normalize('NFC', '^' + string + '$'), **kw).split()
             return _tokenizer
 

Support multiple profiles
Do we have a standard way of supporting multiple orthographic profiles? Or should we try to always use a single file, even when the code and the profile become more complex?

My question concerns mostly WOLD, with 41 very different profiles (one per language). I couldn't find a non-hacky way (involving lambdas and the like) to tell pylexibank "use the language-id of each row as a tokenizer-id".

The `_tokenizer` in Dataset does accept a column name via `**kwargs` (https://github.com/lexibank/pylexibank/blob/021f37481d37b472715033033773543273ed301f/src/pylexibank/dataset.py#L152), but there are conflicts when a grapheme is missing from one or more profiles (e.g., in some language "ch" is just "c"+"h", and even redundantly listing all missing graphemes in all languages is not enough, as it can lead to inconsistencies, like consuming characters in the stream that are not supposed to be consumed).

Besides, it doesn't seem that LexibankWriter allows me to pass such information (cf. https://github.com/lexibank/pylexibank/blob/021f37481d37b472715033033773543273ed301f/src/pylexibank/cldf.py#L93), as it only takes `item` (the **kw passed to `add_form`) and `string` (the value to be segmented). 

Currently, while refining WOLD, I'm manually loading all profiles in a `tokenizer` dictionary, inside `cmd_makecldf()`, and calling it directly:

```python
for row in progressbar(lexemes_rows):
    args.writer.add_form_with_segments(
        Language_ID=row["Language_ID"],
        Parameter_ID=row["Parameter_ID"],
        Form=row["Form"],
        Value=row["Form"],
        Segments=tokenizer[row["Language_ID"]]({}, row["Form"]),
        (...)
```

Note that this is **not** urgent: CLICS3 won't depend on this (we don't need segmented data, and I'll get back to it after we resubmit).
diff --git a/tests/test_dataset.py b/tests/test_dataset.py
index 4d9f66b..a656b42 100644
--- a/tests/test_dataset.py
+++ b/tests/test_dataset.py
@@ -1,12 +1,11 @@
 import sys
 import json
-import argparse
 import importlib
+import pathlib
 
 import pytest
 from clldutils.path import sys_path
 from cldfbench.dataset import NOOP
-from pyclts import TranscriptionSystem
 
 from pylexibank import Lexeme, Dataset
 from pylexibank.dataset import Unmapped
@@ -44,6 +43,26 @@ def test_invalid_dataset():
         Test()
 
 
+def test_Dataset_tokenizer(tmpdir):
+    etc = pathlib.Path(str(tmpdir)).joinpath('etc')
+    etc.mkdir()
+    orth_dir = etc.joinpath('orthography')
+    orth_dir.mkdir()
+    orth_dir.joinpath('l1.tsv').write_text('Grapheme\tIPA\na\tb')
+
+    class DS(Dataset):
+        id = '1'
+        dir = etc.parent
+
+    ds = DS()
+    assert ds.tokenizer({}, 'a', profile='l1') == ['b']
+    assert ds.tokenizer({'Language_ID': 'l1'}, 'a') == ['b']
+
+    etc.joinpath('orthography.tsv').write_text('Grapheme\tIPA\na\tc')
+    ds = DS()
+    assert ds.tokenizer({}, 'a') == ['c']
+
+
 def test_BaseDataset(mocker, repos):
     class TestDataset(Dataset):
         dir = repos / 'datasets' / 'test_dataset'

pip install -e .[dev]
parse_log_pytest
requirements.txt
pytest pytest-cov
apt-get update apt-get install -y gcc
3.9
requirements.txt
pytest --no-header -rA --tb=line --color=no -p no:cacheprovider -W ignore::DeprecationWarning
appdirs==1.4.4
attrs==25.3.0
babel==2.17.0
backports.tarfile==1.2.0
beautifulsoup4==4.13.3
bibtexparser==2.0.0b8
bs4==0.0.2
cdstarcat==1.5.0
certifi==2025.1.31
cffi==1.17.1
charset-normalizer==3.4.1
cldfbench==1.14.1
cldfcatalog==1.6.0
cldfzenodo==2.1.2
clldutils==3.24.2
colorama==0.4.6
colorlog==6.9.0
commonnexus==1.9.2
coverage==7.8.0
cryptography==44.0.2
csvw==3.5.1
docopt==0.6.2
docutils==0.21.2
et_xmlfile==2.0.0
exceptiongroup==1.2.2
flake8==7.2.0
gitdb==4.0.12
GitPython==3.1.44
greenlet==3.1.1
id==1.5.0
idna==3.10
importlib_metadata==8.6.1
iniconfig==2.1.0
isodate==0.7.2
jaraco.classes==3.4.0
jaraco.context==6.0.1
jaraco.functools==4.1.0
jeepney==0.9.0
jmespath==1.0.1
jsonschema==4.23.0
jsonschema-specifications==2024.10.1
keyring==25.6.0
language-tags==1.2.0
latexcodec==3.0.0
linglit==1.7.1
lingpy==2.6.13
lxml==5.3.1
Markdown==3.7
markdown-it-py==3.0.0
MarkupSafe==3.0.2
mccabe==0.7.0
mdurl==0.1.2
more-itertools==10.6.0
nameparser==1.1.3
networkx==3.2.1
newick==1.10.0
nh3==0.2.21
numpy==2.0.2
openpyxl==3.1.5
packaging==24.2
platformdirs==4.3.7
pluggy==1.5.0
prompt_toolkit==3.0.50
pybtex==0.24.0
pycdstar==1.1.0
pycldf==1.41.0
pyclts==3.2.0
pycodestyle==2.13.0
pyconcepticon==3.1.0
pycountry==24.6.1
pycparser==2.22
pyflakes==3.3.2
pyglottolog==3.14.0
Pygments==2.19.1
pyigt==2.2.0
pylatexenc==2.10
-e git+https://github.com/lexibank/pylexibank.git@7a360022dbae0311095feb903bb6924e1b659c3a#egg=pylexibank
pyparsing==3.2.3
pytest==8.3.5
pytest-cov==6.0.0
python-dateutil==2.9.0.post0
python-frontmatter==1.1.0
python-nexus==2.9.0
PyYAML==6.0.2
RapidFuzz==3.12.2
rdflib==7.1.4
readme_renderer==44.0
referencing==0.36.2
regex==2024.11.6
requests==2.32.3
requests-toolbelt==1.0.0
rfc3986==1.5.0
rich==14.0.0
rpds-py==0.24.0
SecretStorage==3.3.3
segments==2.3.0
six==1.17.0
smmap==5.0.2
soupsieve==2.6
SQLAlchemy==1.4.54
tabulate==0.9.0
termcolor==3.0.0
TexSoup==0.3.1
thefuzz==0.22.1
tomli==2.2.1
tqdm==4.67.1
twine==6.1.0
typing_extensions==4.13.0
Unidecode==1.3.8
uritemplate==4.1.1
urllib3==2.3.0
wcwidth==0.2.13
Whoosh==2.7.4
xlrd==2.0.1
zenodoclient==0.5.1
zipp==3.21.0

name: pylexibank
channels:
  - defaults
  - https://repo.anaconda.com/pkgs/main
  - https://repo.anaconda.com/pkgs/r
  - conda-forge
dependencies:
  - _libgcc_mutex=0.1=main
  - _openmp_mutex=5.1=1_gnu
  - ca-certificates=2025.2.25=h06a4308_0
  - ld_impl_linux-64=2.40=h12ee557_0
  - libffi=3.4.4=h6a678d5_1
  - libgcc-ng=11.2.0=h1234567_1
  - libgomp=11.2.0=h1234567_1
  - libstdcxx-ng=11.2.0=h1234567_1
  - ncurses=6.4=h6a678d5_0
  - openssl=3.0.16=h5eee18b_0
  - pip=25.0=py39h06a4308_0
  - python=3.9.21=he870216_1
  - readline=8.2=h5eee18b_0
  - setuptools=75.8.0=py39h06a4308_0
  - sqlite=3.45.3=h5eee18b_0
  - tk=8.6.14=h39e8969_0
  - tzdata=2025a=h04d1e81_0
  - wheel=0.45.1=py39h06a4308_0
  - xz=5.6.4=h5eee18b_1
  - zlib=1.2.13=h5eee18b_1
  - pip:
      - appdirs==1.4.4
      - attrs==25.3.0
      - babel==2.17.0
      - backports-tarfile==1.2.0
      - beautifulsoup4==4.13.3
      - bibtexparser==2.0.0b8
      - bs4==0.0.2
      - cdstarcat==1.5.0
      - certifi==2025.1.31
      - cffi==1.17.1
      - charset-normalizer==3.4.1
      - cldfbench==1.14.1
      - cldfcatalog==1.6.0
      - cldfzenodo==2.1.2
      - clldutils==3.24.2
      - colorama==0.4.6
      - colorlog==6.9.0
      - commonnexus==1.9.2
      - coverage==7.8.0
      - cryptography==44.0.2
      - csvw==3.5.1
      - docopt==0.6.2
      - docutils==0.21.2
      - et-xmlfile==2.0.0
      - exceptiongroup==1.2.2
      - flake8==7.2.0
      - gitdb==4.0.12
      - gitpython==3.1.44
      - greenlet==3.1.1
      - id==1.5.0
      - idna==3.10
      - importlib-metadata==8.6.1
      - iniconfig==2.1.0
      - isodate==0.7.2
      - jaraco-classes==3.4.0
      - jaraco-context==6.0.1
      - jaraco-functools==4.1.0
      - jeepney==0.9.0
      - jmespath==1.0.1
      - jsonschema==4.23.0
      - jsonschema-specifications==2024.10.1
      - keyring==25.6.0
      - language-tags==1.2.0
      - latexcodec==3.0.0
      - linglit==1.7.1
      - lingpy==2.6.13
      - lxml==5.3.1
      - markdown==3.7
      - markdown-it-py==3.0.0
      - markupsafe==3.0.2
      - mccabe==0.7.0
      - mdurl==0.1.2
      - more-itertools==10.6.0
      - nameparser==1.1.3
      - networkx==3.2.1
      - newick==1.10.0
      - nh3==0.2.21
      - numpy==2.0.2
      - openpyxl==3.1.5
      - packaging==24.2
      - platformdirs==4.3.7
      - pluggy==1.5.0
      - prompt-toolkit==3.0.50
      - pybtex==0.24.0
      - pycdstar==1.1.0
      - pycldf==1.41.0
      - pyclts==3.2.0
      - pycodestyle==2.13.0
      - pyconcepticon==3.1.0
      - pycountry==24.6.1
      - pycparser==2.22
      - pyflakes==3.3.2
      - pyglottolog==3.14.0
      - pygments==2.19.1
      - pyigt==2.2.0
      - pylatexenc==2.10
      - pyparsing==3.2.3
      - pytest==8.3.5
      - pytest-cov==6.0.0
      - python-dateutil==2.9.0.post0
      - python-frontmatter==1.1.0
      - python-nexus==2.9.0
      - pyyaml==6.0.2
      - rapidfuzz==3.12.2
      - rdflib==7.1.4
      - readme-renderer==44.0
      - referencing==0.36.2
      - regex==2024.11.6
      - requests==2.32.3
      - requests-toolbelt==1.0.0
      - rfc3986==1.5.0
      - rich==14.0.0
      - rpds-py==0.24.0
      - secretstorage==3.3.3
      - segments==2.3.0
      - six==1.17.0
      - smmap==5.0.2
      - soupsieve==2.6
      - sqlalchemy==1.4.54
      - tabulate==0.9.0
      - termcolor==3.0.0
      - texsoup==0.3.1
      - thefuzz==0.22.1
      - tomli==2.2.1
      - tqdm==4.67.1
      - twine==6.1.0
      - typing-extensions==4.13.0
      - unidecode==1.3.8
      - uritemplate==4.1.1
      - urllib3==2.3.0
      - wcwidth==0.2.13
      - whoosh==2.7.4
      - xlrd==2.0.1
      - zenodoclient==0.5.1
      - zipp==3.21.0
prefix: /opt/conda/envs/pylexibank

tests/test_dataset.py::test_Dataset_tokenizer
tests/test_dataset.py::test_Item tests/test_dataset.py::test_Unmapped tests/test_dataset.py::test_invalid_dataset tests/test_dataset.py::test_tokenizer[^b$-b] tests/test_dataset.py::test_tokenizer[aba-c tests/test_dataset.py::test_tokenizer[bab-b
src/pylexibank/dataset.py
src/pylexibank/dataset.py