  from __future__ import unicode_literals
  
  import pytest
  from spacy.lang.en import English
  from spacy.tokenizer import Tokenizer
  from spacy.util import compile_prefix_regex, compile_suffix_regex
  from spacy.util import compile_infix_regex
  
  
 
</s> remove             for string, substrings in data.get("rules", {}).items():
                self.add_special_case(string, substrings)
 </s> add             self._load_special_tokenization(data.get("rules", {})) </s> remove         for chunk, substrings in sorted(special_cases.items()):
            self.add_special_case(chunk, substrings)
 </s> add         if special_cases is not None:
            for chunk, substrings in sorted(special_cases.items()):
                self.add_special_case(chunk, substrings) </s> remove             if self.token_match and self.token_match(string) \
                    and not self.find_prefix(string) \
                    and not self.find_suffix(string):
                break
 </s> add 
import re
  from __future__ import unicode_literals
  
  import pytest
import re
  from spacy.lang.en import English
  from spacy.tokenizer import Tokenizer
  from spacy.util import compile_prefix_regex, compile_suffix_regex
  from spacy.util import compile_infix_regex
  
  
keep keep add keep keep keep keep keep keep
 Add tokenizer explain() debugging method (#4596)

* Expose tokenizer rules as a property

Expose the tokenizer rules property in the same way as the other core
properties. (The cache resetting is overkill, but consistent with
`from_bytes` for now.)

Add tests and update Tokenizer API docs.

* Update Hungarian punctuation to remove empty string

Update Hungarian punctuation definitions so that `_units` does not match
an empty string.

* Use _load_special_tokenization consistently

Use `_load_special_tokenization()` and have it to handle `None` checks.

* Fix precedence of `token_match` vs. special cases

Remove `token_match` check from `_split_affixes()` so that special cases
have precedence over `token_match`. `token_match` is checked only before
infixes are split.

* Add `make_debug_doc()` to the Tokenizer

Add `make_debug_doc()` to the Tokenizer as a working implementation of
the pseudo-code in the docs.

Add a test (marked as slow) that checks that `nlp.tokenizer()` and
`nlp.tokenizer.make_debug_doc()` return the same non-whitespace tokens
for all languages that have `examples.sentences` that can be imported.

* Update tokenization usage docs

Update pseudo-code and algorithm description to correspond to
`nlp.tokenizer.make_debug_doc()` with example debugging usage.

Add more examples for customizing tokenizers while preserving the
existing defaults.

Minor edits / clarifications.

* Revert "Update Hungarian punctuation to remove empty string"

This reverts commit f0a577f7a5c67f55807fdbda9e9a936464723931.

* Rework `make_debug_doc()` as `explain()`

Rework `make_debug_doc()` as `explain()`, which returns a list of
`(pattern_string, token_string)` tuples rather than a non-standard
`Doc`. Update docs and tests accordingly, leaving the visualization for
future work.

* Handle cases with bad tokenizer patterns

Detect when tokenizer patterns match empty prefixes and suffixes so that
`explain()` does not hang on bad patterns.

* Remove unused displacy image

* Add tokenizer.explain() to usage docs 
spacy/tests/lang/en/test_customized_tokenizer.py