          cdef unicode minus_pre
          cdef unicode minus_suf
          cdef size_t last_size = 0
          while string and len(string) != last_size:
              if self.token_match and self.token_match(string) \
                      and not self.find_prefix(string) \
                      and not self.find_suffix(string):
                  break
              if self._specials.get(hash_string(string)) != NULL:
                  has_special[0] = 1
                  break
              last_size = len(string)
              pre_len = self.find_prefix(string)
 
</s> remove         token_match=None,
 </s> add         token_match=token_match_re.match, </s> remove         for chunk, substrings in sorted(special_cases.items()):
            self.add_special_case(chunk, substrings)
 </s> add         if special_cases is not None:
            for chunk, substrings in sorted(special_cases.items()):
                self.add_special_case(chunk, substrings) </s> remove         if rules is not None:
            for chunk, substrings in sorted(rules.items()):
                self.add_special_case(chunk, substrings)
 </s> add         self._load_special_tokenization(rules)
          cdef unicode minus_pre
          cdef unicode minus_suf
          cdef size_t last_size = 0
          while string and len(string) != last_size:




              if self._specials.get(hash_string(string)) != NULL:
                  has_special[0] = 1
                  break
              last_size = len(string)
              pre_len = self.find_prefix(string)
keep keep keep keep replace replace replace replace keep keep keep keep keep
 Add tokenizer explain() debugging method (#4596)

* Expose tokenizer rules as a property

Expose the tokenizer rules property in the same way as the other core
properties. (The cache resetting is overkill, but consistent with
`from_bytes` for now.)

Add tests and update Tokenizer API docs.

* Update Hungarian punctuation to remove empty string

Update Hungarian punctuation definitions so that `_units` does not match
an empty string.

* Use _load_special_tokenization consistently

Use `_load_special_tokenization()` and have it to handle `None` checks.

* Fix precedence of `token_match` vs. special cases

Remove `token_match` check from `_split_affixes()` so that special cases
have precedence over `token_match`. `token_match` is checked only before
infixes are split.

* Add `make_debug_doc()` to the Tokenizer

Add `make_debug_doc()` to the Tokenizer as a working implementation of
the pseudo-code in the docs.

Add a test (marked as slow) that checks that `nlp.tokenizer()` and
`nlp.tokenizer.make_debug_doc()` return the same non-whitespace tokens
for all languages that have `examples.sentences` that can be imported.

* Update tokenization usage docs

Update pseudo-code and algorithm description to correspond to
`nlp.tokenizer.make_debug_doc()` with example debugging usage.

Add more examples for customizing tokenizers while preserving the
existing defaults.

Minor edits / clarifications.

* Revert "Update Hungarian punctuation to remove empty string"

This reverts commit f0a577f7a5c67f55807fdbda9e9a936464723931.

* Rework `make_debug_doc()` as `explain()`

Rework `make_debug_doc()` as `explain()`, which returns a list of
`(pattern_string, token_string)` tuples rather than a non-standard
`Doc`. Update docs and tests accordingly, leaving the visualization for
future work.

* Handle cases with bad tokenizer patterns

Detect when tokenizer patterns match empty prefixes and suffixes so that
`explain()` does not hang on bad patterns.

* Remove unused displacy image

* Add tokenizer.explain() to usage docs 
spacy/tokenizer.pyx