Doc - Update Model part of the Pipeline page
:class:`~tokenizers.pre_tokenizers.Whitespace`
PreTokenizer
:class:`~tokenizers.pre_tokenizers.PreTokenizer`
+ models.BPE
+ :class:`~tokenizers.models.BPE`
+ models.Unigram
+ :class:`~tokenizers.models.Unigram`
+ models.WordLevel
+ :class:`~tokenizers.models.WordLevel`
+ models.WordPiece
+ :class:`~tokenizers.models.WordPiece`
.. entities:: rust
:rust:struct:`~tokenizers::normalizers::whitespace::Whitespace`
PreTokenizer
:rust:trait:`~tokenizers::tokenizer::PreTokenizer`
+ models.BPE
+ :rust:struct:`~tokenizers::models::bpe::BPE`
+ models.Unigram
+ :rust:struct:`~tokenizers::models::unigram::Unigram`
+ models.WordLevel
+ :rust:struct:`~tokenizers::models::wordlevel::WordLevel`
+ models.WordPiece
+ :rust:struct:`~tokenizers::models::wordpiece::WordPiece`
.. entities:: node
:obj:`Whitespace`
PreTokenizer
:obj:`PreTokenizer`
+ models.BPE
+ :obj:`BPE`
+ models.Unigram
+ :obj:`Unigram`
+ models.WordLevel
+ :obj:`WordLevel`
+ models.WordPiece
+ :obj:`WordPiece`
