File size: 769 Bytes
035c85d e0efd45 035c85d e0efd45 035c85d e0efd45 035c85d e0efd45 035c85d e0efd45 035c85d e0efd45 035c85d e0efd45 035c85d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 |
---
library_name: transformers
license: apache-2.0
---
See [transformers で複数のトークナイザーを一つのプロセッサーで扱う](https://zenn.dev/platina/articles/732feb7c3e9852).
https://zenn.dev/platina/articles/732feb7c3e9852
## Example usage
```py
from transformers import AutoProcessor
processor = AutoProcessor.from_pretrained(
"p1atdev/multi-tokenizers-processor-sample",
trust_remote_code=True,
commit_hash="111e8a30609fb5bc13e16d08f7c49196b23d5056"
)
print(processor(
text_1="テキスト1",
text_2="テキスト2",
))
# {'input_ids': tensor([[ 1, 43412, 28745]]), 'attention_mask': tensor([[1, 1, 1]]), 'input_ids_2': tensor([[56833, 61803, 70534, 17]]), 'attention_mask_2': tensor([[1, 1, 1, 1]])}
```
|