Add SetFit model

1_Pooling/config.json

@@ -0,0 +1,10 @@
+{
+  "word_embedding_dimension": 768,
+  "pooling_mode_cls_token": false,
+  "pooling_mode_mean_tokens": true,
+  "pooling_mode_max_tokens": false,
+  "pooling_mode_mean_sqrt_len_tokens": false,
+  "pooling_mode_weightedmean_tokens": false,
+  "pooling_mode_lasttoken": false,
+  "include_prompt": true
+}

README.md

@@ -0,0 +1,313 @@
+---
+language: en
+license: apache-2.0
+library_name: setfit
+tags:
+- setfit
+- sentence-transformers
+- text-classification
+- generated_from_setfit_trainer
+base_model: sentence-transformers/paraphrase-mpnet-base-v2
+metrics:
+- accuracy
+- precision
+- recall
+- f1
+widget:
+- text: 'in the shadows of arrakis lie one secret... dune 2 is a masterpiece the film
+    starts exactly at the moment the first part ended. although according to villeneuve
+    it is designed to be understood without having seen the first part, i highly recommend
+    watching dune: part one before.the first half of the film is poetic. great shots
+    of the desert. developing new characters (especially fremen) that we had seen
+    briefly in the first part and, above all, we can see an evolution of some characters
+    that we already knew. i highlight here the character of stilgar (javier bardem)
+    and chani (zendaya), who undoubtedly gets one of the great performances of the
+    film. bravo to villeneuve for having developed chani''s character (wasted in herbert''s
+    novel), here chani shines until the end. bardem, is also a fundamental piece that
+    gives freshness to the serious and majestic plot, adding dynamism and even funny
+    moments.the second half of the film goes to paul atreides. i can''t say too much
+    here, only that we see one of the greatest evolutions of a character that i''ve
+    been seen in cinema. the last 45 minutes of the film are simply some of the best
+    i have ever witnessed on a screen. everything that was harvested in dune: part
+    one now makes complete sense. the characters of florence pugh and austin butler
+    stand out, the first one, in the background but being a totally key character,
+    especially for everything that is going to come (messiah); and the second one
+    for his excellent performance as a villain (far above the baron, a villain who
+    personally never convinced me).all of this is great, however, an element of equal
+    importance to everything mentioned was: the music of hans zimmer. spectacular
+    songs and sounds that make you simply feel vibrate (literally... if you see it
+    in imax) every moment inside you. it is no coincidence that nolan''s best films
+    have zimmer''s seal. his music creates art. his music creates blockbusters. not
+    because it''s commercial, but because it creates emotions within you. every note,
+    sound and song are placed at the right time and moment in the film. hans zimmer''s
+    music is undoubtedly the specie, the water of life... of dune: part two.with all
+    this we are left with what is not only the best science fiction film ever created,
+    but what is (be careful what i am going to say) the best film of this 21st century
+    (at least to date), with permission from the dark knight and interstellar.villeneuve
+    gives us a great masterpiece. here, however, the messiah is not paul atreides,
+    nor villeneuve himself... the messiah is the spectator. you. this film will push
+    you to be a better you, it will make you dream big, it will remind you that, like
+    paul, you can decide when to lead your life and inspire others. you are the chosen
+    one to, within a simple movie, find the gold that will make you better in your
+    daily life.this valuable lesson is not achieved by ordinary pieces of art, but
+    only by a few works... which some of us dare to call... masterpieces.because...
+    in the shadows of arrakis lie one secret... dune 2 is a masterpiece.'
+- text: does justice to the books awesome! stunning! the film follows through the
+    spirit of the books with the ever-present internal turmoil of paul. it has the
+    feel of mysticism of the books. the cinematography is spectacular and so is hans
+    zimmer's music. the freemen and harkons are depicted really well and so is the
+    planet dune. at the same time, there are many details and happenings left behind,
+    so for people who have not read the books it's a bit difficult to follow the narrative.
+    the movie doesn't follow thoroughly the destiny of the other of the closest friends
+    and servents of the duke, who survived the massacre in part 1. 5 out of 8 found
+    this helpful. was this review helpful? sign in to vote. permalink
+- text: epic but empty i'll precursor this by saying 1) i have not read the books
+    2) i am a huge denis villeneuve fan 3) i found part one underwhelming.while i
+    find there to be many issues with this film, perhaps its biggest weakness imo
+    is that i hold no emotional connection with any of the characters, despite the
+    film's lengthy runtime. the protagonist's mother is written as incredibly annoying,
+    the villains are underused, florence pugh's character has 0 likability and acts
+    simply as a story device. bardem's character is ok and funny at times and brolin
+    doesn't have a standout moment. zendaya's character is the sole exception to this,
+    who is the beating heart of the film.at times, i also felt that chalamet, despite
+    being an amazing actor, is perhaps miscast, though having not read the material,
+    i cannot comment further on this.the pacing felt off, as it stays too long in
+    places it shouldn't, while sacrificing moments that could linger longer. this
+    coming from a person that enjoys long epics and slow burn films. the movie looks
+    outstanding though feels closer to a connection of rushed story beats. the direction
+    it takes with the main character also does not feel earned.perhaps the second
+    biggest issue is that i do not totally buy into the world that has been created.
+    it feels large but empty and lacking character. almost too clean and strange for
+    the sake of it. style over substance. we stay for most of the time in one or two
+    places which doesn't help. there were also some weird choice of cuts and jumps
+    in scenes that hampered the flow of the film.with that said, there are a couple
+    of epic moments that stood out, which are 1) the whole zendaya bazooka scene and
+    2) the giant worm riding.overall, the film is enjoyable. however without engaging
+    characters, a believable world, and pacing issues, the film falls short of greatness,
+    perhaps way short.
+- text: pure respect for the original book dune has got to be one of the hardest books
+    to translate into a film. so much of the writing comes from within each characters
+    head.watching denis villeneuve take every important detail and bring it to life
+    was nothing short of pure wonder. you can really tell that dune was one of his
+    favourite stories. his visual direction, the breathtaking audio, the beautiful
+    sets, and the top-tier acting are all big elements in the overall production...
+    but sometimes there's there's something else beyond that. the passion that really
+    shines through.one of the best movies ever made. period. kudos to everyone involved
+    in every aspect of the production. and a huge thank you to the studio for allowing
+    these people to perform their craft to the best of their abilities. 2 out of 3
+    found this helpful. was this review helpful? sign in to vote. permalink
+- text: 'movie-making excellence!!! denis villeneuve''s dune: part two is one of the
+    best sci-fi fantasy movies to be ever produced.the adapted screenplay is almost
+    true to its source material. the "spice" of the book has been captured faithfully
+    and even there are few deviations from the book, i choose to believe the same
+    would have taken to adapt the movie in best possible manner.the soundtrack of
+    the movie by the han zimmer elevates the mood and make each scene feel meaningful
+    and epic. the sound design complements the visual prowess perfectly.dennis villeneue''s
+    dune trilogy will be his magnus opus. each shot has been carefully crafted to
+    almost perfection leaving no waste behind. we are truly blessed to witness his
+    work.with an ensemble cast of industry veterans and hollywood''s future biggest
+    and brightest stars, the casting of new characters can be considered as great.
+    each actor/actress have done justice to their roles with standout performances
+    by rebecca ferguson, zendaya, timothee and austin butler.production design, costumes
+    and make-up is excellenta must watch for every movie-goer. experience it in imax
+    if possible.'
+pipeline_tag: text-classification
+inference: true
+model-index:
+- name: SetFit with sentence-transformers/paraphrase-mpnet-base-v2 on data/raw/15239678.jsonl
+  results:
+  - task:
+      type: text-classification
+      name: Text Classification
+    dataset:
+      name: data/raw/15239678.jsonl
+      type: unknown
+      split: test
+    metrics:
+    - type: accuracy
+      value: 0.8571428571428571
+      name: Accuracy
+    - type: precision
+      value: 0.9959514170040485
+      name: Precision
+    - type: recall
+      value: 0.8512110726643599
+      name: Recall
+    - type: f1
+      value: 0.917910447761194
+      name: F1
+---
+
+# SetFit with sentence-transformers/paraphrase-mpnet-base-v2 on data/raw/15239678.jsonl
+
+This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification.
+
+The model has been trained using an efficient few-shot learning technique that involves:
+
+1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning.
+2. Training a classification head with features from the fine-tuned Sentence Transformer.
+
+## Model Details
+
+### Model Description
+- **Model Type:** SetFit
+- **Sentence Transformer body:** [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2)
+- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance
+- **Maximum Sequence Length:** 512 tokens
+- **Number of Classes:** 2 classes
+<!-- - **Training Dataset:** [Unknown](https://huggingface.co/datasets/unknown) -->
+- **Language:** en
+- **License:** apache-2.0
+
+### Model Sources
+
+- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit)
+- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055)
+- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit)
+
+### Model Labels
+| Label    | Examples                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
+|:---------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| negative | <ul><li>'slow, uninspiring, borng i think the second film just confirmed that dune isn\'t for me.i found the first one rather long and a bit boring. this second part didn\'t do much to change my mind. i gave it a try through.some things didn\'t make sense.1. why, if there are rifles/laser weapons are the majority of the battles found hand to hand with swords? especially when protecting the valuable spice depot on arrakis. it seemed like a really illogical move and a terrible military tactic. i felt like it was done more for the visual "spectacle" of having hand to hand fighting but it didn\'t make sense in universe.2. the fremen seemingly have to trek great distances to get to these attack sites, there was a scene near the end where their army all.aprar st the top of the sand dune by the depot and would be absolutely knackered by the time they traversed the miles of sand between them and the enemy3. this brings me on to my point about them "riding" the worms as they seemingly have little control apart from nominal left and right movement. you never see them stopping the worms and dismounting. the worms never seem to stop so you would likely have to leap off, almost certainly to your death.linked into this there were scenes where there was a whole caravan of people including lady jessica in a fremen style sedan chair, how the hell did they all safely mount the worm and remain tethered and again how did they all dismount?lastly there was the horrible dirge of hans zimmer\'s "score". i know this is just personal taste, but i prefer music with a theme and melody rather than just atonal blats from a foghorn.i\'m sure dune fans will absolutely love this and i\'m pleased they\'ve got a faithful adaptation. i also recognise the efforts of the cast and crew i just found it to be overly long and a bit dull.this was a watch once and don\'t need to see it again film for me.i won\'t be watching the next one.so caught up in it\'s own mythology that it fails to give any interesting characters that you can root for.it inspires no emotional engagement when the characters are all dull.visually bland.'</li><li>"broke the story for no reason movies don't perfectly reflections the book and we shouldn't expect them to.that said, dune part 2 diverged in key areas for no good reason.the most inexplicable is that the whole storyline from 'will the tribe accept paul and jessica?' to paul uniting the tribes and leading them in the huge final battle takes place while jessica is pregnant. in the book her child is already about 3 - a far more plausible timescale.no changes to the movie would have been needed except adding the birth and the character of alia in a few scenes - the key one being the between the baron and the emperor.also, chani is shown as opposing paul and leaves him, the opposite of their relationship in the book.it's hard to see how a sequel can make these changes fit the storyline while keeping it intact."</li><li>"it just didn't work for me my apologies to all who were enthralled by dune ii. i respect your experience. maybe if i'd have seen dune i it would have helped but i don't think so. the jerry-rigging of several religions seemed pretentious and trite while the half-dozen plot threads unraveled regularly. sweeping cig backgrounds soon became tiresome. trying to keep supporting characters straight was difficult. i tried but even at the end i was unsure of what had happened to relationships, especially to the women characters. critics have outlined what it all was supposed to be about, but i found it a bit dull. maybe i'll wait for dune i on the streams. apologies to all. 5 out of 21 found this helpful. was this review helpful? sign in to vote. permalink"</li></ul> |
+| positive | <ul><li>"a visual masterpiece dune part 2 is dope to the most extent.visually wonderful and elevative screenplay works big time.that sound mixing was from a different planet, outrageous at times. i'd be damned if visuals and sound won't win the oscars.it stays mostly true to the novel. it's been a few years since i've read it but i didn't find stark differences.there's an ample scope for all the lead cast and everyone did a fantastic job. timothee chalamet as paul artreidis and zendaya as chani had a terrific screen presence.a must watch.at times i felt that it was an indian film due to the screenplay and action sequences.8.5 stars."</li><li>"no words the best sequel i have ever seen in cinema. a masterpiece.coming into dune part two on day 2 of it's uk wide release and having already seen the unwavering reviews i was in high anticipation for what i believed could be the triumph in cinema we have been needing for quite some time. a feat that could actually set me up for disappointment. however, true to the words of my predecessors who saw the film before me, it was simply otherworldly.denis is a visionary. a creative mind that only comes around in hollywood once in a blue moon. this film is beautiful, it's vivid, it's dense in all the best ways.imax has to be a non negotiable. the images and sound are unlike many movies i have ever seen.go and see this picture immediately. 3 out of 4 found this helpful. was this review helpful? sign in to vote. permalink"</li><li>"a cinematic masterpiece.. i can't believe my eyes what just i experienced in theatre, this movie is one of the best cinematic experience i've ever had.. from opening scene to end credit every second is worth it.. epic score by hans zimmer elevates the experience, filled with beautiful shots and cinematography to a great scrrenplay and some top notch acting by every cast member dune 2 will be remembered as one of the best films of this generation even after 50 years later that's my guarantee.. denis villeneuve is one of the best director of this generation who gave us masterpiece like blade runner 2049 and currently working on 3rd part of this trilogy which i can't wait to see.. action scenes were great featuring big ships, blasters laser guns and every other kind of weapons, sound effects were amazing but most importantly screenplay was well paced and story featuring blend of sci fi and fantasy elements is absolute best of the best even though it is an adaptation of 1965 novel of same name it is one of the best adaptations ever.. i loved both movies and can't wait to see what happens in future because i enjoy films about the chosen one prophecy like star wars, the matrix and now this.. i won't give any spoiler here and will not talk about anything shown in the movie since you need to see this movie for yourself to feel the magic.. don't miss it at any cost.. watch it on the biggest screen possible especially imax to enjoy epic sound effects and see the larger than life picture you'll love it.."</li></ul>                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
+
+## Evaluation
+
+### Metrics
+| Label   | Accuracy | Precision | Recall | F1     |
+|:--------|:---------|:----------|:-------|:-------|
+| **all** | 0.8571   | 0.9960    | 0.8512 | 0.9179 |
+
+## Uses
+
+### Direct Use for Inference
+
+First install the SetFit library:
+
+```bash
+pip install setfit
+```
+
+Then you can load this model and run inference.
+
+```python
+from setfit import SetFitModel
+
+# Download from the 🤗 Hub
+model = SetFitModel.from_pretrained("carlesoctav/SentimentClassifierDune64shot")
+# Run inference
+preds = model("does justice to the books awesome! stunning! the film follows through the spirit of the books with the ever-present internal turmoil of paul. it has the feel of mysticism of the books. the cinematography is spectacular and so is hans zimmer's music. the freemen and harkons are depicted really well and so is the planet dune. at the same time, there are many details and happenings left behind, so for people who have not read the books it's a bit difficult to follow the narrative. the movie doesn't follow thoroughly the destiny of the other of the closest friends and servents of the duke, who survived the massacre in part 1. 5 out of 8 found this helpful. was this review helpful? sign in to vote. permalink")
+```
+
+<!--
+### Downstream Use
+
+*List how someone could finetune this model on their own dataset.*
+-->
+
+<!--
+### Out-of-Scope Use
+
+*List how the model may foreseeably be misused and address what users ought not to do with the model.*
+-->
+
+<!--
+## Bias, Risks and Limitations
+
+*What are the known or foreseeable issues stemming from this model? You could also flag here known failure cases or weaknesses of the model.*
+-->
+
+<!--
+### Recommendations
+
+*What are recommendations with respect to the foreseeable issues? For example, filtering explicit content.*
+-->
+
+## Training Details
+
+### Training Set Metrics
+| Training set | Min | Median   | Max  |
+|:-------------|:----|:---------|:-----|
+| Word count   | 108 | 228.9219 | 1595 |
+
+| Label    | Training Sample Count |
+|:---------|:----------------------|
+| negative | 64                    |
+| positive | 64                    |
+
+### Training Hyperparameters
+- batch_size: (16, 16)
+- num_epochs: (1, 1)
+- max_steps: -1
+- sampling_strategy: oversampling
+- body_learning_rate: (2e-05, 1e-05)
+- head_learning_rate: 0.01
+- loss: CosineSimilarityLoss
+- distance_metric: cosine_distance
+- margin: 0.25
+- end_to_end: False
+- use_amp: False
+- warmup_proportion: 0.1
+- seed: 42
+- eval_max_steps: -1
+- load_best_model_at_end: True
+
+### Training Results
+| Epoch   | Step    | Training Loss | Validation Loss |
+|:-------:|:-------:|:-------------:|:---------------:|
+| 0.0019  | 1       | 0.2744        | -               |
+| 0.0962  | 50      | 0.0513        | -               |
+| 0.1923  | 100     | 0.0022        | -               |
+| 0.2885  | 150     | 0.0003        | -               |
+| 0.3846  | 200     | 0.0001        | -               |
+| 0.4808  | 250     | 0.0001        | -               |
+| 0.5769  | 300     | 0.0001        | -               |
+| 0.6731  | 350     | 0.0001        | -               |
+| 0.7692  | 400     | 0.0001        | -               |
+| 0.8654  | 450     | 0.0001        | -               |
+| 0.9615  | 500     | 0.0001        | -               |
+| **1.0** | **520** | **-**         | **0.1845**      |
+
+* The bold row denotes the saved checkpoint.
+### Framework Versions
+- Python: 3.10.11
+- SetFit: 1.0.3
+- Sentence Transformers: 2.5.1
+- Transformers: 4.38.2
+- PyTorch: 2.0.1
+- Datasets: 2.18.0
+- Tokenizers: 0.15.2
+
+## Citation
+
+### BibTeX
+```bibtex
+@article{https://doi.org/10.48550/arxiv.2209.11055,
+    doi = {10.48550/ARXIV.2209.11055},
+    url = {https://arxiv.org/abs/2209.11055},
+    author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren},
+    keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences},
+    title = {Efficient Few-Shot Learning Without Prompts},
+    publisher = {arXiv},
+    year = {2022},
+    copyright = {Creative Commons Attribution 4.0 International}
+}
+```
+
+<!--
+## Glossary
+
+*Clearly define terms in order to be accessible across audiences.*
+-->
+
+<!--
+## Model Card Authors
+
+*Lists the people who create the model card, providing recognition and accountability for the detailed work that goes into its construction.*
+-->
+
+<!--
+## Model Card Contact
+
+*Provides a way for people who have updates to the Model Card, suggestions, or questions, to contact the Model Card authors.*
+-->

config.json

@@ -0,0 +1,24 @@
+{
+  "_name_or_path": "exp/Dune2Classifier64shot/step_520",
+  "architectures": [
+    "MPNetModel"
+  ],
+  "attention_probs_dropout_prob": 0.1,
+  "bos_token_id": 0,
+  "eos_token_id": 2,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.1,
+  "hidden_size": 768,
+  "initializer_range": 0.02,
+  "intermediate_size": 3072,
+  "layer_norm_eps": 1e-05,
+  "max_position_embeddings": 514,
+  "model_type": "mpnet",
+  "num_attention_heads": 12,
+  "num_hidden_layers": 12,
+  "pad_token_id": 1,
+  "relative_attention_num_buckets": 32,
+  "torch_dtype": "float32",
+  "transformers_version": "4.38.2",
+  "vocab_size": 30527
+}

config_sentence_transformers.json

@@ -0,0 +1,9 @@
+{
+  "__version__": {
+    "sentence_transformers": "2.0.0",
+    "transformers": "4.7.0",
+    "pytorch": "1.9.0+cu102"
+  },
+  "prompts": {},
+  "default_prompt_name": null
+}

config_setfit.json

@@ -0,0 +1,7 @@
+{
+  "labels": [
+    "negative",
+    "positive"
+  ],
+  "normalize_embeddings": false
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6ea14d6928c5a41ab0f31eace5c153fe78a34996cf2bade02d6fb97514dc80c9
+size 437967672

model_head.pkl

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:1d47ad04c5caf47e03c41efcd026345e1455e6170d4dfeef5b712ba1cabb0287
+size 6997

modules.json

@@ -0,0 +1,14 @@
+[
+  {
+    "idx": 0,
+    "name": "0",
+    "path": "",
+    "type": "sentence_transformers.models.Transformer"
+  },
+  {
+    "idx": 1,
+    "name": "1",
+    "path": "1_Pooling",
+    "type": "sentence_transformers.models.Pooling"
+  }
+]

sentence_bert_config.json

@@ -0,0 +1,4 @@
+{
+  "max_seq_length": 512,
+  "do_lower_case": false
+}

special_tokens_map.json

@@ -0,0 +1,51 @@
+{
+  "bos_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "cls_token": {
+    "content": "<s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "eos_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "mask_token": {
+    "content": "<mask>",
+    "lstrip": true,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "pad_token": {
+    "content": "<pad>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "sep_token": {
+    "content": "</s>",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  },
+  "unk_token": {
+    "content": "[UNK]",
+    "lstrip": false,
+    "normalized": false,
+    "rstrip": false,
+    "single_word": false
+  }
+}

tokenizer_config.json

@@ -0,0 +1,66 @@
+{
+  "added_tokens_decoder": {
+    "0": {
+      "content": "<s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "1": {
+      "content": "<pad>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "2": {
+      "content": "</s>",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "104": {
+      "content": "[UNK]",
+      "lstrip": false,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    },
+    "30526": {
+      "content": "<mask>",
+      "lstrip": true,
+      "normalized": false,
+      "rstrip": false,
+      "single_word": false,
+      "special": true
+    }
+  },
+  "bos_token": "<s>",
+  "clean_up_tokenization_spaces": true,
+  "cls_token": "<s>",
+  "do_basic_tokenize": true,
+  "do_lower_case": true,
+  "eos_token": "</s>",
+  "mask_token": "<mask>",
+  "max_length": 512,
+  "model_max_length": 512,
+  "never_split": null,
+  "pad_to_multiple_of": null,
+  "pad_token": "<pad>",
+  "pad_token_type_id": 0,
+  "padding_side": "right",
+  "sep_token": "</s>",
+  "stride": 0,
+  "strip_accents": null,
+  "tokenize_chinese_chars": true,
+  "tokenizer_class": "MPNetTokenizer",
+  "truncation_side": "right",
+  "truncation_strategy": "longest_first",
+  "unk_token": "[UNK]"
+}