diff --git "a/README.md" "b/README.md" new file mode 100644--- /dev/null +++ "b/README.md" @@ -0,0 +1,407 @@ +--- +library_name: setfit +tags: +- setfit +- sentence-transformers +- text-classification +- generated_from_setfit_trainer +metrics: +- accuracy +widget: +- text: hi invoice wheel bearing bernard kavanagh invoice thanks cathy +- text: receive message order id asin product name autoshack radiator replacement + mitsubishi outlander lancer awd fwd message shipment taking long service provide + solely communication buyer please aware amazon never ask provide login information + verify identity service receive message service request seller central login account + information report message ignore request wa email helpful response need report + questionable activity copyright amazon inc affiliate right reserve amazon com + terry avenue north seattle wa information help protect trust safety marketplace + help arbitrate potential dispute retain message buyer seller send amazon com two + year include response message amazon com us filter technology protect buyer seller + possible fraud message fail filter transmit want buy confidence anytime purchase + product amazon com learn safe online shopping safe buying guarantee commmgrtok +- text: amazon com receive message order id asin product name autoshack front drill + slot brake kit rotor silver performance ceramic pad pair driver passenger side + replacement chevrolet hhr fwd message return package week ago thru receive refund + yet issue service provide solely communication buyer please aware amazon never + ask provide login information verify identity service receive message service + request seller central login account information report message ignore request + wa email helpful response need mso +- text: amazon com receive message order id asin product name autoshack catalytic + converter direct fit passenger side replacement infiniti nissan pathfinder armada + titan ar autoshack catalytic converter exhaust pipe direct fit driver side replacement + infiniti nissan pathfinder armada message hello ve contact customer regard order + identify order item reason receive damage defective item detail item ha arrive + wa defective customer want return item please research issue contact customer + respond customer please reply e mail visit seller account following link http + sellercentral amazon com gp communication manager inbox html sincerely customer + service department amazon com http www amazon com service provide solely communication + buyer please aware amazon never ask provide login information verify identity + service receive message service request seller central login account information + report message ignore request wa email helpful response need +- text: create repick order sale order repick order +pipeline_tag: text-classification +inference: true +base_model: sentence-transformers/all-mpnet-base-v2 +model-index: +- name: SetFit with sentence-transformers/all-mpnet-base-v2 + results: + - task: + type: text-classification + name: Text Classification + dataset: + name: Unknown + type: unknown + split: test + metrics: + - type: accuracy + value: 0.2964972866304884 + name: Accuracy +--- + +# SetFit with sentence-transformers/all-mpnet-base-v2 + +This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification. + +The model has been trained using an efficient few-shot learning technique that involves: + +1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning. +2. Training a classification head with features from the fine-tuned Sentence Transformer. + +## Model Details + +### Model Description +- **Model Type:** SetFit +- **Sentence Transformer body:** [sentence-transformers/all-mpnet-base-v2](https://huggingface.co/sentence-transformers/all-mpnet-base-v2) +- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance +- **Maximum Sequence Length:** 384 tokens +- **Number of Classes:** 29 classes + + + + +### Model Sources + +- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit) +- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055) +- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit) + +### Model Labels +| Label | Examples | +|:------------|:----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| __label__25 | | +| __label__27 | | +| __label__16 | | +| __label__17 | | +| __label__30 | | +| __label__18 | | +| __label__29 | | +| __label__19 | | +| __label__26 | | +| __label__24 | | +| __label__31 | | +| __label__0 | | +| __label__22 | | +| __label__23 | | +| __label__33 | | +| __label__28 | | +| __label__13 | | +| __label__4 | | +| __label__7 | | +| __label__8 | | +| __label__21 | | +| __label__3 | | +| __label__15 | | +| __label__20 | | +| __label__2 | | +| __label__35 | | +| __label__5 | | +| __label__34 | | +| __label__11 | | + +## Evaluation + +### Metrics +| Label | Accuracy | +|:--------|:---------| +| **all** | 0.2965 | + +## Uses + +### Direct Use for Inference + +First install the SetFit library: + +```bash +pip install setfit +``` + +Then you can load this model and run inference. + +```python +from setfit import SetFitModel + +# Download from the 🤗 Hub +model = SetFitModel.from_pretrained("pankajkmr/setfit-paraphrase-mpnet-base-v2-sst2-model2") +# Run inference +preds = model("create repick order sale order repick order") +``` + + + + + + + + + +## Training Details + +### Training Set Metrics +| Training set | Min | Median | Max | +|:-------------|:----|:--------|:----| +| Word count | 1 | 90.1931 | 355 | + +| Label | Training Sample Count | +|:------------|:----------------------| +| __label__0 | 15 | +| __label__11 | 15 | +| __label__13 | 15 | +| __label__15 | 15 | +| __label__16 | 15 | +| __label__17 | 15 | +| __label__18 | 15 | +| __label__19 | 15 | +| __label__2 | 15 | +| __label__20 | 15 | +| __label__21 | 15 | +| __label__22 | 15 | +| __label__23 | 15 | +| __label__24 | 15 | +| __label__25 | 15 | +| __label__26 | 15 | +| __label__27 | 15 | +| __label__28 | 15 | +| __label__29 | 15 | +| __label__3 | 15 | +| __label__30 | 15 | +| __label__31 | 15 | +| __label__33 | 15 | +| __label__34 | 15 | +| __label__35 | 15 | +| __label__4 | 15 | +| __label__5 | 15 | +| __label__7 | 15 | +| __label__8 | 15 | + +### Training Hyperparameters +- batch_size: (8, 8) +- num_epochs: (3, 3) +- max_steps: -1 +- sampling_strategy: oversampling +- num_iterations: 20 +- body_learning_rate: (2e-05, 2e-05) +- head_learning_rate: 2e-05 +- loss: CosineSimilarityLoss +- distance_metric: cosine_distance +- margin: 0.25 +- end_to_end: False +- use_amp: False +- warmup_proportion: 0.1 +- seed: 42 +- eval_max_steps: -1 +- load_best_model_at_end: False + +### Training Results +| Epoch | Step | Training Loss | Validation Loss | +|:------:|:----:|:-------------:|:---------------:| +| 0.0005 | 1 | 0.1608 | - | +| 0.0230 | 50 | 0.293 | - | +| 0.0460 | 100 | 0.2219 | - | +| 0.0690 | 150 | 0.3463 | - | +| 0.0920 | 200 | 0.3157 | - | +| 0.1149 | 250 | 0.2663 | - | +| 0.1379 | 300 | 0.1209 | - | +| 0.1609 | 350 | 0.1651 | - | +| 0.1839 | 400 | 0.0464 | - | +| 0.2069 | 450 | 0.1462 | - | +| 0.2299 | 500 | 0.2213 | - | +| 0.2529 | 550 | 0.0992 | - | +| 0.2759 | 600 | 0.2794 | - | +| 0.2989 | 650 | 0.0496 | - | +| 0.3218 | 700 | 0.0408 | - | +| 0.3448 | 750 | 0.064 | - | +| 0.3678 | 800 | 0.1193 | - | +| 0.3908 | 850 | 0.1822 | - | +| 0.4138 | 900 | 0.1966 | - | +| 0.4368 | 950 | 0.1215 | - | +| 0.4598 | 1000 | 0.1847 | - | +| 0.4828 | 1050 | 0.1406 | - | +| 0.5057 | 1100 | 0.2141 | - | +| 0.5287 | 1150 | 0.1418 | - | +| 0.5517 | 1200 | 0.0398 | - | +| 0.5747 | 1250 | 0.1079 | - | +| 0.5977 | 1300 | 0.0704 | - | +| 0.6207 | 1350 | 0.0942 | - | +| 0.6437 | 1400 | 0.0751 | - | +| 0.6667 | 1450 | 0.1463 | - | +| 0.6897 | 1500 | 0.1015 | - | +| 0.7126 | 1550 | 0.104 | - | +| 0.7356 | 1600 | 0.0278 | - | +| 0.7586 | 1650 | 0.0897 | - | +| 0.7816 | 1700 | 0.0089 | - | +| 0.8046 | 1750 | 0.228 | - | +| 0.8276 | 1800 | 0.0159 | - | +| 0.8506 | 1850 | 0.0039 | - | +| 0.8736 | 1900 | 0.0203 | - | +| 0.8966 | 1950 | 0.0768 | - | +| 0.9195 | 2000 | 0.0567 | - | +| 0.9425 | 2050 | 0.0952 | - | +| 0.9655 | 2100 | 0.0251 | - | +| 0.9885 | 2150 | 0.0425 | - | +| 1.0115 | 2200 | 0.0121 | - | +| 1.0345 | 2250 | 0.1579 | - | +| 1.0575 | 2300 | 0.0892 | - | +| 1.0805 | 2350 | 0.0142 | - | +| 1.1034 | 2400 | 0.1206 | - | +| 1.1264 | 2450 | 0.0257 | - | +| 1.1494 | 2500 | 0.102 | - | +| 1.1724 | 2550 | 0.0521 | - | +| 1.1954 | 2600 | 0.0273 | - | +| 1.2184 | 2650 | 0.0205 | - | +| 1.2414 | 2700 | 0.0179 | - | +| 1.2644 | 2750 | 0.0074 | - | +| 1.2874 | 2800 | 0.007 | - | +| 1.3103 | 2850 | 0.1178 | - | +| 1.3333 | 2900 | 0.0051 | - | +| 1.3563 | 2950 | 0.1062 | - | +| 1.3793 | 3000 | 0.0214 | - | +| 1.4023 | 3050 | 0.0295 | - | +| 1.4253 | 3100 | 0.0967 | - | +| 1.4483 | 3150 | 0.0683 | - | +| 1.4713 | 3200 | 0.0019 | - | +| 1.4943 | 3250 | 0.1584 | - | +| 1.5172 | 3300 | 0.0719 | - | +| 1.5402 | 3350 | 0.0091 | - | +| 1.5632 | 3400 | 0.1362 | - | +| 1.5862 | 3450 | 0.055 | - | +| 1.6092 | 3500 | 0.0095 | - | +| 1.6322 | 3550 | 0.194 | - | +| 1.6552 | 3600 | 0.004 | - | +| 1.6782 | 3650 | 0.0807 | - | +| 1.7011 | 3700 | 0.0566 | - | +| 1.7241 | 3750 | 0.0024 | - | +| 1.7471 | 3800 | 0.0374 | - | +| 1.7701 | 3850 | 0.013 | - | +| 1.7931 | 3900 | 0.0662 | - | +| 1.8161 | 3950 | 0.0871 | - | +| 1.8391 | 4000 | 0.0112 | - | +| 1.8621 | 4050 | 0.03 | - | +| 1.8851 | 4100 | 0.1157 | - | +| 1.9080 | 4150 | 0.1204 | - | +| 1.9310 | 4200 | 0.0019 | - | +| 1.9540 | 4250 | 0.0083 | - | +| 1.9770 | 4300 | 0.055 | - | +| 2.0 | 4350 | 0.1002 | - | +| 2.0230 | 4400 | 0.0335 | - | +| 2.0460 | 4450 | 0.038 | - | +| 2.0690 | 4500 | 0.0134 | - | +| 2.0920 | 4550 | 0.042 | - | +| 2.1149 | 4600 | 0.089 | - | +| 2.1379 | 4650 | 0.0408 | - | +| 2.1609 | 4700 | 0.0022 | - | +| 2.1839 | 4750 | 0.118 | - | +| 2.2069 | 4800 | 0.0632 | - | +| 2.2299 | 4850 | 0.0046 | - | +| 2.2529 | 4900 | 0.0054 | - | +| 2.2759 | 4950 | 0.0159 | - | +| 2.2989 | 5000 | 0.0049 | - | +| 2.3218 | 5050 | 0.0032 | - | +| 2.3448 | 5100 | 0.0334 | - | +| 2.3678 | 5150 | 0.0104 | - | +| 2.3908 | 5200 | 0.0171 | - | +| 2.4138 | 5250 | 0.0723 | - | +| 2.4368 | 5300 | 0.101 | - | +| 2.4598 | 5350 | 0.0785 | - | +| 2.4828 | 5400 | 0.0686 | - | +| 2.5057 | 5450 | 0.012 | - | +| 2.5287 | 5500 | 0.1446 | - | +| 2.5517 | 5550 | 0.032 | - | +| 2.5747 | 5600 | 0.0022 | - | +| 2.5977 | 5650 | 0.0127 | - | +| 2.6207 | 5700 | 0.1638 | - | +| 2.6437 | 5750 | 0.0039 | - | +| 2.6667 | 5800 | 0.0242 | - | +| 2.6897 | 5850 | 0.0337 | - | +| 2.7126 | 5900 | 0.0325 | - | +| 2.7356 | 5950 | 0.0024 | - | +| 2.7586 | 6000 | 0.0165 | - | +| 2.7816 | 6050 | 0.0015 | - | +| 2.8046 | 6100 | 0.0293 | - | +| 2.8276 | 6150 | 0.0008 | - | +| 2.8506 | 6200 | 0.0407 | - | +| 2.8736 | 6250 | 0.0032 | - | +| 2.8966 | 6300 | 0.0312 | - | +| 2.9195 | 6350 | 0.0143 | - | +| 2.9425 | 6400 | 0.0291 | - | +| 2.9655 | 6450 | 0.0017 | - | +| 2.9885 | 6500 | 0.1199 | - | + +### Framework Versions +- Python: 3.10.12 +- SetFit: 1.0.1 +- Sentence Transformers: 2.2.2 +- Transformers: 4.36.0 +- PyTorch: 2.0.0 +- Datasets: 2.16.1 +- Tokenizers: 0.15.0 + +## Citation + +### BibTeX +```bibtex +@article{https://doi.org/10.48550/arxiv.2209.11055, + doi = {10.48550/ARXIV.2209.11055}, + url = {https://arxiv.org/abs/2209.11055}, + author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren}, + keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences}, + title = {Efficient Few-Shot Learning Without Prompts}, + publisher = {arXiv}, + year = {2022}, + copyright = {Creative Commons Attribution 4.0 International} +} +``` + + + + + + \ No newline at end of file