diff --git "a/README.md" "b/README.md" new file mode 100644--- /dev/null +++ "b/README.md" @@ -0,0 +1,502 @@ +--- +tags: +- setfit +- sentence-transformers +- text-classification +- generated_from_setfit_trainer +widget: +- text: 'client: Thank you for [NAME], partners with [ORGANIZATION]. Please listen + to the options carefully, secure the right team. We record calls for quality and + training purposes. Please enter your account number using your telephone keypad, + ignoring any letters, then press the hash key. Just use your telephone keypad + to enter your account number, then press the hash key. + + customer: I''m trying. + + client: This may start with it. Okay, just a moment whilst I look that up. I''m + afraid that account number doesn''t match our records. Please try again. Okay, + just a moment whilst I put that up. Please enter your phone number using the telephone + teapad. + + customer: Thank you. + + client: I''m sorry, I didn''t understand. Try entering your phone number. I''m + sorry, I still didn''t understand. Please by entering it again.' +- text: 'client: Thank you for calling Dino Plumbing. Please note all our calls are + recorded for monitoring and training passes. Press one for an update on parts. + Press two for an update on your appointment. Press three to discuss an ongoing + complaint. Press four for sales. Thank you for calling Dino Plumbing. Please note + all our calls are recorded for monitoring and training purposes. Press one for + an update on parts. Press two for an update on your appointment. Press three to + discuss an ongoing complaint and press four for sales. Thank you for calling dino + plumbing. + + customer: and get some rolls. + + client: Your call is very important to us. Your call is very important to us. + All of our agents are currently busy assisting other customers. Please continue + to hold and the next available agent will take your call. Alternatively, you can + email. Good afternoon if you''re starting and I can help. + + customer: Oh, hello. I called yesterday regarding a plumbing issue, a drainage + issue at one of our properties. He took all of my details and said that we were + going to call us back, but I never got a call back. + + client: I''m not sure postcode or have a little look for you to apologize. + + customer: The post code of the property, NW6, NW6, 1 UJ. + + client: Yes, please. No problem. And it''s the door number 46? 48, okay. + + customer: 48, which in [LOCATION]. But maybe, I can''t remember if it was a charge + of [LOCATION]. Yes, yes, yes, yes, yes. + + client: It was for a chargeable job, wasn''t it? + + customer: All right, lovely, thank you. Yes, it''s a charge call job, yes, yes. + + client: No problem. Let me just see if a gentleman''s available. Give me two seconds. + We''ll go straight through if it does connect. Thank you. + + customer: I can''t do everything? It''s [LOCATION], hi [NAME]. [NAME]. Can I get + to keep your cautious on the other line? He has [NAME]. Yes, no problem. Okay. + Haven''t a plan for what? Okay, but I need it over to me. There''s no point... + Yeah, and one about the electric, because the bills... Yes, it''s a gas and the + electric payment. Just a gas, all about the electric. Because the bill was not + correct. [NAME], okay. I''ve emailed it to her. All right, fine, yeah, perfect. + Once you''ve got all that just emailed across the street. Okay, thank you. They''re + going to get such a shock. Oh dear. All that broken furniture, they threw it in + the bushes. They bought me the cotton. I swear to God, when the landlord was going + through it, yeah? She found it. I wasn''t, I''m not going through the bushes. + Not at all. Sort of that. Wouldn''t even go through my own bushes. Let''s learn + someone else''s. Please speak companies, I''m telling you, they''re just about + the place. They just put me on the whole now. I didn''t even know. I''m using + your butter, is that all right? Can''t you? Do you be joint tenants. Timeer, you''d + be joint tenants. I just need to eat something for me. You''d be joint tenants. + You''d be joint tenants not living together unless you are a couple. + + client: I do apologize about the way I''m just trying to get through to them I + just speak to somebody who''s in the main office and they said they''ve just popped + out for their lunch and so they''ll be back within the next hour I can give him + a call personally if you want and just let him know that you know you didn''t + receive a call yesterday so it has to be today as soon as possible. + + customer: Yeah, I mean the trouble is they told me that yesterday and they didn''t + call back. This is my issue is that, this is my issue that you know, are they + going to call back today? + + client: Yeah, no, I can''t understand. I will have a word for him myself, and + I have them just let the people in the office know if they could just let me know + when he''s in there and I can give him a call personally and just let him know. + I know it has been very busy, but I will make sure I''ll just say if you can give + her a call up right now, because she has been waiting. + + customer: Yeah, it really is urgent. + + client: And he should have, you know, been able to investigate, whatever he had + to do. + + customer: Yeah. Yeah, it really is urgent. Yeah. Okay, all right then. + + client: I''m really sorry about the way. + + customer: Okay, all right then. Okay, all right then, okay. All right then I will + wait. + + client: Thank you so much. + + customer: All right, okay. All right, then I will wait. All right, okay. Should + I, if I don''t hear anything, what, within an hour, should I give a callback or + not? + + client: Yeah that''s completely fine and they said he will be back within the + hour so if it isn''t just after at all then you can always call back but it would + definitely be before five I''ll definitely make sure he calls you myself. + + customer: Yeah, okay. + + client: It''s no worries. + + customer: All right, lovely. Appreciate it. + + client: Thank you so much. + + customer: Thank you. All right. All right, thanks.' +- text: 'client: Thank you for calling [NAME], partners with [ORGANIZATION]. Please + listen to the options carefully so we can get you to the right team. If you need + any help with a drainage problem, press one. If you require our plumbing services, + press two. Or if you have an existing home care policy, it''s fee. Did you know + that you can now book plumbing services online in a few easy clicks? Visit dino.com + for details. Or heavily please hold and your call will be transferred to a customer + service advisor. We record calls for quality and training purposes. + + customer: I did. I spoke to one of your engineers recently due to and being a + family member and I was just saying he said to bring off and see if he''s with + him taking anyone else on. A promen. + + client: Is it plumbing or drainage? I''m going to say the postcode where you are. + + customer: 8th, [LOCATION], 5JG. + + client: Yep. What I''ll do is I''ll put you through to the plumbers that cover + that postcode and they can, you know, you + + customer: Yeah, that''s great. Thank you so much. Oh. + + client: [NAME], your local drains and plumbing experts. Drainage inquiries, press + 1. For plumbing inquiries, press 2. If you are a nosly business owner, press 3, + and for finance, press 4. Thank you for calling [NAME], your local drains and + plumbing experts, for drain inquiries, press 1, for [ORGANIZATION], press 2. If + you are a nosly business owner, press 3, and for finance, press 4. Your call is + very important to us. Please reach the next available agent. Hello, you''re through + the next available agent. + + customer: I''ve recently spoke to a family member who works for yourselves and + he said to bring up to you to take him anyone else on. It''s [NAME]. It''s [NAME]. + He said to bring up to you to take him anyone else on. It''s [NAME]. + + client: and who''s your family member? + + customer: It''s [NAME]. + + client: Oh no! No, we''re not having another one of him here. Right, okay, hang + on. + + customer: Yeah, I have I have. + + client: Have you got a C visa, though? Can you email it to [NAME]? Yeah? Yeah, + [NAME]. + + customer: Yeah, of course, have you got it, please. + + client: Yeah. Yeah, [NAME]. + + customer: Thank you very much. [NAME] [LOCATION]. + + client: Yeah. + + customer: I just felt sorry. + + client: G-I-L-L-I-E-S-P-I-E. + + customer: G-I-L-I-S-G-I-L-I-E-S. G-G-I-L-I-E-S. + + client: Why? + + customer: Yeah. Yeah. + + client: What? Yeah, that''s what I''m supposed for? Right. I-N-G-I-L-L. + + customer: Yeah. + + client: Yeah, E-S-P-I-E-P-I-E-E-P-I-T, right if you got that? Yes, P-I-E, yeah. + Can have my job if you want? + + customer: Right, I''ve got that now. + + client: And it''s at [LOCATION]. Yeah, M-A-C-E. M-E. Doco. [LOCATION]. What''s + your name? Sorry? Elias, all right, no bother. Bye, darling. Bye, bye, bye.' +- text: 'client: Good morning, [NAME]. How can I help? + + customer: Oh, hi yeah, is that [NAME]? + + client: It is indeed. Right, potentially. + + customer: Yeah, I had an engineer out yesterday who kindly put a new central heating + pump in for me. But he''s left a tool here. Looks like I''m speaking as a non-tall + person. Look like mole grips, big pair of yellow mole grips. + + client: Right, potentially what the postcode there? + + customer: I just realized you''d left them on the table after he''s gone, so. + + client: That''s fine. 22 orchard close. + + customer: That''s the one, yes. + + client: Excellent, and that would be you. + + customer: I just thought if someone''s passing at any point, you could pick them + up, but I, yeah. + + client: That would be you in. Now that''s fine. What I do is I''ll speak to my + plumbing department and let them know, and I will get them to get somebody back + there to pick them up for you. + + customer: Yeah, whenever, yeah, I''ll be around, but I just thought I''ve let + you know. + + client: I''ll get them to get somebody back there to pick them up for you. I''ll + get them to get somebody back there to pick them up for you. I''ll get them to + + customer: Lovely. Okay. Thank you. Bye. + + client: No worries to talk. Thanks now. Thanks. Bye bye.' +- text: 'client: Thank you for calling [NAME], partners with [ORGANIZATION]. Please + listen to the options carefully, so we can get you to the right team. If you need + any help with a drainage problem, press one. + + customer: You''re + + client: If you require our plumbing service, press two. Did you know that you + can now book drainage services online in a few easy clicks, visit Dino. For details. + Alternatively, please hold and your call will be transferred to a customer service + advisor. We record calls for quality and training purposes. + + customer: Hi, good morning. I''ve just had a survey done on a property that I + want to buy and they recommend that I have a [ORGANIZATION] drainage report and + I was just wondering do you cover [LOCATION]? + + client: Yes, we do cover everywhere with [ORGANIZATION] surveys, yeah. + + customer: Okay and how much is the cost of a [ORGANIZATION] camera drainage report? + + client: So with the [ORGANIZATION] surveys, we don''t actually book them here + at the core centre, they''re booked directly with our local offices because it''s + the equipment that needs booking, not the person. + + customer: Okay. + + client: We don''t get visibility of the equipment, I''m afraid. And then, so I''ve + never booked one, so I don''t know the price. I do believe, I''m pretty sure a + customer told me they''d had one done and it was around 250 pounds. + + customer: Okay, all right then. So I mean what do we do we book it with a local + office then? + + client: Yeah so I could certainly give you their phone number but just to let + you know they''re not open at the weekends they won''t be open until tomorrow + now. Yeah, it''s [LOCATION] you say. + + customer: I think so long. Yes it is, yeah. + + client: Yeah, let me find the number for you. + + customer: But then group. Okay. How quick is it done? Okay. + + client: Again I couldn''t tell you I''m afraid because I''ve never booked one + like I say and the actual process of doing it again I''ve never been out with + an engineer so I''ve never seen it done I know that each office only has basically + the equipment uses it''s like millions of pounds so each office only has one basically + that''s why it''s the equipment that needs booking you see but yeah let me give + you their phone number have you got a pen + + customer: Okay. Have you. Have you? Yeah. Yeah. Yeah. Yeah. Great. Great. Great. + Great. + + client: Yeah, it''s 0151 545 0913. Yeah, that''s all right, no problem. + + customer: Great. Great. Great. Great. + + client: Take care. Bye. + + customer: Great. Great. Great. Great. Lovely. Thank you very much for help. Thanks + a lot.' +metrics: +- metric +pipeline_tag: text-classification +library_name: setfit +inference: true +base_model: sentence-transformers/paraphrase-mpnet-base-v2 +model-index: +- name: SetFit with sentence-transformers/paraphrase-mpnet-base-v2 + results: + - task: + type: text-classification + name: Text Classification + dataset: + name: Unknown + type: unknown + split: test + metrics: + - type: metric + value: 0.5307449553507265 + name: Metric +--- + +# SetFit with sentence-transformers/paraphrase-mpnet-base-v2 + +This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification. + +The model has been trained using an efficient few-shot learning technique that involves: + +1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning. +2. Training a classification head with features from the fine-tuned Sentence Transformer. + +## Model Details + +### Model Description +- **Model Type:** SetFit +- **Sentence Transformer body:** [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) +- **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance +- **Maximum Sequence Length:** 512 tokens +- **Number of Classes:** 11 classes + + + + +### Model Sources + +- **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit) +- **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055) +- **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit) + +### Model Labels +| Label | Examples | +|:------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 1 | | +| 3 | | +| 9 | | +| 7 | | +| 0 | | +| 6 | | +| 10 | | +| 2 | | +| 5 | | +| 8 | | +| 4 | | + +## Evaluation + +### Metrics +| Label | Metric | +|:--------|:-------| +| **all** | 0.5307 | + +## Uses + +### Direct Use for Inference + +First install the SetFit library: + +```bash +pip install setfit +``` + +Then you can load this model and run inference. + +```python +from setfit import SetFitModel + +# Download from the 🤗 Hub +model = SetFitModel.from_pretrained("automated-analytics/setfit-paraphrase-mpnet") +# Run inference +preds = model("client: Thank you for [NAME], partners with [ORGANIZATION]. Please listen to the options carefully, secure the right team. We record calls for quality and training purposes. Please enter your account number using your telephone keypad, ignoring any letters, then press the hash key. Just use your telephone keypad to enter your account number, then press the hash key. +customer: I'm trying. +client: This may start with it. Okay, just a moment whilst I look that up. I'm afraid that account number doesn't match our records. Please try again. Okay, just a moment whilst I put that up. Please enter your phone number using the telephone teapad. +customer: Thank you. +client: I'm sorry, I didn't understand. Try entering your phone number. I'm sorry, I still didn't understand. Please by entering it again.") +``` + + + + + + + + + +## Training Details + +### Training Set Metrics +| Training set | Min | Median | Max | +|:-------------|:----|:---------|:-----| +| Word count | 38 | 413.8182 | 3760 | + +| Label | Training Sample Count | +|:------|:----------------------| +| 0 | 8 | +| 1 | 8 | +| 2 | 8 | +| 3 | 8 | +| 4 | 8 | +| 5 | 4 | +| 6 | 8 | +| 7 | 8 | +| 8 | 1 | +| 9 | 8 | +| 10 | 8 | + +### Training Hyperparameters +- batch_size: (64, 64) +- num_epochs: (1, 1) +- max_steps: -1 +- sampling_strategy: oversampling +- body_learning_rate: (2e-05, 1e-05) +- head_learning_rate: 0.01 +- loss: CachedMultipleNegativesRankingLoss +- distance_metric: cosine_distance +- margin: 0.25 +- end_to_end: False +- use_amp: False +- warmup_proportion: 0.1 +- l2_weight: 0.01 +- seed: 42 +- eval_max_steps: -1 +- load_best_model_at_end: True + +### Training Results +| Epoch | Step | Training Loss | Validation Loss | +|:------:|:----:|:-------------:|:---------------:| +| 0.0119 | 1 | 4.6669 | - | +| 0.5952 | 50 | 4.0552 | - | +| 1.0 | 84 | - | 4.0622 | + +### Framework Versions +- Python: 3.10.16 +- SetFit: 1.1.1 +- Sentence Transformers: 3.4.1 +- Transformers: 4.49.0 +- PyTorch: 2.2.1+cu121 +- Datasets: 3.3.2 +- Tokenizers: 0.21.0 + +## Citation + +### BibTeX +```bibtex +@article{https://doi.org/10.48550/arxiv.2209.11055, + doi = {10.48550/ARXIV.2209.11055}, + url = {https://arxiv.org/abs/2209.11055}, + author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren}, + keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences}, + title = {Efficient Few-Shot Learning Without Prompts}, + publisher = {arXiv}, + year = {2022}, + copyright = {Creative Commons Attribution 4.0 International} +} +``` + + + + + + \ No newline at end of file