--- tags: - setfit - sentence-transformers - text-classification - generated_from_setfit_trainer widget: - text: 'client: Thank you for [NAME], partners with [ORGANIZATION]. Please listen to the options carefully, secure the right team. We record calls for quality and training purposes. Please enter your account number using your telephone keypad, ignoring any letters, then press the hash key. Just use your telephone keypad to enter your account number, then press the hash key. customer: I''m trying. client: This may start with it. Okay, just a moment whilst I look that up. I''m afraid that account number doesn''t match our records. Please try again. Okay, just a moment whilst I put that up. Please enter your phone number using the telephone teapad. customer: Thank you. client: I''m sorry, I didn''t understand. Try entering your phone number. I''m sorry, I still didn''t understand. Please by entering it again.' - text: 'client: Thank you for calling Dino Plumbing. Please note all our calls are recorded for monitoring and training passes. Press one for an update on parts. Press two for an update on your appointment. Press three to discuss an ongoing complaint. Press four for sales. Thank you for calling Dino Plumbing. Please note all our calls are recorded for monitoring and training purposes. Press one for an update on parts. Press two for an update on your appointment. Press three to discuss an ongoing complaint and press four for sales. Thank you for calling dino plumbing. customer: and get some rolls. client: Your call is very important to us. Your call is very important to us. All of our agents are currently busy assisting other customers. Please continue to hold and the next available agent will take your call. Alternatively, you can email. Good afternoon if you''re starting and I can help. customer: Oh, hello. I called yesterday regarding a plumbing issue, a drainage issue at one of our properties. He took all of my details and said that we were going to call us back, but I never got a call back. client: I''m not sure postcode or have a little look for you to apologize. customer: The post code of the property, NW6, NW6, 1 UJ. client: Yes, please. No problem. And it''s the door number 46? 48, okay. customer: 48, which in [LOCATION]. But maybe, I can''t remember if it was a charge of [LOCATION]. Yes, yes, yes, yes, yes. client: It was for a chargeable job, wasn''t it? customer: All right, lovely, thank you. Yes, it''s a charge call job, yes, yes. client: No problem. Let me just see if a gentleman''s available. Give me two seconds. We''ll go straight through if it does connect. Thank you. customer: I can''t do everything? It''s [LOCATION], hi [NAME]. [NAME]. Can I get to keep your cautious on the other line? He has [NAME]. Yes, no problem. Okay. Haven''t a plan for what? Okay, but I need it over to me. There''s no point... Yeah, and one about the electric, because the bills... Yes, it''s a gas and the electric payment. Just a gas, all about the electric. Because the bill was not correct. [NAME], okay. I''ve emailed it to her. All right, fine, yeah, perfect. Once you''ve got all that just emailed across the street. Okay, thank you. They''re going to get such a shock. Oh dear. All that broken furniture, they threw it in the bushes. They bought me the cotton. I swear to God, when the landlord was going through it, yeah? She found it. I wasn''t, I''m not going through the bushes. Not at all. Sort of that. Wouldn''t even go through my own bushes. Let''s learn someone else''s. Please speak companies, I''m telling you, they''re just about the place. They just put me on the whole now. I didn''t even know. I''m using your butter, is that all right? Can''t you? Do you be joint tenants. Timeer, you''d be joint tenants. I just need to eat something for me. You''d be joint tenants. You''d be joint tenants not living together unless you are a couple. client: I do apologize about the way I''m just trying to get through to them I just speak to somebody who''s in the main office and they said they''ve just popped out for their lunch and so they''ll be back within the next hour I can give him a call personally if you want and just let him know that you know you didn''t receive a call yesterday so it has to be today as soon as possible. customer: Yeah, I mean the trouble is they told me that yesterday and they didn''t call back. This is my issue is that, this is my issue that you know, are they going to call back today? client: Yeah, no, I can''t understand. I will have a word for him myself, and I have them just let the people in the office know if they could just let me know when he''s in there and I can give him a call personally and just let him know. I know it has been very busy, but I will make sure I''ll just say if you can give her a call up right now, because she has been waiting. customer: Yeah, it really is urgent. client: And he should have, you know, been able to investigate, whatever he had to do. customer: Yeah. Yeah, it really is urgent. Yeah. Okay, all right then. client: I''m really sorry about the way. customer: Okay, all right then. Okay, all right then, okay. All right then I will wait. client: Thank you so much. customer: All right, okay. All right, then I will wait. All right, okay. Should I, if I don''t hear anything, what, within an hour, should I give a callback or not? client: Yeah that''s completely fine and they said he will be back within the hour so if it isn''t just after at all then you can always call back but it would definitely be before five I''ll definitely make sure he calls you myself. customer: Yeah, okay. client: It''s no worries. customer: All right, lovely. Appreciate it. client: Thank you so much. customer: Thank you. All right. All right, thanks.' - text: 'client: Thank you for calling [NAME], partners with [ORGANIZATION]. Please listen to the options carefully so we can get you to the right team. If you need any help with a drainage problem, press one. If you require our plumbing services, press two. Or if you have an existing home care policy, it''s fee. Did you know that you can now book plumbing services online in a few easy clicks? Visit dino.com for details. Or heavily please hold and your call will be transferred to a customer service advisor. We record calls for quality and training purposes. customer: I did. I spoke to one of your engineers recently due to and being a family member and I was just saying he said to bring off and see if he''s with him taking anyone else on. A promen. client: Is it plumbing or drainage? I''m going to say the postcode where you are. customer: 8th, [LOCATION], 5JG. client: Yep. What I''ll do is I''ll put you through to the plumbers that cover that postcode and they can, you know, you customer: Yeah, that''s great. Thank you so much. Oh. client: [NAME], your local drains and plumbing experts. Drainage inquiries, press 1. For plumbing inquiries, press 2. If you are a nosly business owner, press 3, and for finance, press 4. Thank you for calling [NAME], your local drains and plumbing experts, for drain inquiries, press 1, for [ORGANIZATION], press 2. If you are a nosly business owner, press 3, and for finance, press 4. Your call is very important to us. Please reach the next available agent. Hello, you''re through the next available agent. customer: I''ve recently spoke to a family member who works for yourselves and he said to bring up to you to take him anyone else on. It''s [NAME]. It''s [NAME]. He said to bring up to you to take him anyone else on. It''s [NAME]. client: and who''s your family member? customer: It''s [NAME]. client: Oh no! No, we''re not having another one of him here. Right, okay, hang on. customer: Yeah, I have I have. client: Have you got a C visa, though? Can you email it to [NAME]? Yeah? Yeah, [NAME]. customer: Yeah, of course, have you got it, please. client: Yeah. Yeah, [NAME]. customer: Thank you very much. [NAME] [LOCATION]. client: Yeah. customer: I just felt sorry. client: G-I-L-L-I-E-S-P-I-E. customer: G-I-L-I-S-G-I-L-I-E-S. G-G-I-L-I-E-S. client: Why? customer: Yeah. Yeah. client: What? Yeah, that''s what I''m supposed for? Right. I-N-G-I-L-L. customer: Yeah. client: Yeah, E-S-P-I-E-P-I-E-E-P-I-T, right if you got that? Yes, P-I-E, yeah. Can have my job if you want? customer: Right, I''ve got that now. client: And it''s at [LOCATION]. Yeah, M-A-C-E. M-E. Doco. [LOCATION]. What''s your name? Sorry? Elias, all right, no bother. Bye, darling. Bye, bye, bye.' - text: 'client: Good morning, [NAME]. How can I help? customer: Oh, hi yeah, is that [NAME]? client: It is indeed. Right, potentially. customer: Yeah, I had an engineer out yesterday who kindly put a new central heating pump in for me. But he''s left a tool here. Looks like I''m speaking as a non-tall person. Look like mole grips, big pair of yellow mole grips. client: Right, potentially what the postcode there? customer: I just realized you''d left them on the table after he''s gone, so. client: That''s fine. 22 orchard close. customer: That''s the one, yes. client: Excellent, and that would be you. customer: I just thought if someone''s passing at any point, you could pick them up, but I, yeah. client: That would be you in. Now that''s fine. What I do is I''ll speak to my plumbing department and let them know, and I will get them to get somebody back there to pick them up for you. customer: Yeah, whenever, yeah, I''ll be around, but I just thought I''ve let you know. client: I''ll get them to get somebody back there to pick them up for you. I''ll get them to get somebody back there to pick them up for you. I''ll get them to customer: Lovely. Okay. Thank you. Bye. client: No worries to talk. Thanks now. Thanks. Bye bye.' - text: 'client: Thank you for calling [NAME], partners with [ORGANIZATION]. Please listen to the options carefully, so we can get you to the right team. If you need any help with a drainage problem, press one. customer: You''re client: If you require our plumbing service, press two. Did you know that you can now book drainage services online in a few easy clicks, visit Dino. For details. Alternatively, please hold and your call will be transferred to a customer service advisor. We record calls for quality and training purposes. customer: Hi, good morning. I''ve just had a survey done on a property that I want to buy and they recommend that I have a [ORGANIZATION] drainage report and I was just wondering do you cover [LOCATION]? client: Yes, we do cover everywhere with [ORGANIZATION] surveys, yeah. customer: Okay and how much is the cost of a [ORGANIZATION] camera drainage report? client: So with the [ORGANIZATION] surveys, we don''t actually book them here at the core centre, they''re booked directly with our local offices because it''s the equipment that needs booking, not the person. customer: Okay. client: We don''t get visibility of the equipment, I''m afraid. And then, so I''ve never booked one, so I don''t know the price. I do believe, I''m pretty sure a customer told me they''d had one done and it was around 250 pounds. customer: Okay, all right then. So I mean what do we do we book it with a local office then? client: Yeah so I could certainly give you their phone number but just to let you know they''re not open at the weekends they won''t be open until tomorrow now. Yeah, it''s [LOCATION] you say. customer: I think so long. Yes it is, yeah. client: Yeah, let me find the number for you. customer: But then group. Okay. How quick is it done? Okay. client: Again I couldn''t tell you I''m afraid because I''ve never booked one like I say and the actual process of doing it again I''ve never been out with an engineer so I''ve never seen it done I know that each office only has basically the equipment uses it''s like millions of pounds so each office only has one basically that''s why it''s the equipment that needs booking you see but yeah let me give you their phone number have you got a pen customer: Okay. Have you. Have you? Yeah. Yeah. Yeah. Yeah. Great. Great. Great. Great. client: Yeah, it''s 0151 545 0913. Yeah, that''s all right, no problem. customer: Great. Great. Great. Great. client: Take care. Bye. customer: Great. Great. Great. Great. Lovely. Thank you very much for help. Thanks a lot.' metrics: - metric pipeline_tag: text-classification library_name: setfit inference: true base_model: sentence-transformers/paraphrase-mpnet-base-v2 model-index: - name: SetFit with sentence-transformers/paraphrase-mpnet-base-v2 results: - task: type: text-classification name: Text Classification dataset: name: Unknown type: unknown split: test metrics: - type: metric value: 0.5307449553507265 name: Metric --- # SetFit with sentence-transformers/paraphrase-mpnet-base-v2 This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) as the Sentence Transformer embedding model. A [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance is used for classification. The model has been trained using an efficient few-shot learning technique that involves: 1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning. 2. Training a classification head with features from the fine-tuned Sentence Transformer. ## Model Details ### Model Description - **Model Type:** SetFit - **Sentence Transformer body:** [sentence-transformers/paraphrase-mpnet-base-v2](https://huggingface.co/sentence-transformers/paraphrase-mpnet-base-v2) - **Classification head:** a [LogisticRegression](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html) instance - **Maximum Sequence Length:** 512 tokens - **Number of Classes:** 11 classes ### Model Sources - **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit) - **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055) - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit) ### Model Labels | Label | Examples | |:------|:-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| | 1 | | | 3 | | | 9 | | | 7 | | | 0 | | | 6 | | | 10 | | | 2 | | | 5 | | | 8 | | | 4 | | ## Evaluation ### Metrics | Label | Metric | |:--------|:-------| | **all** | 0.5307 | ## Uses ### Direct Use for Inference First install the SetFit library: ```bash pip install setfit ``` Then you can load this model and run inference. ```python from setfit import SetFitModel # Download from the 🤗 Hub model = SetFitModel.from_pretrained("automated-analytics/setfit-paraphrase-mpnet") # Run inference preds = model("client: Thank you for [NAME], partners with [ORGANIZATION]. Please listen to the options carefully, secure the right team. We record calls for quality and training purposes. Please enter your account number using your telephone keypad, ignoring any letters, then press the hash key. Just use your telephone keypad to enter your account number, then press the hash key. customer: I'm trying. client: This may start with it. Okay, just a moment whilst I look that up. I'm afraid that account number doesn't match our records. Please try again. Okay, just a moment whilst I put that up. Please enter your phone number using the telephone teapad. customer: Thank you. client: I'm sorry, I didn't understand. Try entering your phone number. I'm sorry, I still didn't understand. Please by entering it again.") ``` ## Training Details ### Training Set Metrics | Training set | Min | Median | Max | |:-------------|:----|:---------|:-----| | Word count | 38 | 413.8182 | 3760 | | Label | Training Sample Count | |:------|:----------------------| | 0 | 8 | | 1 | 8 | | 2 | 8 | | 3 | 8 | | 4 | 8 | | 5 | 4 | | 6 | 8 | | 7 | 8 | | 8 | 1 | | 9 | 8 | | 10 | 8 | ### Training Hyperparameters - batch_size: (64, 64) - num_epochs: (1, 1) - max_steps: -1 - sampling_strategy: oversampling - body_learning_rate: (2e-05, 1e-05) - head_learning_rate: 0.01 - loss: CachedMultipleNegativesRankingLoss - distance_metric: cosine_distance - margin: 0.25 - end_to_end: False - use_amp: False - warmup_proportion: 0.1 - l2_weight: 0.01 - seed: 42 - eval_max_steps: -1 - load_best_model_at_end: True ### Training Results | Epoch | Step | Training Loss | Validation Loss | |:------:|:----:|:-------------:|:---------------:| | 0.0119 | 1 | 4.6669 | - | | 0.5952 | 50 | 4.0552 | - | | 1.0 | 84 | - | 4.0622 | ### Framework Versions - Python: 3.10.16 - SetFit: 1.1.1 - Sentence Transformers: 3.4.1 - Transformers: 4.49.0 - PyTorch: 2.2.1+cu121 - Datasets: 3.3.2 - Tokenizers: 0.21.0 ## Citation ### BibTeX ```bibtex @article{https://doi.org/10.48550/arxiv.2209.11055, doi = {10.48550/ARXIV.2209.11055}, url = {https://arxiv.org/abs/2209.11055}, author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren}, keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences}, title = {Efficient Few-Shot Learning Without Prompts}, publisher = {arXiv}, year = {2022}, copyright = {Creative Commons Attribution 4.0 International} } ```