diff --git "a/README.md" "b/README.md" --- "a/README.md" +++ "b/README.md" @@ -189,7 +189,7 @@ model-index: split: test metrics: - type: accuracy - value: 0.009170806266717615 + value: 0.6908674054260604 name: Accuracy --- @@ -221,58 +221,58 @@ The model has been trained using an efficient few-shot learning technique that i - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit) ### Model Labels -| Label | Examples | -|:------|:--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| -| 5 | | -| 17 | | -| 0 | | -| 15 | | -| 29 | | -| 28 | | -| 16 | | -| 40 | | -| 30 | | -| 10 | | -| 37 | | -| 4 | | -| 26 | | -| 20 | | -| 13 | | -| 33 | | -| 7 | | -| 3 | | -| 21 | | -| 32 | | -| 19 | | -| 36 | | -| 42 | | -| 2 | | -| 39 | | -| 27 | | -| 24 | | -| 9 | | -| 8 | | -| 25 | | -| 34 | | -| 23 | | -| 12 | | -| 31 | | -| 38 | | -| 6 | | -| 18 | | -| 14 | | -| 11 | | -| 41 | | -| 22 | | -| 35 | | -| 1 | | +| Label | Examples | +|:------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| 20 | | +| 42 | | +| 19 | | +| 11 | | +| 17 | | +| 14 | | +| 36 | | +| 27 | | +| 9 | | +| 29 | | +| 13 | | +| 7 | | +| 23 | | +| 25 | | +| 37 | | +| 30 | | +| 2 | | +| 0 | | +| 39 | | +| 21 | | +| 18 | | +| 5 | | +| 28 | | +| 38 | | +| 40 | | +| 4 | | +| 6 | | +| 35 | | +| 8 | | +| 12 | | +| 33 | | +| 22 | | +| 3 | | +| 34 | | +| 31 | | +| 41 | | +| 16 | | +| 24 | | +| 10 | | +| 1 | | +| 32 | | +| 26 | | +| 15 | | ## Evaluation ### Metrics | Label | Accuracy | |:--------|:---------| -| **all** | 0.0092 | +| **all** | 0.6909 | ## Uses @@ -324,57 +324,57 @@ preds = model("##rch procedure that evaluates the objective function p x display ### Training Set Metrics | Training set | Min | Median | Max | |:-------------|:----|:---------|:----| -| Word count | 2 | 375.0186 | 509 | +| Word count | 1 | 370.3098 | 509 | | Label | Training Sample Count | |:------|:----------------------| -| 0 | 10 | -| 1 | 10 | -| 2 | 10 | -| 3 | 10 | -| 4 | 10 | -| 5 | 10 | -| 6 | 10 | -| 7 | 10 | -| 8 | 10 | -| 9 | 10 | -| 10 | 10 | -| 11 | 10 | -| 12 | 10 | -| 13 | 10 | -| 14 | 10 | -| 15 | 10 | -| 16 | 10 | -| 17 | 10 | -| 18 | 10 | -| 19 | 10 | -| 20 | 10 | -| 21 | 10 | -| 22 | 10 | -| 23 | 10 | -| 24 | 10 | -| 25 | 10 | -| 26 | 10 | -| 27 | 10 | -| 28 | 10 | -| 29 | 10 | -| 30 | 10 | -| 31 | 10 | -| 32 | 10 | -| 33 | 10 | -| 34 | 10 | -| 35 | 10 | -| 36 | 10 | -| 37 | 10 | -| 38 | 10 | -| 39 | 10 | -| 40 | 10 | -| 41 | 10 | -| 42 | 10 | +| 0 | 50 | +| 1 | 50 | +| 2 | 50 | +| 3 | 50 | +| 4 | 50 | +| 5 | 50 | +| 6 | 50 | +| 7 | 50 | +| 8 | 50 | +| 9 | 50 | +| 10 | 50 | +| 11 | 50 | +| 12 | 50 | +| 13 | 50 | +| 14 | 50 | +| 15 | 50 | +| 16 | 50 | +| 17 | 50 | +| 18 | 50 | +| 19 | 50 | +| 20 | 50 | +| 21 | 50 | +| 22 | 50 | +| 23 | 50 | +| 24 | 50 | +| 25 | 50 | +| 26 | 50 | +| 27 | 50 | +| 28 | 50 | +| 29 | 50 | +| 30 | 50 | +| 31 | 50 | +| 32 | 50 | +| 33 | 50 | +| 34 | 50 | +| 35 | 50 | +| 36 | 50 | +| 37 | 50 | +| 38 | 50 | +| 39 | 50 | +| 40 | 50 | +| 41 | 50 | +| 42 | 50 | ### Training Hyperparameters - batch_size: (16, 16) -- num_epochs: (2, 2) +- num_epochs: (1, 4) - max_steps: -1 - sampling_strategy: oversampling - num_iterations: 10 @@ -383,7 +383,7 @@ preds = model("##rch procedure that evaluates the objective function p x display - loss: CosineSimilarityLoss - distance_metric: cosine_distance - margin: 0.25 -- end_to_end: True +- end_to_end: False - use_amp: False - warmup_proportion: 0.1 - max_length: 512 @@ -392,12 +392,16 @@ preds = model("##rch procedure that evaluates the objective function p x display - load_best_model_at_end: True ### Training Results -| Epoch | Step | Training Loss | Validation Loss | -|:------:|:----:|:-------------:|:---------------:| -| 0.0019 | 1 | 0.2819 | - | -| 0.9294 | 500 | 0.0065 | - | -| 1.8587 | 1000 | 0.0049 | - | - +| Epoch | Step | Training Loss | Validation Loss | +|:----------:|:--------:|:-------------:|:---------------:| +| 0.0004 | 1 | 0.3114 | - | +| 0.1860 | 500 | 0.0379 | - | +| 0.3720 | 1000 | 0.1131 | - | +| 0.5580 | 1500 | 0.0567 | - | +| **0.7440** | **2000** | **0.0168** | **0.1033** | +| 0.9301 | 2500 | 0.0033 | - | + +* The bold row denotes the saved checkpoint. ### Framework Versions - Python: 3.10.12 - SetFit: 1.0.3