Hugging Face's logo

saattrupdan
/
verdict-classifier

Text Classification
Transformers
PyTorch
TensorBoard
Safetensors
xlm-roberta
Generated from Trainer
Model card Files Metrics Community
1
saattrupdan commited on
Commit
de2946a
·
1 Parent(s): 7d696c7

update model card README.md

Browse files
Files changed (1) hide show
  1. README.md +76 -90
README.md CHANGED
@@ -1,96 +1,40 @@
1
  ---
2
  license: mit
3
- language:
4
- - am
5
- - ar
6
- - hy
7
- - eu
8
- - bn
9
- - bs
10
- - bg
11
- - my
12
- - hr
13
- - ca
14
- - cs
15
- - da
16
- - nl
17
- - en
18
- - et
19
- - fi
20
- - fr
21
- - ka
22
- - de
23
- - el
24
- - gu
25
- - ht
26
- - iw
27
- - hi
28
- - hu
29
- - is
30
- - in
31
- - it
32
- - ja
33
- - kn
34
- - km
35
- - ko
36
- - lo
37
- - lv
38
- - lt
39
- - ml
40
- - mr
41
- - ne
42
- - no
43
- - or
44
- - pa
45
- - ps
46
- - fa
47
- - pl
48
- - pt
49
- - ro
50
- - ru
51
- - sr
52
- - zh
53
- - sd
54
- - si
55
- - sk
56
- - sl
57
- - es
58
- - sv
59
- - tl
60
- - ta
61
- - te
62
- - th
63
- - tr
64
- - uk
65
- - ur
66
- - ug
67
- - vi
68
- - cy
69
  tags:
70
  - generated_from_trainer
71
  model-index:
72
- - name: verdict-classifier-en
73
- results:
74
- - task:
75
- type: text-classification
76
- name: Verdict Classification
77
- widget:
78
- - "One might think that this is true, but it's taken out of context."
79
  ---
80
 
81
- # Multilingual Verdict Classifier
 
82
 
83
- This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on 1,500 deduplicated multilingual verdicts from [Google Fact Check Tools API](https://developers.google.com/fact-check/tools/api/reference/rest/v1alpha1/claims/search), translated into 65 languages with the [Google Cloud Translation API](https://cloud.google.com/translate/docs/reference/rest/).
84
- It achieves the following results on the evaluation set, being 1,000 such verdicts, but here including duplicates to represent the true distribution:
85
- - Loss: 0.1856
86
- - F1 Macro: 0.8148
87
- - F1 Misinformation: 0.9764
88
- - F1 Factual: 0.9375
89
- - F1 Other: 0.5306
90
- - Precision Macro: 0.8117
91
- - Precision Misinformation: 0.9775
92
- - Precision Factual: 0.9375
93
- - Precision Other: 0.52
 
 
 
 
 
 
 
 
 
 
 
 
 
 
94
 
95
  ## Training procedure
96
 
@@ -110,12 +54,54 @@ The following hyperparameters were used during training:
110
 
111
  ### Training results
112
 
113
- | Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Precision Macro | Precision Misinformation | Precision Factual | Precision Other |
114
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:|
115
- | 0.8707 | 1.0 | 3758 | 0.2414 | 0.7832 | 0.9639 | 0.7857 | 0.6 | 0.7950 | 0.9683 | 0.9167 | 0.5 |
116
- | 0.3918 | 2.0 | 7516 | 0.1856 | 0.8148 | 0.9764 | 0.9375 | 0.5306 | 0.8117 | 0.9775 | 0.9375 | 0.52 |
117
- | 0.1766 | 3.0 | 11274 | 0.1942 | 0.8394 | 0.9809 | 0.9538 | 0.5833 | 0.8349 | 0.9820 | 0.9394 | 0.5833 |
118
- | 0.1071 | 4.0 | 15032 | 0.2078 | 0.8676 | 0.9786 | 0.9841 | 0.64 | 0.8650 | 0.9797 | 1.0 | 0.6154 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
119
 
120
 
121
  ### Framework versions
 
1
  ---
2
  license: mit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  tags:
4
  - generated_from_trainer
5
  model-index:
6
+ - name: verdict-classifier
7
+ results: []
 
 
 
 
 
8
  ---
9
 
10
+ <!-- This model card has been generated automatically according to the information the Trainer had access to. You
11
+ should probably proofread and complete it, then remove this comment. -->
12
 
13
+ # verdict-classifier
14
+
15
+ This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the None dataset.
16
+ It achieves the following results on the evaluation set:
17
+ - Loss: 0.2245
18
+ - F1 Macro: 0.8818
19
+ - F1 Misinformation: 0.9842
20
+ - F1 Factual: 0.9688
21
+ - F1 Other: 0.6923
22
+ - Prec Macro: 0.8668
23
+ - Prec Misinformation: 0.9887
24
+ - Prec Factual: 0.9688
25
+ - Prec Other: 0.6429
26
+
27
+ ## Model description
28
+
29
+ More information needed
30
+
31
+ ## Intended uses & limitations
32
+
33
+ More information needed
34
+
35
+ ## Training and evaluation data
36
+
37
+ More information needed
38
 
39
  ## Training procedure
40
 
 
54
 
55
  ### Training results
56
 
57
+ | Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Prec Macro | Prec Misinformation | Prec Factual | Prec Other |
58
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:|
59
+ | 1.0833 | 0.13 | 500 | 0.9331 | 0.3136 | 0.9408 | 0.0 | 0.0 | 0.2961 | 0.8882 | 0.0 | 0.0 |
60
+ | 0.9697 | 0.27 | 1000 | 0.8839 | 0.3136 | 0.9408 | 0.0 | 0.0 | 0.2961 | 0.8882 | 0.0 | 0.0 |
61
+ | 0.9406 | 0.4 | 1500 | 0.7238 | 0.3591 | 0.9394 | 0.0 | 0.1379 | 0.4304 | 0.8911 | 0.0 | 0.4 |
62
+ | 0.9183 | 0.53 | 2000 | 0.6494 | 0.3482 | 0.9336 | 0.0 | 0.1111 | 0.3528 | 0.8916 | 0.0 | 0.1667 |
63
+ | 0.9103 | 0.67 | 2500 | 0.5686 | 0.3621 | 0.9323 | 0.0 | 0.1538 | 0.3643 | 0.8930 | 0.0 | 0.2 |
64
+ | 0.8637 | 0.8 | 3000 | 0.5720 | 0.4233 | 0.9367 | 0.0 | 0.3333 | 0.4036 | 0.9108 | 0.0 | 0.3 |
65
+ | 0.7905 | 0.93 | 3500 | 0.3918 | 0.4721 | 0.9418 | 0.0 | 0.4746 | 0.4402 | 0.9206 | 0.0 | 0.4 |
66
+ | 0.6625 | 1.06 | 4000 | 0.2771 | 0.8225 | 0.9666 | 0.9375 | 0.5634 | 0.7853 | 0.9929 | 0.9375 | 0.4255 |
67
+ | 0.5403 | 1.2 | 4500 | 0.2187 | 0.8083 | 0.9703 | 0.9062 | 0.5484 | 0.7799 | 0.9861 | 0.9062 | 0.4474 |
68
+ | 0.4837 | 1.33 | 5000 | 0.1708 | 0.8138 | 0.9728 | 0.9231 | 0.5455 | 0.7916 | 0.9817 | 0.9091 | 0.4839 |
69
+ | 0.4411 | 1.46 | 5500 | 0.1734 | 0.7873 | 0.9707 | 0.9206 | 0.4706 | 0.7843 | 0.9729 | 0.9355 | 0.4444 |
70
+ | 0.3991 | 1.6 | 6000 | 0.1671 | 0.8131 | 0.9705 | 0.9355 | 0.5333 | 0.7976 | 0.9816 | 0.9667 | 0.4444 |
71
+ | 0.344 | 1.73 | 6500 | 0.1719 | 0.7989 | 0.9749 | 0.8955 | 0.5263 | 0.7667 | 0.9885 | 0.8571 | 0.4545 |
72
+ | 0.3005 | 1.86 | 7000 | 0.1855 | 0.8052 | 0.9704 | 0.9206 | 0.5246 | 0.7839 | 0.9838 | 0.9355 | 0.4324 |
73
+ | 0.2638 | 2.0 | 7500 | 0.1802 | 0.7896 | 0.9752 | 0.9231 | 0.4706 | 0.7777 | 0.9796 | 0.9091 | 0.4444 |
74
+ | 0.2362 | 2.13 | 8000 | 0.1752 | 0.7762 | 0.9718 | 0.8986 | 0.4583 | 0.7578 | 0.9773 | 0.8378 | 0.4583 |
75
+ | 0.2077 | 2.26 | 8500 | 0.1739 | 0.8101 | 0.9740 | 0.9206 | 0.5357 | 0.7953 | 0.9817 | 0.9355 | 0.4688 |
76
+ | 0.1858 | 2.39 | 9000 | 0.1986 | 0.8073 | 0.9716 | 0.9412 | 0.5091 | 0.7748 | 0.9839 | 0.8889 | 0.4516 |
77
+ | 0.1755 | 2.53 | 9500 | 0.1945 | 0.7872 | 0.9754 | 0.9180 | 0.4681 | 0.8049 | 0.9710 | 0.9655 | 0.4783 |
78
+ | 0.1591 | 2.66 | 10000 | 0.2366 | 0.7880 | 0.9692 | 0.8857 | 0.5091 | 0.7504 | 0.9838 | 0.8158 | 0.4516 |
79
+ | 0.1457 | 2.79 | 10500 | 0.2346 | 0.7614 | 0.9671 | 0.8857 | 0.4314 | 0.7334 | 0.9771 | 0.8158 | 0.4074 |
80
+ | 0.1376 | 2.93 | 11000 | 0.2361 | 0.8015 | 0.9729 | 0.9118 | 0.52 | 0.7802 | 0.9795 | 0.8611 | 0.5 |
81
+ | 0.126 | 3.06 | 11500 | 0.2276 | 0.8331 | 0.9751 | 0.9688 | 0.5556 | 0.8168 | 0.9818 | 0.9688 | 0.5 |
82
+ | 0.1133 | 3.19 | 12000 | 0.2972 | 0.8014 | 0.9727 | 0.9231 | 0.5085 | 0.7746 | 0.9861 | 0.9091 | 0.4286 |
83
+ | 0.1114 | 3.33 | 12500 | 0.2600 | 0.8038 | 0.9705 | 0.8955 | 0.5455 | 0.7742 | 0.9816 | 0.8571 | 0.4839 |
84
+ | 0.1099 | 3.46 | 13000 | 0.3221 | 0.8273 | 0.9738 | 0.9118 | 0.5965 | 0.7882 | 0.9884 | 0.8611 | 0.5152 |
85
+ | 0.1116 | 3.59 | 13500 | 0.2277 | 0.8376 | 0.9775 | 0.9231 | 0.6122 | 0.8296 | 0.9797 | 0.9091 | 0.6 |
86
+ | 0.106 | 3.73 | 14000 | 0.2347 | 0.8148 | 0.9774 | 0.8955 | 0.5714 | 0.7997 | 0.9819 | 0.8571 | 0.56 |
87
+ | 0.098 | 3.86 | 14500 | 0.2337 | 0.8487 | 0.9775 | 0.9688 | 0.6 | 0.8418 | 0.9797 | 0.9688 | 0.5769 |
88
+ | 0.0899 | 3.99 | 15000 | 0.2072 | 0.8636 | 0.9820 | 0.9688 | 0.64 | 0.8561 | 0.9842 | 0.9688 | 0.6154 |
89
+ | 0.0855 | 4.12 | 15500 | 0.2385 | 0.8409 | 0.9762 | 0.9538 | 0.5926 | 0.8189 | 0.9840 | 0.9394 | 0.5333 |
90
+ | 0.0864 | 4.26 | 16000 | 0.2780 | 0.8462 | 0.9774 | 0.9688 | 0.5926 | 0.8287 | 0.9841 | 0.9688 | 0.5333 |
91
+ | 0.0784 | 4.39 | 16500 | 0.2668 | 0.8277 | 0.9776 | 0.9524 | 0.5532 | 0.8361 | 0.9754 | 0.9677 | 0.5652 |
92
+ | 0.0923 | 4.52 | 17000 | 0.2893 | 0.8399 | 0.9738 | 0.9254 | 0.6207 | 0.8012 | 0.9884 | 0.8857 | 0.5294 |
93
+ | 0.0794 | 4.66 | 17500 | 0.3101 | 0.8556 | 0.9773 | 0.9688 | 0.6207 | 0.8289 | 0.9885 | 0.9688 | 0.5294 |
94
+ | 0.082 | 4.79 | 18000 | 0.2245 | 0.8818 | 0.9842 | 0.9688 | 0.6923 | 0.8668 | 0.9887 | 0.9688 | 0.6429 |
95
+ | 0.084 | 4.92 | 18500 | 0.2771 | 0.8247 | 0.9797 | 0.8986 | 0.5957 | 0.8102 | 0.9841 | 0.8378 | 0.6087 |
96
+ | 0.0757 | 5.06 | 19000 | 0.2971 | 0.8594 | 0.9773 | 0.9677 | 0.6333 | 0.8388 | 0.9885 | 1.0 | 0.5278 |
97
+ | 0.0709 | 5.19 | 19500 | 0.3601 | 0.8410 | 0.9774 | 0.9688 | 0.5769 | 0.8288 | 0.9819 | 0.9688 | 0.5357 |
98
+ | 0.0698 | 5.32 | 20000 | 0.2772 | 0.8333 | 0.9762 | 0.9524 | 0.5714 | 0.8173 | 0.9840 | 0.9677 | 0.5 |
99
+ | 0.0652 | 5.45 | 20500 | 0.3397 | 0.8186 | 0.9752 | 0.9524 | 0.5283 | 0.8100 | 0.9796 | 0.9677 | 0.4828 |
100
+ | 0.0735 | 5.59 | 21000 | 0.3027 | 0.8412 | 0.9785 | 0.9524 | 0.5926 | 0.8284 | 0.9841 | 0.9677 | 0.5333 |
101
+ | 0.0746 | 5.72 | 21500 | 0.3122 | 0.8384 | 0.9751 | 0.9688 | 0.5714 | 0.8176 | 0.9840 | 0.9688 | 0.5 |
102
+ | 0.0714 | 5.85 | 22000 | 0.2683 | 0.8381 | 0.9787 | 0.9524 | 0.5833 | 0.8429 | 0.9776 | 0.9677 | 0.5833 |
103
+ | 0.073 | 5.99 | 22500 | 0.2436 | 0.8676 | 0.9786 | 0.9841 | 0.64 | 0.8650 | 0.9797 | 1.0 | 0.6154 |
104
+ | 0.0653 | 6.12 | 23000 | 0.3380 | 0.8559 | 0.9761 | 0.9688 | 0.6230 | 0.8243 | 0.9907 | 0.9688 | 0.5135 |
105
 
106
 
107
  ### Framework versions