Hugging Face's logo

saattrupdan
/
verdict-classifier

Text Classification
Transformers
PyTorch
TensorBoard
Safetensors
xlm-roberta
Generated from Trainer
Model card Files Metrics Community
1
saattrupdan commited on
Commit
c6878df
·
1 Parent(s): de2946a

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +91 -77
README.md CHANGED
@@ -1,40 +1,96 @@
1
  ---
2
  license: mit
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
3
  tags:
4
  - generated_from_trainer
5
  model-index:
6
- - name: verdict-classifier
7
- results: []
 
 
 
 
 
8
  ---
9
 
10
- <!-- This model card has been generated automatically according to the information the Trainer had access to. You
11
- should probably proofread and complete it, then remove this comment. -->
12
 
13
- # verdict-classifier
14
-
15
- This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on the None dataset.
16
- It achieves the following results on the evaluation set:
17
- - Loss: 0.2245
18
- - F1 Macro: 0.8818
19
- - F1 Misinformation: 0.9842
20
- - F1 Factual: 0.9688
21
- - F1 Other: 0.6923
22
- - Prec Macro: 0.8668
23
- - Prec Misinformation: 0.9887
24
- - Prec Factual: 0.9688
25
- - Prec Other: 0.6429
26
-
27
- ## Model description
28
-
29
- More information needed
30
-
31
- ## Intended uses & limitations
32
-
33
- More information needed
34
-
35
- ## Training and evaluation data
36
-
37
- More information needed
38
 
39
  ## Training procedure
40
 
@@ -54,54 +110,12 @@ The following hyperparameters were used during training:
54
 
55
  ### Training results
56
 
57
- | Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Prec Macro | Prec Misinformation | Prec Factual | Prec Other |
58
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:|
59
- | 1.0833 | 0.13 | 500 | 0.9331 | 0.3136 | 0.9408 | 0.0 | 0.0 | 0.2961 | 0.8882 | 0.0 | 0.0 |
60
- | 0.9697 | 0.27 | 1000 | 0.8839 | 0.3136 | 0.9408 | 0.0 | 0.0 | 0.2961 | 0.8882 | 0.0 | 0.0 |
61
- | 0.9406 | 0.4 | 1500 | 0.7238 | 0.3591 | 0.9394 | 0.0 | 0.1379 | 0.4304 | 0.8911 | 0.0 | 0.4 |
62
- | 0.9183 | 0.53 | 2000 | 0.6494 | 0.3482 | 0.9336 | 0.0 | 0.1111 | 0.3528 | 0.8916 | 0.0 | 0.1667 |
63
- | 0.9103 | 0.67 | 2500 | 0.5686 | 0.3621 | 0.9323 | 0.0 | 0.1538 | 0.3643 | 0.8930 | 0.0 | 0.2 |
64
- | 0.8637 | 0.8 | 3000 | 0.5720 | 0.4233 | 0.9367 | 0.0 | 0.3333 | 0.4036 | 0.9108 | 0.0 | 0.3 |
65
- | 0.7905 | 0.93 | 3500 | 0.3918 | 0.4721 | 0.9418 | 0.0 | 0.4746 | 0.4402 | 0.9206 | 0.0 | 0.4 |
66
- | 0.6625 | 1.06 | 4000 | 0.2771 | 0.8225 | 0.9666 | 0.9375 | 0.5634 | 0.7853 | 0.9929 | 0.9375 | 0.4255 |
67
- | 0.5403 | 1.2 | 4500 | 0.2187 | 0.8083 | 0.9703 | 0.9062 | 0.5484 | 0.7799 | 0.9861 | 0.9062 | 0.4474 |
68
- | 0.4837 | 1.33 | 5000 | 0.1708 | 0.8138 | 0.9728 | 0.9231 | 0.5455 | 0.7916 | 0.9817 | 0.9091 | 0.4839 |
69
- | 0.4411 | 1.46 | 5500 | 0.1734 | 0.7873 | 0.9707 | 0.9206 | 0.4706 | 0.7843 | 0.9729 | 0.9355 | 0.4444 |
70
- | 0.3991 | 1.6 | 6000 | 0.1671 | 0.8131 | 0.9705 | 0.9355 | 0.5333 | 0.7976 | 0.9816 | 0.9667 | 0.4444 |
71
- | 0.344 | 1.73 | 6500 | 0.1719 | 0.7989 | 0.9749 | 0.8955 | 0.5263 | 0.7667 | 0.9885 | 0.8571 | 0.4545 |
72
- | 0.3005 | 1.86 | 7000 | 0.1855 | 0.8052 | 0.9704 | 0.9206 | 0.5246 | 0.7839 | 0.9838 | 0.9355 | 0.4324 |
73
- | 0.2638 | 2.0 | 7500 | 0.1802 | 0.7896 | 0.9752 | 0.9231 | 0.4706 | 0.7777 | 0.9796 | 0.9091 | 0.4444 |
74
- | 0.2362 | 2.13 | 8000 | 0.1752 | 0.7762 | 0.9718 | 0.8986 | 0.4583 | 0.7578 | 0.9773 | 0.8378 | 0.4583 |
75
- | 0.2077 | 2.26 | 8500 | 0.1739 | 0.8101 | 0.9740 | 0.9206 | 0.5357 | 0.7953 | 0.9817 | 0.9355 | 0.4688 |
76
- | 0.1858 | 2.39 | 9000 | 0.1986 | 0.8073 | 0.9716 | 0.9412 | 0.5091 | 0.7748 | 0.9839 | 0.8889 | 0.4516 |
77
- | 0.1755 | 2.53 | 9500 | 0.1945 | 0.7872 | 0.9754 | 0.9180 | 0.4681 | 0.8049 | 0.9710 | 0.9655 | 0.4783 |
78
- | 0.1591 | 2.66 | 10000 | 0.2366 | 0.7880 | 0.9692 | 0.8857 | 0.5091 | 0.7504 | 0.9838 | 0.8158 | 0.4516 |
79
- | 0.1457 | 2.79 | 10500 | 0.2346 | 0.7614 | 0.9671 | 0.8857 | 0.4314 | 0.7334 | 0.9771 | 0.8158 | 0.4074 |
80
- | 0.1376 | 2.93 | 11000 | 0.2361 | 0.8015 | 0.9729 | 0.9118 | 0.52 | 0.7802 | 0.9795 | 0.8611 | 0.5 |
81
- | 0.126 | 3.06 | 11500 | 0.2276 | 0.8331 | 0.9751 | 0.9688 | 0.5556 | 0.8168 | 0.9818 | 0.9688 | 0.5 |
82
- | 0.1133 | 3.19 | 12000 | 0.2972 | 0.8014 | 0.9727 | 0.9231 | 0.5085 | 0.7746 | 0.9861 | 0.9091 | 0.4286 |
83
- | 0.1114 | 3.33 | 12500 | 0.2600 | 0.8038 | 0.9705 | 0.8955 | 0.5455 | 0.7742 | 0.9816 | 0.8571 | 0.4839 |
84
- | 0.1099 | 3.46 | 13000 | 0.3221 | 0.8273 | 0.9738 | 0.9118 | 0.5965 | 0.7882 | 0.9884 | 0.8611 | 0.5152 |
85
- | 0.1116 | 3.59 | 13500 | 0.2277 | 0.8376 | 0.9775 | 0.9231 | 0.6122 | 0.8296 | 0.9797 | 0.9091 | 0.6 |
86
- | 0.106 | 3.73 | 14000 | 0.2347 | 0.8148 | 0.9774 | 0.8955 | 0.5714 | 0.7997 | 0.9819 | 0.8571 | 0.56 |
87
- | 0.098 | 3.86 | 14500 | 0.2337 | 0.8487 | 0.9775 | 0.9688 | 0.6 | 0.8418 | 0.9797 | 0.9688 | 0.5769 |
88
- | 0.0899 | 3.99 | 15000 | 0.2072 | 0.8636 | 0.9820 | 0.9688 | 0.64 | 0.8561 | 0.9842 | 0.9688 | 0.6154 |
89
- | 0.0855 | 4.12 | 15500 | 0.2385 | 0.8409 | 0.9762 | 0.9538 | 0.5926 | 0.8189 | 0.9840 | 0.9394 | 0.5333 |
90
- | 0.0864 | 4.26 | 16000 | 0.2780 | 0.8462 | 0.9774 | 0.9688 | 0.5926 | 0.8287 | 0.9841 | 0.9688 | 0.5333 |
91
- | 0.0784 | 4.39 | 16500 | 0.2668 | 0.8277 | 0.9776 | 0.9524 | 0.5532 | 0.8361 | 0.9754 | 0.9677 | 0.5652 |
92
- | 0.0923 | 4.52 | 17000 | 0.2893 | 0.8399 | 0.9738 | 0.9254 | 0.6207 | 0.8012 | 0.9884 | 0.8857 | 0.5294 |
93
- | 0.0794 | 4.66 | 17500 | 0.3101 | 0.8556 | 0.9773 | 0.9688 | 0.6207 | 0.8289 | 0.9885 | 0.9688 | 0.5294 |
94
- | 0.082 | 4.79 | 18000 | 0.2245 | 0.8818 | 0.9842 | 0.9688 | 0.6923 | 0.8668 | 0.9887 | 0.9688 | 0.6429 |
95
- | 0.084 | 4.92 | 18500 | 0.2771 | 0.8247 | 0.9797 | 0.8986 | 0.5957 | 0.8102 | 0.9841 | 0.8378 | 0.6087 |
96
- | 0.0757 | 5.06 | 19000 | 0.2971 | 0.8594 | 0.9773 | 0.9677 | 0.6333 | 0.8388 | 0.9885 | 1.0 | 0.5278 |
97
- | 0.0709 | 5.19 | 19500 | 0.3601 | 0.8410 | 0.9774 | 0.9688 | 0.5769 | 0.8288 | 0.9819 | 0.9688 | 0.5357 |
98
- | 0.0698 | 5.32 | 20000 | 0.2772 | 0.8333 | 0.9762 | 0.9524 | 0.5714 | 0.8173 | 0.9840 | 0.9677 | 0.5 |
99
- | 0.0652 | 5.45 | 20500 | 0.3397 | 0.8186 | 0.9752 | 0.9524 | 0.5283 | 0.8100 | 0.9796 | 0.9677 | 0.4828 |
100
- | 0.0735 | 5.59 | 21000 | 0.3027 | 0.8412 | 0.9785 | 0.9524 | 0.5926 | 0.8284 | 0.9841 | 0.9677 | 0.5333 |
101
- | 0.0746 | 5.72 | 21500 | 0.3122 | 0.8384 | 0.9751 | 0.9688 | 0.5714 | 0.8176 | 0.9840 | 0.9688 | 0.5 |
102
- | 0.0714 | 5.85 | 22000 | 0.2683 | 0.8381 | 0.9787 | 0.9524 | 0.5833 | 0.8429 | 0.9776 | 0.9677 | 0.5833 |
103
- | 0.073 | 5.99 | 22500 | 0.2436 | 0.8676 | 0.9786 | 0.9841 | 0.64 | 0.8650 | 0.9797 | 1.0 | 0.6154 |
104
- | 0.0653 | 6.12 | 23000 | 0.3380 | 0.8559 | 0.9761 | 0.9688 | 0.6230 | 0.8243 | 0.9907 | 0.9688 | 0.5135 |
105
 
106
 
107
  ### Framework versions
@@ -109,4 +123,4 @@ The following hyperparameters were used during training:
109
  - Transformers 4.11.3
110
  - Pytorch 1.9.0+cu102
111
  - Datasets 1.9.0
112
- - Tokenizers 0.10.2
 
1
  ---
2
  license: mit
3
+ language:
4
+ - am
5
+ - ar
6
+ - hy
7
+ - eu
8
+ - bn
9
+ - bs
10
+ - bg
11
+ - my
12
+ - hr
13
+ - ca
14
+ - cs
15
+ - da
16
+ - nl
17
+ - en
18
+ - et
19
+ - fi
20
+ - fr
21
+ - ka
22
+ - de
23
+ - el
24
+ - gu
25
+ - ht
26
+ - iw
27
+ - hi
28
+ - hu
29
+ - is
30
+ - in
31
+ - it
32
+ - ja
33
+ - kn
34
+ - km
35
+ - ko
36
+ - lo
37
+ - lv
38
+ - lt
39
+ - ml
40
+ - mr
41
+ - ne
42
+ - no
43
+ - or
44
+ - pa
45
+ - ps
46
+ - fa
47
+ - pl
48
+ - pt
49
+ - ro
50
+ - ru
51
+ - sr
52
+ - zh
53
+ - sd
54
+ - si
55
+ - sk
56
+ - sl
57
+ - es
58
+ - sv
59
+ - tl
60
+ - ta
61
+ - te
62
+ - th
63
+ - tr
64
+ - uk
65
+ - ur
66
+ - ug
67
+ - vi
68
+ - cy
69
  tags:
70
  - generated_from_trainer
71
  model-index:
72
+ - name: verdict-classifier-en
73
+ results:
74
+ - task:
75
+ type: text-classification
76
+ name: Verdict Classification
77
+ widget:
78
+ - "One might think that this is true, but it's taken out of context."
79
  ---
80
 
81
+ # Multilingual Verdict Classifier
 
82
 
83
+ This model is a fine-tuned version of [xlm-roberta-base](https://huggingface.co/xlm-roberta-base) on 1,500 deduplicated multilingual verdicts from [Google Fact Check Tools API](https://developers.google.com/fact-check/tools/api/reference/rest/v1alpha1/claims/search), translated into 65 languages with the [Google Cloud Translation API](https://cloud.google.com/translate/docs/reference/rest/).
84
+ It achieves the following results on the evaluation set, being 1,000 such verdicts, but here including duplicates to represent the true distribution:
85
+ - Loss: 0.1856
86
+ - F1 Macro: 0.8148
87
+ - F1 Misinformation: 0.9764
88
+ - F1 Factual: 0.9375
89
+ - F1 Other: 0.5306
90
+ - Precision Macro: 0.8117
91
+ - Precision Misinformation: 0.9775
92
+ - Precision Factual: 0.9375
93
+ - Precision Other: 0.52
 
 
 
 
 
 
 
 
 
 
 
 
 
 
94
 
95
  ## Training procedure
96
 
 
110
 
111
  ### Training results
112
 
113
+ | Training Loss | Epoch | Step | Validation Loss | F1 Macro | F1 Misinformation | F1 Factual | F1 Other | Precision Macro | Precision Misinformation | Precision Factual | Precision Other |
114
  |:-------------:|:-----:|:-----:|:---------------:|:--------:|:-----------------:|:----------:|:--------:|:----------:|:-------------------:|:------------:|:----------:|
115
+ | 0.8707 | 1.0 | 3758 | 0.2414 | 0.7832 | 0.9639 | 0.7857 | 0.6 | 0.7950 | 0.9683 | 0.9167 | 0.5 |
116
+ | 0.3918 | 2.0 | 7516 | 0.1856 | 0.8148 | 0.9764 | 0.9375 | 0.5306 | 0.8117 | 0.9775 | 0.9375 | 0.52 |
117
+ | 0.1766 | 3.0 | 11274 | 0.1942 | 0.8394 | 0.9809 | 0.9538 | 0.5833 | 0.8349 | 0.9820 | 0.9394 | 0.5833 |
118
+ | 0.1071 | 4.0 | 15032 | 0.2078 | 0.8676 | 0.9786 | 0.9841 | 0.64 | 0.8650 | 0.9797 | 1.0 | 0.6154 |
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
119
 
120
 
121
  ### Framework versions
 
123
  - Transformers 4.11.3
124
  - Pytorch 1.9.0+cu102
125
  - Datasets 1.9.0
126
+ - Tokenizers 0.10.2