Hugging Face's logo

tokyotech-llm
/
Swallow-MS-7b-instruct-v0.1

Text Generation
Transformers
Safetensors
English
Japanese
mistral
conversational
text-generation-inference
Model card Files Community
Taishi-N324 commited on
Commit
d98656e
·
verified ·
1 Parent(s): d9432ff

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +1 -61
README.md CHANGED
@@ -15,7 +15,7 @@ Our Swallow-MS-7b-v0.1 model has undergone continuous pre-training from the Mist
15
  # Model Release Updates
16
 
17
  We are excited to share the release schedule for our latest models:
18
- - **April 25, 2024**: Released the [Swallow-MS-7b-instruct-v1.0](https://huggingface.co/tokyotech-llm/Swallow-MS-7b-instruct-v1.0).
19
  - **March 11, 2024**: Released the [Swallow-MS-7b-v0.1](https://huggingface.co/tokyotech-llm/Swallow-MS-7b-v0.1)
20
  ![logo](./logo.png)
21
 
@@ -36,66 +36,6 @@ TODO
36
 
37
  ## Base Model Performance
38
 
39
- ### Japanese tasks
40
- |Model|Size|JCommonsenseQA|JEMHopQA|NIILC|JSQuAD|XL-Sum|MGSM|WMT20-en-ja|WMT20-ja-en|Average|
41
- |---------------------------|-------|---------|-------|-------|-------|------|------------|------------|------|-----|
42
- | | |4-shot|4-shot|4-shot|4-shot|1-shot|4-shot|4-shot|4-shot||
43
- | CyberAgentLM2-7B |7B| 0.2198 | 0.5047 | 0.5066 | 0.7799 | 0.0233 | 0.0600 | 0.2345 | 0.1499 | 0.3098 |
44
- | Llama 2 |7B| 0.3852 | 0.4240 | 0.3410 | 0.7917 | 0.1905 | 0.0760 | 0.1783 | 0.1738 | 0.3201 |
45
- | japanese-stablelm-base-beta-7b|7B| 0.3610 | 0.4478 | 0.4432 | 0.8318 | 0.2195 | 0.0720 | 0.1946 | 0.1226 | 0.3366 |
46
- | japanese-stablelm-base-ja_vocab-beta-7b|7B| 0.2172 | 0.4482 | 0.4309 | 0.8202 | 0.0757 | 0.0520 | 0.1601 | 0.1453 | 0.2937 |
47
- | ELYZA-japanese-Llama-2-7b|7B| 0.5791 | 0.4703 | 0.4019 | 0.8226 | 0.1312 | 0.0600 | 0.1795 | 0.1289 | 0.3467 |
48
- | ELYZA-japanese-Llama-2-7b-fast|7B| 0.5308 | 0.4330 | 0.3898 | 0.8131 | 0.1289 | 0.0720 | 0.1678 | 0.1143 | 0.3312 |
49
- | youri-7b (base) |7B| 0.4620 | 0.4776 | 0.4999 | 0.8506 | 0.1957 | 0.0640 | 0.2671 | **0.1971** | 0.3768 |
50
- | Swallow-7b |7B| 0.4808 | 0.5078 | 0.5968 | 0.8573 | 0.1830 | 0.1240 | 0.2510 | 0.1511 | 0.3940 |
51
- | Swallow-7b-plus |7B| 0.5478 | **0.5493** | **0.6030** | 0.8544 | 0.1806 | 0.1360 | 0.2568 | 0.1441 | 0.4090 |
52
- | Qwen-7B |7B| 0.7712 | 0.4234 | 0.2376 | 0.8594 | 0.1371 | 0.2160 | 0.1689 | 0.1801 | 0.3742 |
53
- | nekomata-7b |7B| 0.7417 | 0.4928 | 0.5022 | 0.8707 | 0.1676 | 0.1240 | **0.2673** | 0.1815 | 0.4185 |
54
- | Mistral-7B-v0.1 |7B| 0.7301 | 0.4245 | 0.2722 | 0.8563 | 0.2006 | 0.1760 | 0.1405 | 0.1733 | 0.3717 |
55
- | japanese-stablelm-base-gamma-7b|7B| 0.7364 | 0.4643 | 0.5568 | **0.8910** | **0.2293** | 0.1680 | 0.2390 | 0.1561 | 0.4301 |
56
- | Swallow-MS-7b-v0.1 |7B| **0.8570** | 0.4915 | 0.5519 | 0.8802 | 0.1988 | **0.2240** | 0.2494 | 0.1667 | **0.4524** |
57
-
58
-
59
- ### English tasks
60
-
61
- |Model|Size|OpenBookQA|TriviaQA|HellaSwag|SQuAD2.0|XWINO|GSM8K|Average|
62
- |---|---|---|---|---|---|---|---|---|
63
- | | |8-shot|8-shot|8-shot|8-shot|8-shot|8-shot||
64
- | CyberAgentLM2-7B |7B| 0.2860 | 0.3496 | 0.5003 | 0.3510 | 0.8581 | 0.0705 | 0.4026 |
65
- | Llama 2 |7B| 0.3580 | 0.6265 | 0.5860 | 0.3207 | 0.9049 | 0.1410 | 0.4895 |
66
- | japanese-stablelm-base-beta-7b|7B| 0.3620 | 0.5903 | 0.5707 | 0.2992 | 0.8994 | 0.1198 | 0.4736 |
67
- | japanese-stablelm-base-ja_vocab-beta-7b|7B| 0.3520 | 0.5549 | 0.5644 | 0.3079 | 0.8942 | 0.0538 | 0.4545 |
68
- | ELYZA-japanese-Llama-2-7b|7B| 0.3400 | 0.5875 | 0.5595 | 0.2721 | 0.8989 | 0.1638 | 0.4703 |
69
- | ELYZA-japanese-Llama-2-7b-fast|7B| 0.3280 | 0.5817 | 0.5530 | 0.2605 | 0.8989 | 0.1425 | 0.4608 |
70
- | youri-7b (base) |7B| 0.3400 | 0.5257 | 0.5540 | 0.3297 | 0.8938 | 0.0963 | 0.4566 |
71
- | Swallow-7b |7B| 0.3180 | 0.4836 | 0.5308 | 0.3125 | 0.8817 | 0.1130 | 0.4399 |
72
- | Swallow-7b-plus |7B| 0.3280 | 0.4558 | 0.5259 | 0.3134 | 0.8929 | 0.1061 | 0.4370 |
73
- | Qwen-7B |7B| 0.3640 | 0.5695 | 0.5787 | **0.3799** | 0.8933 | **0.4617** | 0.5412 |
74
- | nekomata-7b |7B| 0.3340 | 0.4371 | 0.5340 | 0.2933 | 0.8766 | 0.1531 | 0.4380 |
75
- | Mistral-7B-v0.1 |7B| **0.3660** | **0.7050** | **0.6264** | **0.3799** | **0.9157** | 0.3533 | **0.5577** |
76
- | japanese-stablelm-base-gamma-7b|7B| 0.3240 | 0.5745 | 0.5739 | 0.3546 | 0.8976 | 0.1911 | 0.4860 |
77
- | Swallow-MS-7b-v0.1 |7B| 0.3440 | 0.5976 | 0.5810 | 0.3364 | 0.9037 | 0.2623 | 0.5042 |
78
-
79
-
80
- ### Code generation tasks
81
-
82
- |Model|Size|JHumanEval|HumanEval|
83
- |---|---|---|---|
84
- | | |pass@1|pass@1|
85
- | CyberAgentLM2-7B |7B|0.0634|0.0756|
86
- | Llama 2 |7B|0.1152|0.1378|
87
- | japanese-stablelm-base-beta-7b|7B|0.1018|0.1280|
88
- | japanese-stablelm-base-ja_vocab-beta-7b|7B|0.0896|0.1122|
89
- | ELYZA-japanese-Llama-2-7b|7B|0.0287|0.0427|
90
- | ELYZA-japanese-Llama-2-7b-fast|7B| 0.0000 |0.0037|
91
- | youri-7b (base) |7B|0.0829|0.0982|
92
- | Swallow-7b |7B|0.0183|0.0183|
93
- | Swallow-7b-plus |7B| 0.0061|0.0037|
94
- | Qwen-7B |7B|0.1701|0.1805|
95
- | nekomata-7b |7B|0.0988|0.1402|
96
- | Mistral-7B-v0.1 |7B|**0.2555**|**0.2933**|
97
- | japanese-stablelm-base-gamma-7b|7B|0.1823|0.1915|
98
- | Swallow-MS-7b-v0.1 |7B|0.2305|0.2768|
99
 
100
  ## Evaluation Benchmarks
101
 
 
15
  # Model Release Updates
16
 
17
  We are excited to share the release schedule for our latest models:
18
+ - **April 26, 2024**: Released the [Swallow-MS-7b-instruct-v0.1](https://huggingface.co/tokyotech-llm/Swallow-MS-7b-instruct-v0.1).
19
  - **March 11, 2024**: Released the [Swallow-MS-7b-v0.1](https://huggingface.co/tokyotech-llm/Swallow-MS-7b-v0.1)
20
  ![logo](./logo.png)
21
 
 
36
 
37
  ## Base Model Performance
38
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
39
 
40
  ## Evaluation Benchmarks
41