Update README.md
Browse files
README.md
CHANGED
|
@@ -20,18 +20,24 @@ For the MNTP Adapter, please refer to [this link](https://huggingface.co/uzabase
|
|
| 20 |
- **License:** Apache2.0
|
| 21 |
- **Finetuned from model:** [Swallow-7b-hf](https://huggingface.co/tokyotech-llm/Swallow-7b-hf)
|
| 22 |
|
| 23 |
-
### Model Sources
|
| 24 |
|
| 25 |
- **Repository:** https://github.com/McGill-NLP/llm2vec
|
| 26 |
- **Paper:** https://arxiv.org/abs/2404.05961
|
| 27 |
|
| 28 |
-
|
| 29 |
|
| 30 |
- Please see [original LLM2Vec repo](https://huggingface.co/McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp-unsup-simcse#usage)
|
| 31 |
|
| 32 |
-
|
|
|
|
|
|
|
| 33 |
|
| 34 |
-
|
|
|
|
|
|
|
|
|
|
|
|
|
| 35 |
|
| 36 |
- Make Corpus from SimCSE from [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia)
|
| 37 |
- Script for making SimCSE Corpus
|
|
@@ -74,7 +80,7 @@ if __name__ == "__main__":
|
|
| 74 |
|
| 75 |
|
| 76 |
|
| 77 |
-
|
| 78 |
- simcse_dropout: 0.3
|
| 79 |
- bidirectional: true
|
| 80 |
- pooling_mode: "mean"
|
|
@@ -92,7 +98,7 @@ if __name__ == "__main__":
|
|
| 92 |
- gradient_checkpointing: true
|
| 93 |
|
| 94 |
|
| 95 |
-
|
| 96 |
- deepspeed_config:
|
| 97 |
- gradient_accumulation_steps: 1
|
| 98 |
- gradient_clipping: 1.0
|
|
@@ -118,7 +124,7 @@ if __name__ == "__main__":
|
|
| 118 |
- quse_cpu: false
|
| 119 |
|
| 120 |
|
| 121 |
-
|
| 122 |
|
| 123 |
- Python: 3.12.3
|
| 124 |
- PEFT 0.11.1
|
|
|
|
| 20 |
- **License:** Apache2.0
|
| 21 |
- **Finetuned from model:** [Swallow-7b-hf](https://huggingface.co/tokyotech-llm/Swallow-7b-hf)
|
| 22 |
|
| 23 |
+
### Model Sources
|
| 24 |
|
| 25 |
- **Repository:** https://github.com/McGill-NLP/llm2vec
|
| 26 |
- **Paper:** https://arxiv.org/abs/2404.05961
|
| 27 |
|
| 28 |
+
# Usage
|
| 29 |
|
| 30 |
- Please see [original LLM2Vec repo](https://huggingface.co/McGill-NLP/LLM2Vec-Llama-2-7b-chat-hf-mntp-unsup-simcse#usage)
|
| 31 |
|
| 32 |
+
# Benchmark
|
| 33 |
+
= Followings are summaries. Details are [here]()
|
| 34 |
+
## MTEB(Japanese)
|
| 35 |
|
| 36 |
+
## MTEB(English)
|
| 37 |
+
|
| 38 |
+
# Training Details
|
| 39 |
+
|
| 40 |
+
## Training Data
|
| 41 |
|
| 42 |
- Make Corpus from SimCSE from [Wikipedia](https://huggingface.co/datasets/wikimedia/wikipedia)
|
| 43 |
- Script for making SimCSE Corpus
|
|
|
|
| 80 |
|
| 81 |
|
| 82 |
|
| 83 |
+
## Training Hyperparameter
|
| 84 |
- simcse_dropout: 0.3
|
| 85 |
- bidirectional: true
|
| 86 |
- pooling_mode: "mean"
|
|
|
|
| 98 |
- gradient_checkpointing: true
|
| 99 |
|
| 100 |
|
| 101 |
+
## Accelerator Settings
|
| 102 |
- deepspeed_config:
|
| 103 |
- gradient_accumulation_steps: 1
|
| 104 |
- gradient_clipping: 1.0
|
|
|
|
| 124 |
- quse_cpu: false
|
| 125 |
|
| 126 |
|
| 127 |
+
## Framework versions
|
| 128 |
|
| 129 |
- Python: 3.12.3
|
| 130 |
- PEFT 0.11.1
|