Update README.md

README.md

@@ -16,9 +16,102 @@ This model is a model that performed continued pre-training and fine-tuning (ins
 
 ### DUS(Depth Up-Scaling) and continued pre-training
 Similar to the methodology disclosed in the paper, we expanded from 32 transformer blocks to 48 blocks and then continued pre-training with the public dataset. Pre-training was performed for 3 days using 4 `ml.g5.48xlarge` instances from AWS (NVIDIA A10G GPU x 32ea). For pre-training, we used a sample set from Wikipedia.
+For distributed training, all weights were trained without adapter techniques, and sharding parallelization was performed with ZeRO-2. The presets are as follows.
+
+```json
+{
+    "fp16": {
+        "enabled": "auto",
+        "loss_scale": 0,
+        "loss_scale_window": 1000,
+        "initial_scale_power": 16,
+        "hysteresis": 2,
+        "min_loss_scale": 1
+    },
+    
+    "bf16": {
+        "enabled": "auto"
+    },    
+
+    "optimizer": {
+        "type": "AdamW",
+        "params": {
+            "lr": "auto",
+            "betas": "auto",
+            "eps": "auto",
+            "weight_decay": "auto"
+        }
+    },
+
+    "scheduler": {
+        "type": "WarmupLR",
+        "params": {
+            "warmup_min_lr": "auto",
+            "warmup_max_lr": "auto",
+            "warmup_num_steps": "auto"
+        }
+    },
+
+    "zero_optimization": {
+        "stage": 2,
+        "allgather_partitions": true,
+        "allgather_bucket_size": 2e8,
+        "overlap_comm": true,
+        "reduce_scatter": true,
+        "reduce_bucket_size": 2e8,
+        "contiguous_gradients": true,
+        "cpu_offload": true
+    },
+
+    "gradient_accumulation_steps": "auto",
+    "gradient_clipping": "auto",
+    "train_batch_size": "auto",
+    "train_micro_batch_size_per_gpu": "auto"
+}
+```
+
+Some hyperparameters are listed below.
+```
+batch_size: 2
+num_epochs: 1
+learning_rate: 3e-4
+gradient_accumulation_steps: 8
+lr_scheduler_type: "linear"
+group_by_length: False
+```
 
 ### Fine-tuning
 After performing pre-training, instruction tuning and alignment tuning were performed sequentially. This process only took about 10 hours using AWS `ml.g5.24xlarge` (NVIDIA A10G GPU x 4ea). The dataset used for instruction tuning is a sample set of the OpenOrca dataset, and the dataset used for alignment tuning is Intel's orca_dpo_pairs dataset.
+All fine-tuning was learned using QLoRA, and the batch sizes were set to 3 and 1, respectively. We used 1,024 for the context length. 2,048 is also possible, but applying DPO often runs out of memory on 24GB GPU memory, so we settled on 1,024.
+Please see below for relevant code snippets.
+
+```python
+peft_config = LoraConfig(
+    r=8,
+    lora_alpha=16,
+    target_modules=["q_proj", "k_proj", "v_proj", "fc1", "fc2"],
+    lora_dropout=0.05,
+    bias="none",
+    task_type="CAUSAL_LM",
+)
+training_arguments = TrainingArguments(
+    output_dir="logs",
+    num_train_epochs=1,
+    per_device_train_batch_size=batch_size,
+    gradient_accumulation_steps=4,
+    optim="paged_adamw_8bit",
+    learning_rate=3e-4,
+    weight_decay=0.001,
+    bf16=True,
+    max_grad_norm=0.3,
+    max_steps=-1,
+    warmup_ratio=0.03,
+    group_by_length=True,
+    lr_scheduler_type="cosine",
+    report_to="wandb", ...
+)
+`
+``
 
 ### References
 - Base model: [microsoft/phi-2](https://huggingface.co/microsoft/phi-2)