gabrielmbmb/Upcycled-Qwen1.5-MoE2.7B

Text Generation • Updated Mar 30, 2024 • 12 • 2

Note This model hasn't been trained, just initialised using upcycling process and the weights from Qwen/Qwen1.5-18B.

gabrielmbmb/Upcycled-Qwen1.5-MoE2.7B-LoRA-merged

Text Generation • Updated Mar 30, 2024 • 5

Note Using LoRA and targeting up_proj, gate_proj, down_proj, gate, and shared_expert_gate. LoRA rank was 8. About 126M trainable parameters. Dataset used was wiki_demo from LLaMa Factory.

gabrielmbmb/Upcycled-Qwen1.5-MoE2.7B-LoRA-merged-32

Text Generation • Updated Mar 31, 2024 • 12

Note Using LoRA targeting all the layers and rank 32. About 500m trainable parameters.