File size: 4,453 Bytes
1e21185 59fd3eb 1e21185 e7163e4 1e21185 fc98b4d da0624f fc98b4d da0624f fc98b4d 995e282 fc98b4d 995e282 fc98b4d 995e282 fc98b4d 995e282 fc98b4d da0624f 59fd3eb da0624f fc98b4d |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 |
---
license: cc-by-nc-4.0
language:
- en
metrics:
- accuracy
base_model:
- meta-llama/Llama-3.1-405B-Instruct
pipeline_tag: text-generation
---
# CALM-405B: The Largest Open-Source Agentic LLM
## 🌟 Model Overview
**CALM-405B** is the **largest open-source Conversational Agentic Language Model (LLM) ever created**. This model sets a new standard in **Conversational AI**, seamlessly integrating both **Task-Oriented Dialogue (TOD) capabilities** and **Language Agent (LA) functionalities**.
It is designed to **push the boundaries** of open-source agentic LLMs, excelling at **multi-turn dialogue, tool usage, reasoning, and API execution**. It is the **best-performing fully open-source LLM** on the **Berkeley Function Calling Leaderboard V3 (BFCL V3)**, marking a historic leap in open-source AI research.
## Model Sources [TODO]
<!-- Provide the basic links for the model. -->
- **Paper:** [More Information Needed]
- **Repository:** [More Information Needed]
---
## 🚀 Model Details
- **Model Name:** CALM-405B
- **Developed by:** Colloboration of UIUC Conversational AI LAB and Oumi
- **License:** Apache 2.0
- **Architecture:** Meta-Llama 3.1-405B Instruct
- **Training Data:** CALM-IT
- **Fine-tuning Framework:** [Oumi](https://github.com/oumi-ai/oumi)
- **Training Hardware:** 8 NVIDIA H100 GPUs
- **Training Duration:** ~6.5 days
- **Evaluation Benchmarks:** MultiWOZ 2.4, BFCL V3, API-Bank
- **Release Date:** February 5, 2025
---
## 🏆 Why CALM-405B is a Game-Changer
- **🚨 Largest Open-Source Agentic LLM:** A **405B** parameter model that brings state-of-the-art agentic capabilities to the public domain.
- **🎯 Best Open-Source Performance on BFCL V3:** Outperforms leading proprietary models like **GPT-4o, Gemini, and Claude** in function-calling tasks.
- **🔍 True Zero-Shot Function Calling:** Generalizes to unseen API tasks with **unmatched accuracy**.
- **🤖 Multi-Turn Dialogue Mastery:** Excels at long conversations, **task tracking, and complex reasoning**.
- **🛠 API Tool Use and Reasoning:** Makes precise API calls, interprets responses, and synthesizes **coherent** multi-step solutions.
- **📜 Fully Open-Source & Reproducible:** Released under **Apache 2.0**, including model weights, training logs, and datasets.
---
## 📊 Benchmark Performance
TODO: Add BFCL results
---
## 🔧 Training Process
### Fine-tuning Stages
1. **TOD Fine-tuning:** Optimized for **dialogue state tracking** (e.g., augmented SNIPS in instruction-tuned format).
2. **Function Calling Fine-tuning:** Trained to generate **highly accurate API calls** from LA datasets.
3. **ReAct-based Fine-tuning:** Enhances multi-turn conversations with structured **thought-action-observation-response reasoning**.
### Training Hyperparameters
- **Base Model:** Meta-Llama 3.1-405B Instruct
- **LoRA Config:** Rank = 16, Scaling Factor = 32
- **Batch Size:** 2
- **Learning Rate:** 1e-4
- **Optimizer:** AdamW (betas = 0.9, 0.999, epsilon = 1e-8)
- **Precision:** q4
- **Warm-up Steps:** 500
- **Gradient Accumulation Steps:** 1
---
## 💡 How to Use CALM-405B
🚨 It requires 16xH100 NVIDIA GPUs for Inference.
### 🏗 How to Load the Model using HuggingFace
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("uiuc-convai/CALM-8B")
model = AutoModelForCausalLM.from_pretrained("uiuc-convai/CALM-8B")
```
### 🛠 Example Oumi Inference
CALM-405B likely requires multi-node inference as most single nodes support up to 640GB of GPU VRAM. To run multi-node inference, we recommend [vLLM](https://docs.vllm.ai/en/latest/serving/distributed_serving.html)
### 🛠 Example Oumi Fine-Tuning
```bash
pip install oumi
# See oumi_train.yaml in this model's /oumi/ directory.
oumi train -c ./oumi_train.yaml
```
More fine-tuning and **community-driven** optimizations are planned to enhance real-world usability.
## License
This model is licensed under [Creative Commons NonCommercial (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/legalcode).
---
## 📖 Citation
If you use **CALM-405B** in your research, please cite:
```
@article{yourpaper2024,
title={CALM: Conversational Agentic Language Model},
author={Your Name and Collaborators},
journal={Your Conference/Journal},
year={2024}
}
```
For more details, visit [Project Repository](https://github.com/your-repo) or contact **[email protected]**.
|