meyceoz commited on
Commit
d0e159a
·
verified ·
1 Parent(s): 372968b

Update README.md

Browse files
Files changed (1) hide show
  1. README.md +5 -23
README.md CHANGED
@@ -14,7 +14,7 @@ base_model:
14
  ---
15
  # Model Card for Granite-3.0-8b-lab-community [Paper](https://arxiv.org/abs/2403.01081)
16
 
17
- # NOTE THIS STILL NEEDS TO BE UPDATED BY SOMEONE WHO KNOWS THIS STUFF. -JJ (awesome@ibm.com)
18
 
19
 
20
  ### Overview
@@ -26,24 +26,6 @@ Granite-3.0-8b-lab-community is the first release in our community-driven, openl
26
 
27
  ![Screenshot 2024-02-22 at 11.26.13 AM.png](model-card/Model%20Card%20for%20Merlinite%207b%2028cc0b72cf574a4a828140d3539ede4a_Screenshot_2024-02-22_at_11.26.13_AM.png)
28
 
29
- ### Performance
30
-
31
- | Model | Alignment | Base | Teacher | MTBench (Avg) * | MMLU(5-shot) |
32
- | --- | --- | --- | --- | --- | --- |
33
- | [Llama-2-13b-chat-hf](https://huggingface.co/meta-llama/Llama-2-13b-chat-hf) | RLHF | Llama-2-13b | Human Annotators | 6.65 |54.58 |
34
- | [Orca-2-13b](https://huggingface.co/microsoft/Orca-2-13b) | Progressive Training | Llama-2-13b | GPT-4 | 6.15 | 60.37 * |
35
- | [WizardLM-13B-V1.2](https://huggingface.co/WizardLM/WizardLM-13B-V1.2) | Evol-Instruct | Llama-2-13b | GPT-4 | 7.20 | 54.83 |
36
- | [Labradorite-13b](https://huggingface.co/ibm/labradorite-13b) | Large-scale Alignment for chatBots (LAB) | Llama-2-13b | Mixtral-8x7B-Instruct | 7.23 | 58.89 |
37
- | [Mistral-7B-Instruct-v0.1](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.1) | SFT | Mistral-7B-v0.1 | - | 6.84 | 60.37 |
38
- | [zephyr-7b-beta](https://huggingface.co/HuggingFaceH4/zephyr-7b-beta) | SFT/DPO | Mistral-7B-v0.1 | GPT-4 | 7.34 | 61.07 |
39
- | [Mistral-7B-Instruct-v0.2](https://huggingface.co/mistralai/Mistral-7B-Instruct-v0.2) | SFT | Mistral-7B-v0.1 | - | 7.6** | 60.78 |
40
- | [Merlinite-7b-lab](https://huggingface.co/instructlab/merlinite-7b-lab) | Large-scale Alignment for chatBots (LAB) | Mistral-7B-v0.1 | Mixtral-8x7B-Instruct | 7.66 |64.88 |
41
- | Granite-7b-lab | Large-scale Alignment for chatBots (LAB) | Granite-7b-base| Mixtral-8x7B-Instruct | 6.69 | 51.91 |
42
-
43
- [*] Numbers for models other than Merlinite-7b-lab, Granite-7b-lab and [Labradorite-13b](https://huggingface.co/ibm/labradorite-13b) are taken from [lmsys/chatbot-arena-leaderboard](https://huggingface.co/spaces/lmsys/chatbot-arena-leaderboard)
44
-
45
- [**] Numbers taken from [MistralAI Release Blog](https://mistral.ai/news/la-plateforme/)
46
-
47
  ### Method
48
 
49
  LAB: **L**arge-scale **A**lignment for chat**B**ots is a novel synthetic data-based alignment tuning method for LLMs from IBM Research. Granite-7b-lab is a Granite-7b-base derivative model trained with the LAB methodology, using Mixtral-8x7b-Instruct as a teacher model.
@@ -81,16 +63,16 @@ Importantly, we use a set of hyper-parameters for training that are very differe
81
  ![Untitled](model-card/Model%20Card%20for%20Merlinite%207b%2028cc0b72cf574a4a828140d3539ede4a_Untitled%202.png)
82
 
83
  ## Model description
84
- - **Model Name**: Granite-7b-lab
85
  - **Language(s):** Primarily English
86
  - **License:** Apache 2.0
87
- - **Base model:** [ibm/granite-7b-base](https://huggingface.co/ibm/granite-7b-base)
88
  - **Teacher Model:** [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)
89
 
90
  ## Prompt Template
91
 
92
  ```python
93
- sys_prompt = "You are an AI language model developed by IBM Research. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior."
94
 
95
  prompt = f'<|system|>\n{sys_prompt}\n<|user|>\n{inputs}\n<|assistant|>\n'
96
  stop_token = '<|endoftext|>'
@@ -102,4 +84,4 @@ We advise utilizing the system prompt employed during the model's training for o
102
 
103
  **Bias, Risks, and Limitations**
104
 
105
- Granite-7b-lab is a base model and has not undergone any safety alignment, there it may produce problematic outputs. In the absence of adequate safeguards and RLHF, there exists a risk of malicious utilization of these models for generating disinformation or harmful content. Caution is urged against complete reliance on a specific language model for crucial decisions or impactful information, as preventing these models from fabricating content is not straightforward. Additionally, it remains uncertain whether smaller models might exhibit increased susceptibility to hallucination in ungrounded generation scenarios due to their reduced sizes and memorization capacities. This aspect is currently an active area of research, and we anticipate more rigorous exploration, comprehension, and mitigations in this domain.
 
14
  ---
15
  # Model Card for Granite-3.0-8b-lab-community [Paper](https://arxiv.org/abs/2403.01081)
16
 
17
+ # NOTE: MODEL CARD UPDATES IN-PROGRESS
18
 
19
 
20
  ### Overview
 
26
 
27
  ![Screenshot 2024-02-22 at 11.26.13 AM.png](model-card/Model%20Card%20for%20Merlinite%207b%2028cc0b72cf574a4a828140d3539ede4a_Screenshot_2024-02-22_at_11.26.13_AM.png)
28
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
29
  ### Method
30
 
31
  LAB: **L**arge-scale **A**lignment for chat**B**ots is a novel synthetic data-based alignment tuning method for LLMs from IBM Research. Granite-7b-lab is a Granite-7b-base derivative model trained with the LAB methodology, using Mixtral-8x7b-Instruct as a teacher model.
 
63
  ![Untitled](model-card/Model%20Card%20for%20Merlinite%207b%2028cc0b72cf574a4a828140d3539ede4a_Untitled%202.png)
64
 
65
  ## Model description
66
+ - **Model Name**: Granite-3.0-8b-lab-community
67
  - **Language(s):** Primarily English
68
  - **License:** Apache 2.0
69
+ - **Base model:** [ibm-granite/granite-3.0-8b-base](https://huggingface.co/ibm-granite/granite-3.0-8b-base)
70
  - **Teacher Model:** [mistralai/Mixtral-8x7B-Instruct-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-Instruct-v0.1)
71
 
72
  ## Prompt Template
73
 
74
  ```python
75
+ sys_prompt = "I am a Red Hat® Instruct Model, an AI language model developed by Red Hat and IBM Research based on the granite-3.0-8b-base model. My primary role is to serve as a chat assistant."
76
 
77
  prompt = f'<|system|>\n{sys_prompt}\n<|user|>\n{inputs}\n<|assistant|>\n'
78
  stop_token = '<|endoftext|>'
 
84
 
85
  **Bias, Risks, and Limitations**
86
 
87
+ Granite-3.0-8b-lab-community is a base model and has not undergone any safety alignment, there it may produce problematic outputs. In the absence of adequate safeguards and RLHF, there exists a risk of malicious utilization of these models for generating disinformation or harmful content. Caution is urged against complete reliance on a specific language model for crucial decisions or impactful information, as preventing these models from fabricating content is not straightforward. Additionally, it remains uncertain whether smaller models might exhibit increased susceptibility to hallucination in ungrounded generation scenarios due to their reduced sizes and memorization capacities. This aspect is currently an active area of research, and we anticipate more rigorous exploration, comprehension, and mitigations in this domain.