robgreenberg3 jennyyyi commited on
Commit
a32a651
·
verified ·
1 Parent(s): dc36722

Update README.md (#2)

Browse files

- Update README.md (d6ed92f3a4f39bf869c5cfb2b2c2f215f484f460)


Co-authored-by: Jenny Y <[email protected]>

Files changed (1) hide show
  1. README.md +164 -2
README.md CHANGED
@@ -18,8 +18,14 @@ pipeline_tag: text-generation
18
  license: llama3.3
19
  base_model: meta-llama/Llama-3.3-70B-Instruct
20
  ---
21
-
22
- # Llama-3.3-70B-Instruct-quantized.w8a8
 
 
 
 
 
 
23
 
24
  ## Model Overview
25
  - **Model Architecture:** Llama
@@ -81,6 +87,162 @@ print(generated_text)
81
 
82
  vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
83
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
84
 
85
  ## Creation
86
 
 
18
  license: llama3.3
19
  base_model: meta-llama/Llama-3.3-70B-Instruct
20
  ---
21
+ <h1 style="display: flex; align-items: center; gap: 10px; margin: 0;">
22
+ Llama-3.3-70B-Instruct-quantized.w8a8
23
+ <img src="https://www.redhat.com/rhdc/managed-files/Catalog-Validated_model_0.png" alt="Model Icon" width="40" style="margin: 0; padding: 0;" />
24
+ </h1>
25
+
26
+ <a href="https://www.redhat.com/en/products/ai/validated-models" target="_blank" style="margin: 0; padding: 0;">
27
+ <img src="https://www.redhat.com/rhdc/managed-files/Validated_badge-Dark.png" alt="Validated Badge" width="250" style="margin: 0; padding: 0;" />
28
+ </a>
29
 
30
  ## Model Overview
31
  - **Model Architecture:** Llama
 
87
 
88
  vLLM aslo supports OpenAI-compatible serving. See the [documentation](https://docs.vllm.ai/en/latest/) for more details.
89
 
90
+ <details>
91
+ <summary>Deploy on <strong>Red Hat AI Inference Server</strong></summary>
92
+
93
+ ```bash
94
+ $ podman run --rm -it --device nvidia.com/gpu=all -p 8000:8000 \
95
+ --ipc=host \
96
+ --env "HUGGING_FACE_HUB_TOKEN=$HF_TOKEN" \
97
+ --env "HF_HUB_OFFLINE=0" -v ~/.cache/vllm:/home/vllm/.cache \
98
+ --name=vllm \
99
+ registry.access.redhat.com/rhaiis/rh-vllm-cuda \
100
+ vllm serve \
101
+ --tensor-parallel-size 8 \
102
+ --max-model-len 32768 \
103
+ --enforce-eager --model RedHatAI/Llama-3.3-70B-Instruct-quantized.w8a8
104
+ ```
105
+ ​​See [Red Hat AI Inference Server documentation](https://docs.redhat.com/en/documentation/red_hat_ai_inference_server/) for more details.
106
+ </details>
107
+
108
+ <details>
109
+ <summary>Deploy on <strong>Red Hat Enterprise Linux AI</strong></summary>
110
+
111
+ ```bash
112
+ # Download model from Red Hat Registry via docker
113
+ # Note: This downloads the model to ~/.cache/instructlab/models unless --model-dir is specified.
114
+ ilab model download --repository docker://registry.redhat.io/rhelai1/llama-3-3-70b-instruct-quantized-w8a8:1.5
115
+ ```
116
+
117
+ ```bash
118
+ # Serve model via ilab
119
+ ilab model serve --model-path ~/.cache/instructlab/models/llama-3-3-70b-instruct-quantized-w8a8
120
+
121
+ # Chat with model
122
+ ilab model chat --model ~/.cache/instructlab/models/llama-3-3-70b-instruct-quantized-w8a8
123
+ ```
124
+ See [Red Hat Enterprise Linux AI documentation](https://docs.redhat.com/en/documentation/red_hat_enterprise_linux_ai/1.4) for more details.
125
+ </details>
126
+
127
+ <details>
128
+ <summary>Deploy on <strong>Red Hat Openshift AI</strong></summary>
129
+
130
+ ```python
131
+ # Setting up vllm server with ServingRuntime
132
+ # Save as: vllm-servingruntime.yaml
133
+ apiVersion: serving.kserve.io/v1alpha1
134
+ kind: ServingRuntime
135
+ metadata:
136
+ name: vllm-cuda-runtime # OPTIONAL CHANGE: set a unique name
137
+ annotations:
138
+ openshift.io/display-name: vLLM NVIDIA GPU ServingRuntime for KServe
139
+ opendatahub.io/recommended-accelerators: '["nvidia.com/gpu"]'
140
+ labels:
141
+ opendatahub.io/dashboard: 'true'
142
+ spec:
143
+ annotations:
144
+ prometheus.io/port: '8080'
145
+ prometheus.io/path: '/metrics'
146
+ multiModel: false
147
+ supportedModelFormats:
148
+ - autoSelect: true
149
+ name: vLLM
150
+ containers:
151
+ - name: kserve-container
152
+ image: quay.io/modh/vllm:rhoai-2.20-cuda # CHANGE if needed. If AMD: quay.io/modh/vllm:rhoai-2.20-rocm
153
+ command:
154
+ - python
155
+ - -m
156
+ - vllm.entrypoints.openai.api_server
157
+ args:
158
+ - "--port=8080"
159
+ - "--model=/mnt/models"
160
+ - "--served-model-name={{.Name}}"
161
+ env:
162
+ - name: HF_HOME
163
+ value: /tmp/hf_home
164
+ ports:
165
+ - containerPort: 8080
166
+ protocol: TCP
167
+ ```
168
+
169
+ ```python
170
+ # Attach model to vllm server. This is an NVIDIA template
171
+ # Save as: inferenceservice.yaml
172
+ apiVersion: serving.kserve.io/v1beta1
173
+ kind: InferenceService
174
+ metadata:
175
+ annotations:
176
+ openshift.io/display-name: llama-3-3-70b-instruct-quantized-w8a8 # OPTIONAL CHANGE
177
+ serving.kserve.io/deploymentMode: RawDeployment
178
+ name: llama-3-3-70b-instruct-quantized-w8a8 # specify model name. This value will be used to invoke the model in the payload
179
+ labels:
180
+ opendatahub.io/dashboard: 'true'
181
+ spec:
182
+ predictor:
183
+ maxReplicas: 1
184
+ minReplicas: 1
185
+ model:
186
+ modelFormat:
187
+ name: vLLM
188
+ name: ''
189
+ resources:
190
+ limits:
191
+ cpu: '2' # this is model specific
192
+ memory: 8Gi # this is model specific
193
+ nvidia.com/gpu: '1' # this is accelerator specific
194
+ requests: # same comment for this block
195
+ cpu: '1'
196
+ memory: 4Gi
197
+ nvidia.com/gpu: '1'
198
+ runtime: vllm-cuda-runtime # must match the ServingRuntime name above
199
+ storageUri: oci://registry.redhat.io/rhelai1/modelcar-llama-3-3-70b-instruct-quantized-w8a8:1.5
200
+ tolerations:
201
+ - effect: NoSchedule
202
+ key: nvidia.com/gpu
203
+ operator: Exists
204
+ ```
205
+
206
+ ```bash
207
+ # make sure first to be in the project where you want to deploy the model
208
+ # oc project <project-name>
209
+
210
+ # apply both resources to run model
211
+
212
+ # Apply the ServingRuntime
213
+ oc apply -f vllm-servingruntime.yaml
214
+
215
+ # Apply the InferenceService
216
+ oc apply -f qwen-inferenceservice.yaml
217
+ ```
218
+
219
+ ```python
220
+ # Replace <inference-service-name> and <cluster-ingress-domain> below:
221
+ # - Run `oc get inferenceservice` to find your URL if unsure.
222
+
223
+ # Call the server using curl:
224
+ curl https://<inference-service-name>-predictor-default.<domain>/v1/chat/completions
225
+ -H "Content-Type: application/json" \
226
+ -d '{
227
+ "model": "llama-3-3-70b-instruct-quantized-w8a8 ",
228
+ "stream": true,
229
+ "stream_options": {
230
+ "include_usage": true
231
+ },
232
+ "max_tokens": 1,
233
+ "messages": [
234
+ {
235
+ "role": "user",
236
+ "content": "How can a bee fly when its wings are so small?"
237
+ }
238
+ ]
239
+ }'
240
+
241
+ ```
242
+
243
+ See [Red Hat Openshift AI documentation](https://docs.redhat.com/en/documentation/red_hat_openshift_ai/2025) for more details.
244
+ </details>
245
+
246
 
247
  ## Creation
248