Building your RHEL AI environment
Building your RHEL AI environment Red Hat Enterprise Linux AI 1.2 Creating accounts, initalizing RHEL AI, downloading models, and serving/chat customizations Red Hat RHEL AI Documentation Team
rhc connect --organization <org id> --activation-key <created key> sudo mkdir -p /etc/ilab sudo touch /etc/ilab/insights-opt-out ilab system info ilab config init Welcome to InstructLab CLI. This guide will help you to setup your environment. Detecting Hardware We chose Nvidia 2x A100 as your designated training profile. This is for systems with 160 GB of vRAM. Is this correct? [y/N]: We chose Nvidia 2x A100 as your designated training profile. This is for systems with 160 GB of vRAM. This profile is the best approximation for your system based off of the amount of vRAM. We modified it to match the number of GPUs you have. Is this profile correct? [Y/n]: Please choose a train profile to use. Train profiles assist with the complexity of configuring specific GPU hardware with the InstructLab Training library. You can still take advantage of hardware acceleration for training even if your hardware is not listed. [0] No profile (CPU, Apple Metal, AMD ROCm) [1] Nvidia A100/H100 x2 (A100_H100_x2.yaml) [2] Nvidia A100/H100 x4 (A100_H100_x4.yaml) [3] Nvidia A100/H100 x8 (A100_H100_x8.yaml) [4] Nvidia L40 x4 (L40_x4.yaml) [5] Nvidia L40 x8 (L40_x8.yaml) [6] Nvidia L4 x8 (L4_x8.yaml) Enter the number of your choice [hit enter for no profile] [0]: You selected: Nvidia A100/H100 x8 (A100_H100_x8.yaml) Initialization completed successfully, you're ready to start using `ilab`. Enjoy! ilab config edit evaluate: base_branch: null base_model: ~/.cache/instructlab/models/granite-7b-starter branch: null gpus: <num-gpus> serve: backend: vllm chat_template: auto host_port: 127.0.0.1:8000 llama_cpp: gpu_layers: -1 llm_family: '' max_ctx_size: 4096 model_path: ~/.cache/instructlab/models/granite-7b-redhat-lab vllm: llm_family: '' vllm_args: ["--tensor-parallel-size", "<num-gpus>"] gpus: <num-gpus> rm -rf ~/.local/share/instructlab/taxonomy/ ; git clone https://github.com/RedHatOfficial/rhelai-sample-taxonomy.git ~/.local/share/instructlab/taxonomy/ ├─ ~/.cache/instructlab/models/ 1 ├─ ~/.local/share/instructlab/datasets 2 ├─ ~/.local/share/instructlab/taxonomy 3 ├─ ~/.local/share/instructlab/phased/<phase1-or-phase2>/checkpoints/ 4 ilab config show ilab config edit ilab config edit train: max_batch_len=220000 nproc_per_node=8 eval: gpus=1 train: max_batch_len=210000 nproc_per_node=4 eval: gpus=1 train: max_batch_len=200000 nproc_per_node=2 eval: gpus=1 ilab model download --repository docker://<repository_and_model> --release <release> ilab model download --repository docker://registry.redhat.io/rhelai1/granite-7b-starter --release latest ilab model list +-----------------------------------+---------------------+---------+ | Model Name | Last Modified | Size | +-----------------------------------+---------------------+---------+ | models/prometheus-8x7b-v2-0 | 2024-08-09 13:28:50 | 87.0 GB| | models/mixtral-8x7b-instruct-v0-1 | 2024-08-09 13:28:24 | 87.0 GB| | models/granite-7b-redhat-lab | 2024-08-09 14:28:40 | 12.6 GB| | models/granite-7b-starter | 2024-08-09 14:40:35 | 12.6 GB| +-----------------------------------+---------------------+---------+ ls ~/.cache/instructlab/models granite-7b-starter granite-7b-redhat-lab ilab model serve ilab model serve --model-path <model-path> ilab model serve --model-path ~/.cache/instructlab/models/granite-8b-code-instruct INFO 2024-03-02 02:21:11,352 lab.py:201 Using model 'models/granite-8b-code-instruct' with -1 gpu-layers and 4096 max context size. Starting server process After application startup complete see http://127.0.0.1:8000/docs for API. Press CTRL+C to shut down the server. mkdir -p USDHOME/.config/systemd/user cat << EOF > USDHOME/.config/systemd/user/ilab-serve.service [Unit] Description=ilab model serve service [Install] WantedBy=multi-user.target default.target 1 [Service] ExecStart=ilab model serve --model-family granite Restart=always EOF systemctl --user daemon-reload systemctl --user start ilab-serve.service systemctl --user status ilab-serve.service journalctl --user-unit ilab-serve.service sudo loginctl enable-linger systemctl --user stop ilab-serve.service mkdir -p `pwd`/nginx/ssl/ cat > openssl.cnf <<EOL [ req ] default_bits = 2048 distinguished_name = <req-distinguished-name> 1 x509_extensions = v3_req prompt = no [ req_distinguished_name ] C = US ST = California L = San Francisco O = My Company OU = My Division CN = rhelai.redhat.com [ v3_req ] subjectAltName = <alt-names> 2 basicConstraints = critical, CA:true subjectKeyIdentifier = hash authorityKeyIdentifier = keyid:always,issuer [ alt_names ] DNS.1 = rhelai.redhat.com 3 DNS.2 = www.rhelai.redhat.com 4 openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout `pwd`/nginx/ssl/rhelai.redhat.com.key -out `pwd`/nginx/ssl/rhelai.redhat.com.crt -config openssl.cnf openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout mkdir -p `pwd`/nginx/conf.d echo 'server { listen 8443 ssl; server_name <rhelai.redhat.com> 1 ssl_certificate /etc/nginx/ssl/rhelai.redhat.com.crt; ssl_certificate_key /etc/nginx/ssl/rhelai.redhat.com.key; location / { proxy_pass http://127.0.0.1:8000; proxy_set_header Host USDhost; proxy_set_header X-Real-IP USDremote_addr; proxy_set_header X-Forwarded-For USDproxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto USDscheme; } } ' > `pwd`/nginx/conf.d/rhelai.redhat.com.conf podman run --net host -v `pwd`/nginx/conf.d:/etc/nginx/conf.d:ro,Z -v `pwd`/nginx/ssl:/etc/nginx/ssl:ro,Z nginx ilab model chat -m /instructlab/instructlab/granite-7b-redhat-lab --endpoint-url curl --location 'https://rhelai.redhat.com:8443/v1' --header 'Content-Type: application/json' --header 'Authorization: Bearer <api-key>' --data '{ "model": "/var/home/cloud-user/.cache/instructlab/models/granite-7b-redhat-lab", "messages": [ { "role": "system", "content": "You are a helpful assistant." }, { "role": "user", "content": "Hello!" } ] }' | jq . openssl s_client -connect rhelai.redhat.com:8443 </dev/null 2>/dev/null | openssl x509 -outform PEM > server.crt sudo cp server.crt /etc/pki/ca-trust/source/anchors/ sudo update-ca-trust cat server.crt >> USD(python -m certifi) ilab model chat -m /instructlab/instructlab/granite-7b-redhat-lab --endpoint-url https://rhelai.redhat.com:8443/v1 ilab model chat ilab model chat --model <model-path> ilab model chat --model ~/.cache/instructlab/models/granite-8b-code-instruct ilab model chat ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Welcome to InstructLab Chat w/ GRANITE-8B-CODE-INSTRUCT (type /h for help) │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ >>> [S][default] export VLLM_API_KEY=USD(python -c 'import secrets; print(secrets.token_urlsafe())') echo USDVLLM_API_KEY ilab config edit serve: vllm: vllm_args: - --api-key - <api-key-string> ilab model chat openai.AuthenticationError: Error code: 401 - {'error': 'Unauthorized'} ilab model chat -m granite-7b-redhat-lab --endpoint-url https://inference.rhelai.com/v1 --api-key USDVLLM_API_KEY ilab model chat ╭────────────────────────────────────────────────────────────────────────────────────────────────────────────────── system ──────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮ │ Welcome to InstructLab Chat w/ GRANITE-7B-LAB (type /h for help) │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ >>> [S][default]