Chapter 2. Initializing InstructLab
Chapter 2. Initializing InstructLab You must initialize the InstructLab environments to begin working with the Red Hat Enterprise Linux AI models. 2.1. Creating your RHEL AI environment You can start interacting with LLMs and the RHEL AI tooling by initializing the InstructLab environment. Prerequisites You installed RHEL AI with the bootable container image. You have root user access on your machine. Procedure Optional: You can view your machine's information by running the following command: USD ilab system info Initialize InstructLab by running the following command: USD ilab config init The RHEL AI CLI starts setting up your environment and config.yaml file. The CLI automatically detects your machine's hardware and selects a training profile based on the system's types of GPUs and available vRAM. The CLI then prompts you to confirm if the training profile it detected is correct. If the profile is correct, your config.yaml file updates with the appropriate train configurations for training your LLM. Example output of profile auto selection Welcome to InstructLab CLI. This guide will help you to setup your environment. Detecting Hardware... We chose Nvidia 2x A100 as your designated training profile. This is for systems with 160 GB of vRAM. Is this correct? [y/N]: If the CLI detects a hardware configuration that does not exactly match your system, the following output displays: We chose Nvidia 2x A100 as your designated training profile. This is for systems with 160 GB of vRAM. This profile is the best approximation for your system based off of the amount of vRAM. We modified it to match the number of GPUs you have. Is this profile correct? [Y/n]: You can continue with the recommendation by typing y in the window. Or, you can manually select your own training profile by typing n in the window. If the CLI displays incorrect hardware configurations, type n into your window. The CLI prompts you to manually select your training profile. Type the number of the YAML file that matches your hardware specifications. Important These profiles only add the necessary configurations to the train section of your config.yaml file, therefore any profile can be selected for inference serving a model. Example output of selecting training profiles Please choose a train profile to use. Train profiles assist with the complexity of configuring specific GPU hardware with the InstructLab Training library. You can still take advantage of hardware acceleration for training even if your hardware is not listed. [0] No profile (CPU, Apple Metal, AMD ROCm) [1] Nvidia A100/H100 x2 (A100_H100_x2.yaml) [2] Nvidia A100/H100 x4 (A100_H100_x4.yaml) [3] Nvidia A100/H100 x8 (A100_H100_x8.yaml) [4] Nvidia L40 x4 (L40_x4.yaml) [5] Nvidia L40 x8 (L40_x8.yaml) [6] Nvidia L4 x8 (L4_x8.yaml) Enter the number of your choice [hit enter for no profile] [0]: Example output of a completed ilab config init run. You selected: Nvidia A100/H100 x8 (A100_H100_x8.yaml) Initialization completed successfully, you're ready to start using `ilab`. Enjoy! Configuring your system's GPU for inference serving: This step is only required if you are using Red Hat Enterprise Linux AI exclusively for inference serving. Edit your config.yaml file by running the following command: USD ilab config edit In the evaluate section of the configurations file, edit the gpus: parameter and add the number of accelerators on your machine. evaluate: base_branch: null base_model: ~/.cache/instructlab/models/granite-7b-starter branch: null gpus: <num-gpus> In the vllm section of the serve field in the configuration file, edit the gpus: and vllm_args: ["--tensor-parallel-size"] parameters and add the number of accelerators on your machine. serve: backend: vllm chat_template: auto host_port: 127.0.0.1:8000 llama_cpp: gpu_layers: -1 llm_family: '' max_ctx_size: 4096 model_path: ~/.cache/instructlab/models/granite-7b-redhat-lab vllm: llm_family: '' vllm_args: ["--tensor-parallel-size", "<num-gpus>"] gpus: <num-gpus> If you want to use the skeleton taxonomy tree, which includes two skills and one knowledge qna.yaml file, you can clone the skeleton repository and place it in the taxonomy directory by running the following command: rm -rf ~/.local/share/instructlab/taxonomy/ ; git clone https://github.com/RedHatOfficial/rhelai-sample-taxonomy.git ~/.local/share/instructlab/taxonomy/ Directory structure of the InstructLab environment 1 ~/.cache/instructlab/models/ : Contains all downloaded large language models, including the saved output of ones you generate with RHEL AI. 2 ~/.local/share/instructlab/datasets/ : Contains data output from the SDG phase, built on modifications to the taxonomy repository. 3 ~/.local/share/instructlab/taxonomy/ : Contains the skill and knowledge data. 4 ~/.local/share/instructlab/phased/<phase1-or-phase2>/checkpoints/ : Contains the output of the multi-phase training process Verification You can view the full config.yaml file by running the following command USD ilab config show You can also manually edit the config.yaml file by running the following command: USD ilab config edit 2.1.1. Configuring the training profile for AMD accelerators (Technology preview) Red Hat Enterprise Linux AI version 1.2 currently does not have an AMD training profile when you initialize InstructLab. You need to manually set up the training parameters in the config.yaml file to run the end-to-end RHEL AI workflow on machines with AMD accelerators. Important Red Hat Enterprise Linux AI on AMD accelerators is a Technology Preview feature only. Technology Preview features are not supported with Red Hat production service level agreements (SLAs) and might not be functionally complete. Red Hat does not recommend using them in production. These features provide early access to upcoming product features, enabling customers to test functionality and provide feedback during the development process. For more information about the support scope of Red Hat Technology Preview features, see Technology Preview Features Support Scope . Prerequisites You installed RHEL AI with the bootable container image on a machine with AMD accelerators. You have root user access on your machine. You initialized InstructLab and set up your RHEL AI environment. Procedure You can edit the config.yaml file with your prefered text editor or you can run the following command: USD ilab config edit There are currently only a few supported configurations for AMD machines. Edit the config.yaml with the parameters that matches your system. AMD 8xMI300X machines: Update the train and eval parameters to the following values. train: max_batch_len=220000 nproc_per_node=8 eval: gpus=1 AMD 4xMI300X machines: Update the train and eval parameters to the following values. train: max_batch_len=210000 nproc_per_node=4 eval: gpus=1 AMD 2xMI300X machines: Update the train and eval parameters to the following values. train: max_batch_len=200000 nproc_per_node=2 eval: gpus=1
ilab system info ilab config init Welcome to InstructLab CLI. This guide will help you to setup your environment. Detecting Hardware We chose Nvidia 2x A100 as your designated training profile. This is for systems with 160 GB of vRAM. Is this correct? [y/N]: We chose Nvidia 2x A100 as your designated training profile. This is for systems with 160 GB of vRAM. This profile is the best approximation for your system based off of the amount of vRAM. We modified it to match the number of GPUs you have. Is this profile correct? [Y/n]: Please choose a train profile to use. Train profiles assist with the complexity of configuring specific GPU hardware with the InstructLab Training library. You can still take advantage of hardware acceleration for training even if your hardware is not listed. [0] No profile (CPU, Apple Metal, AMD ROCm) [1] Nvidia A100/H100 x2 (A100_H100_x2.yaml) [2] Nvidia A100/H100 x4 (A100_H100_x4.yaml) [3] Nvidia A100/H100 x8 (A100_H100_x8.yaml) [4] Nvidia L40 x4 (L40_x4.yaml) [5] Nvidia L40 x8 (L40_x8.yaml) [6] Nvidia L4 x8 (L4_x8.yaml) Enter the number of your choice [hit enter for no profile] [0]: You selected: Nvidia A100/H100 x8 (A100_H100_x8.yaml) Initialization completed successfully, you're ready to start using `ilab`. Enjoy! ilab config edit evaluate: base_branch: null base_model: ~/.cache/instructlab/models/granite-7b-starter branch: null gpus: <num-gpus> serve: backend: vllm chat_template: auto host_port: 127.0.0.1:8000 llama_cpp: gpu_layers: -1 llm_family: '' max_ctx_size: 4096 model_path: ~/.cache/instructlab/models/granite-7b-redhat-lab vllm: llm_family: '' vllm_args: ["--tensor-parallel-size", "<num-gpus>"] gpus: <num-gpus> rm -rf ~/.local/share/instructlab/taxonomy/ ; git clone https://github.com/RedHatOfficial/rhelai-sample-taxonomy.git ~/.local/share/instructlab/taxonomy/ ├─ ~/.cache/instructlab/models/ 1 ├─ ~/.local/share/instructlab/datasets 2 ├─ ~/.local/share/instructlab/taxonomy 3 ├─ ~/.local/share/instructlab/phased/<phase1-or-phase2>/checkpoints/ 4 ilab config show ilab config edit ilab config edit train: max_batch_len=220000 nproc_per_node=8 eval: gpus=1 train: max_batch_len=210000 nproc_per_node=4 eval: gpus=1 train: max_batch_len=200000 nproc_per_node=2 eval: gpus=1