Chapter 3. Setting up TrustyAI for your project
Chapter 3. Setting up TrustyAI for your project To set up model monitoring with TrustyAI for a data science project, a data scientist does the following tasks: Authenticate the TrustyAI service Send training data to TrustyAI for bias or data drift monitoring Label your data fields (optional) After setting up, a data scientist can create and view bias and data drift metrics for deployed models. 3.1. Authenticating the TrustyAI service To access TrustyAI service external endpoints, you must provide OAuth proxy (oauth-proxy) authentication. You must obtain a user token, or a token from a service account with sufficient privileges, and then pass the token to the TrustyAI service when using curl commands. Prerequisites You installed the OpenShift command line interface ( oc ) as described in Installing the OpenShift CLI (OpenShift Dedicated) or Installing the OpenShift CLI (Red Hat OpenShift Service on AWS) . Your OpenShift cluster administrator added you as a user to the OpenShift cluster and has installed the TrustyAI service for the data science project that contains the deployed models. Procedure Open a new terminal window. Follow these steps to log in to your OpenShift cluster: In the upper-right corner of the OpenShift web console, click your user name and select Copy login command . After you have logged in, click Display token . Copy the Log in with this token command and paste it in the OpenShift command-line interface (CLI). Enter the following command to set a user token variable on OpenShift: Verification Enter the following command to check the user token variable: step When running curl commands, pass the token to the TrustyAI service using the Authorization header. For example: 3.2. Sending training data to TrustyAI To use TrustyAI for bias monitoring or data drift detection, you must send training data for your model to TrustyAI. Prerequisites Your OpenShift cluster administrator added you as a user to the OpenShift cluster and has installed the TrustyAI service for the data science project that contains the deployed models. You authenticated the TrustyAI service as described in Authenticating the TrustyAI service . Your deployed model is registered with TrustyAI. Verify that the TrustyAI service has registered your deployed model, as follows: In the OpenShift web console, navigate to Workloads Pods . From the project list, select the project that contains your deployed model. Select the pod for your serving platform (for example, modelmesh-serving-ovms-1.x-xxxxx ). On the Environment tab, verify that the MM_PAYLOAD_PROCESSORS environment variable is set. Procedure Set the TRUSTY_ROUTE variable to the external route for the TrustyAI service pod. Get the inference endpoints for the deployed model, as described in Accessing the inference endpoint for a deployed model . Send data to this endpoint. For more information, see the KServe v2 Inference Protocol documentation . Verification Follow these steps to view cluster metrics and verify that TrustyAI is receiving data. Log in to the OpenShift web console. Switch to the Developer perspective. In the left menu, click Observe . On the Metrics page, click the Select query list and then select Custom query . In the Expression field, enter trustyai_model_observations_total and press Enter. Your model should be listed and reporting observed inferences. Optional: Select a time range from the list above the graph. For example, select 5m . 3.3. Labeling data fields After you send model training data to TrustyAI, you might want to apply a set of name mappings to your inputs and outputs so that the field names are meaningful and easier to work with. Prerequisites Your OpenShift cluster administrator added you as a user to the OpenShift cluster and has installed the TrustyAI service for the data science project that contains the deployed models. You sent training data to TrustyAI as described in Sending training data to TrustyAI . Procedure Open a new terminal window. Follow these steps to log in to your OpenShift cluster: In the upper-right corner of the OpenShift web console, click your user name and select Copy login command . After you have logged in, click Display token . Copy the Log in with this token command and paste it in the OpenShift command-line interface (CLI). In the OpenShift CLI, get the route to the TrustyAI service: To examine TrustyAI's model metadata, query the /info endpoint: This outputs a JSON file containing the following information for each model: The names, data types, and positions of input fields and output fields. The observed field values. The total number of input-output pairs observed. Use POST /info/names to apply name mappings to the fields, similar to the following example. Change the model-name , original-name , and Prediction values to those used in your model. Change the New name values to the labels that you want to use. For another example, see https://github.com/trustyai-explainability/odh-trustyai-demos/blob/main/2-BiasMonitoring/kserve-demo/scripts/apply_name_mapping.sh . Verification A "Feature and output name mapping successfully applied" message is displayed.
oc login --token= <token> --server= <openshift_cluster_url> export TOKEN=USD(oc whoami -t) echo USDTOKEN curl -H "Authorization: Bearer USDTOKEN" USDTRUSTY_ROUTE TRUSTY_ROUTE=https://USD(oc get route/trustyai-service --template={{.spec.host}}) oc login --token= <token> --server= <openshift_cluster_url> TRUSTY_ROUTE=https://USD(oc get route/trustyai-service --template={{.spec.host}}) curl -H "Authorization: Bearer USDTOKEN" USDTRUSTY_ROUTE/info | jq ".[0].data" curl -sk -H "Authorization: Bearer USDTOKEN" -X POST --location USDTRUSTY_ROUTE/info/names -H "Content-Type: application/json" -d "{ \"modelId\": \"model-name\", \"inputMapping\": { \"original-name-0\": \"New name 0\", \"original-name-1\": \"New name 1\", \"original-name-2\": \"New name 2\", \"original-name-3\": \"New name 3\", }, \"outputMapping\": { \"predict-0\": \"Prediction 0\" } }"