Chapter 13. Accessing monitoring APIs by using the CLI
Chapter 13. Accessing monitoring APIs by using the CLI In OpenShift Container Platform 4.12, you can access web service APIs for some monitoring components from the command line interface (CLI). Important In certain situations, accessing API endpoints can degrade the performance and scalability of your cluster, especially if you use endpoints to retrieve, send, or query large amounts of metrics data. To avoid these issues, follow these recommendations: Avoid querying endpoints frequently. Limit queries to a maximum of one every 30 seconds. Do not try to retrieve all metrics data via the /federate endpoint for Prometheus. Query it only when you want to retrieve a limited, aggregated data set. For example, retrieving fewer than 1,000 samples for each request helps minimize the risk of performance degradation. 13.1. About accessing monitoring web service APIs You can directly access web service API endpoints from the command line for the following monitoring stack components: Prometheus Alertmanager Thanos Ruler Thanos Querier Note To access Thanos Ruler and Thanos Querier service APIs, the requesting account must have get permission on the namespaces resource, which can be granted by binding the cluster-monitoring-view cluster role to the account. When you access web service API endpoints for monitoring components, be aware of the following limitations: You can only use bearer token authentication to access API endpoints. You can only access endpoints in the /api path for a route. If you try to access an API endpoint in a web browser, an Application is not available error occurs. To access monitoring features in a web browser, use the OpenShift Container Platform web console to review monitoring dashboards. Additional resources Reviewing monitoring dashboards 13.2. Accessing a monitoring web service API The following example shows how to query the service API receivers for the Alertmanager service used in core platform monitoring. You can use a similar method to access the prometheus-k8s service for core platform Prometheus and the thanos-ruler service for Thanos Ruler. Prerequisites You are logged in to an account that is bound against the monitoring-alertmanager-edit role in the openshift-monitoring namespace. You are logged in to an account that has permission to get the Alertmanager API route. Note If your account does not have permission to get the Alertmanager API route, a cluster administrator can provide the URL for the route. Procedure Extract an authentication token by running the following command: USD TOKEN=USD(oc whoami -t) Extract the alertmanager-main API route URL by running the following command: USD HOST=USD(oc -n openshift-monitoring get route alertmanager-main -ojsonpath='{.status.ingress[].host}') Query the service API receivers for Alertmanager by running the following command: USD curl -H "Authorization: Bearer USDTOKEN" -k "https://USDHOST/api/v2/receivers" 13.3. Querying metrics by using the federation endpoint for Prometheus You can use the federation endpoint for Prometheus to scrape platform and user-defined metrics from a network location outside the cluster. To do so, access the Prometheus /federate endpoint for the cluster via an OpenShift Container Platform route. Important A delay in retrieving metrics data occurs when you use federation. This delay can affect the accuracy and timeliness of the scraped metrics. Using the federation endpoint can also degrade the performance and scalability of your cluster, especially if you use the federation endpoint to retrieve large amounts of metrics data. To avoid these issues, follow these recommendations: Do not try to retrieve all metrics data via the federation endpoint for Prometheus. Query it only when you want to retrieve a limited, aggregated data set. For example, retrieving fewer than 1,000 samples for each request helps minimize the risk of performance degradation. Avoid frequent querying of the federation endpoint for Prometheus. Limit queries to a maximum of one every 30 seconds. If you need to forward large amounts of data outside the cluster, use remote write instead. For more information, see the Configuring remote write storage section. Prerequisites You have installed the OpenShift CLI ( oc ). You have access to the cluster as a user with the cluster-monitoring-view cluster role or have obtained a bearer token with get permission on the namespaces resource. Note You can only use bearer token authentication to access the Prometheus federation endpoint. You are logged in to an account that has permission to get the Prometheus federation route. Note If your account does not have permission to get the Prometheus federation route, a cluster administrator can provide the URL for the route. Procedure Retrieve the bearer token by running the following the command: USD TOKEN=USD(oc whoami -t) Get the Prometheus federation route URL by running the following command: USD HOST=USD(oc -n openshift-monitoring get route prometheus-k8s-federate -ojsonpath='{.status.ingress[].host}') Query metrics from the /federate route. The following example command queries up metrics: USD curl -G -k -H "Authorization: Bearer USDTOKEN" https://USDHOST/federate --data-urlencode 'match[]=up' Example output # TYPE up untyped up{apiserver="kube-apiserver",endpoint="https",instance="10.0.143.148:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035322214 up{apiserver="kube-apiserver",endpoint="https",instance="10.0.148.166:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035338597 up{apiserver="kube-apiserver",endpoint="https",instance="10.0.173.16:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035343834 ... 13.4. Accessing metrics from outside the cluster for custom applications You can query Prometheus metrics from outside the cluster when monitoring your own services with user-defined projects. Access this data from outside the cluster by using the thanos-querier route. This access only supports using a bearer token for authentication. Prerequisites You have deployed your own service, following the "Enabling monitoring for user-defined projects" procedure. You are logged in to an account with the cluster-monitoring-view cluster role, which provides permission to access the Thanos Querier API. You are logged in to an account that has permission to get the Thanos Querier API route. Note If your account does not have permission to get the Thanos Querier API route, a cluster administrator can provide the URL for the route. Procedure Extract an authentication token to connect to Prometheus by running the following command: USD TOKEN=USD(oc whoami -t) Extract the thanos-querier API route URL by running the following command: USD HOST=USD(oc -n openshift-monitoring get route thanos-querier -ojsonpath='{.status.ingress[].host}') Set the namespace to the namespace in which your service is running by using the following command: USD NAMESPACE=ns1 Query the metrics of your own services in the command line by running the following command: USD curl -H "Authorization: Bearer USDTOKEN" -k "https://USDHOST/api/v1/query?" --data-urlencode "query=up{namespace='USDNAMESPACE'}" The output shows the status for each application pod that Prometheus is scraping: The formatted example output { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "up", "endpoint": "web", "instance": "10.129.0.46:8080", "job": "prometheus-example-app", "namespace": "ns1", "pod": "prometheus-example-app-68d47c4fb6-jztp2", "service": "prometheus-example-app" }, "value": [ 1591881154.748, "1" ] } ], } } Note The formatted example output uses a filtering tool, such as jq , to provide the formatted indented JSON. See the jq Manual (jq documentation) for more information about using jq . The command requests an instant query endpoint of the Thanos Querier service, which evaluates selectors at one point in time. 13.5. Additional resources Enabling monitoring for user-defined projects Configuring remote write storage Managing metrics Managing alerts
TOKEN=USD(oc whoami -t) HOST=USD(oc -n openshift-monitoring get route alertmanager-main -ojsonpath='{.status.ingress[].host}') curl -H "Authorization: Bearer USDTOKEN" -k "https://USDHOST/api/v2/receivers" TOKEN=USD(oc whoami -t) HOST=USD(oc -n openshift-monitoring get route prometheus-k8s-federate -ojsonpath='{.status.ingress[].host}') curl -G -k -H "Authorization: Bearer USDTOKEN" https://USDHOST/federate --data-urlencode 'match[]=up' TYPE up untyped up{apiserver="kube-apiserver",endpoint="https",instance="10.0.143.148:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035322214 up{apiserver="kube-apiserver",endpoint="https",instance="10.0.148.166:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035338597 up{apiserver="kube-apiserver",endpoint="https",instance="10.0.173.16:6443",job="apiserver",namespace="default",service="kubernetes",prometheus="openshift-monitoring/k8s",prometheus_replica="prometheus-k8s-0"} 1 1657035343834 TOKEN=USD(oc whoami -t) HOST=USD(oc -n openshift-monitoring get route thanos-querier -ojsonpath='{.status.ingress[].host}') NAMESPACE=ns1 curl -H "Authorization: Bearer USDTOKEN" -k "https://USDHOST/api/v1/query?" --data-urlencode "query=up{namespace='USDNAMESPACE'}" { "status": "success", "data": { "resultType": "vector", "result": [ { "metric": { "__name__": "up", "endpoint": "web", "instance": "10.129.0.46:8080", "job": "prometheus-example-app", "namespace": "ns1", "pod": "prometheus-example-app-68d47c4fb6-jztp2", "service": "prometheus-example-app" }, "value": [ 1591881154.748, "1" ] } ], } }