Chapter 3. Editing kubelet log level verbosity and gathering logs
Chapter 3. Editing kubelet log level verbosity and gathering logs To troubleshoot some issues with nodes, establish the kubelet's log level verbosity depending on the issue to be tracked. 3.1. Modifying the kubelet as a one-time scenario To modify the kubelet in a one-time scenario without rebooting the node due to the change of machine-config(spec":{"paused":false}}) , allowing you to modify the kubelet without affecting the service, follow this procedure. Procedure Connect to the node in debug mode: USD oc debug node/<node> USD chroot /host Alternatively, it is possible to SSH to the node and become root. After access is established, check the default log level: USD systemctl cat kubelet Example output # /etc/systemd/system/kubelet.service.d/20-logging.conf [Service] Environment="KUBELET_LOG_LEVEL=2" Define the new verbosity required in a new /etc/systemd/system/kubelet.service.d/30-logging.conf file, which overrides /etc/systemd/system/kubelet.service.d/20-logging.conf . In this example, the verbosity is changed from 2 to 8 : USD echo -e "[Service]\nEnvironment=\"KUBELET_LOG_LEVEL=8\"" > /etc/systemd/system/kubelet.service.d/30-logging.conf Reload systemd and restart the service: USD systemctl daemon-reload USD systemctl restart kubelet Gather the logs, and then revert the log level increase: USD rm -f /etc/systemd/system/kubelet.service.d/30-logging.conf USD systemctl daemon-reload USD systemctl restart kubelet 3.2. Persistent kubelet log level configuration Procedure Use the following MachineConfig object for persistent kubelet log level configuration: apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: master name: 99-master-kubelet-loglevel spec: config: ignition: version: 3.2.0 systemd: units: - name: kubelet.service enabled: true dropins: - name: 30-logging.conf contents: | [Service] Environment="KUBELET_LOG_LEVEL=2" Generally, it is recommended to apply 0-4 as debug-level logs and 5-8 as trace-level logs. 3.3. Log verbosity descriptions Log verbosity Description --v=0 Always visible to an Operator. --v=1 A reasonable default log level if you do not want verbosity. --v=2 Useful steady state information about the service and important log messages that might correlate to significant changes in the system. This is the recommended default log level. --v=3 Extended information about changes. --v=4 Debug level verbosity. --v=6 Display requested resources. --v=7 Display HTTP request headers. --v=8 Display HTTP request contents. 3.4. Gathering kubelet logs Procedure After the kubelet's log level verbosity is configured properly, you can gather logs by running the following commands: USD oc adm node-logs --role master -u kubelet USD oc adm node-logs --role worker -u kubelet Alternatively, inside the node, run the following command: USD journalctl -b -f -u kubelet.service To collect master container logs, run the following command: USD sudo tail -f /var/log/containers/* To directly gather the logs of all nodes, run the following command: - for n in USD(oc get node --no-headers | awk '{print USD1}'); do oc adm node-logs USDn | gzip > USDn.log.gz; done
oc debug node/<node> chroot /host systemctl cat kubelet /etc/systemd/system/kubelet.service.d/20-logging.conf [Service] Environment="KUBELET_LOG_LEVEL=2" echo -e "[Service]\nEnvironment=\"KUBELET_LOG_LEVEL=8\"" > /etc/systemd/system/kubelet.service.d/30-logging.conf systemctl daemon-reload systemctl restart kubelet rm -f /etc/systemd/system/kubelet.service.d/30-logging.conf systemctl daemon-reload systemctl restart kubelet apiVersion: machineconfiguration.openshift.io/v1 kind: MachineConfig metadata: labels: machineconfiguration.openshift.io/role: master name: 99-master-kubelet-loglevel spec: config: ignition: version: 3.2.0 systemd: units: - name: kubelet.service enabled: true dropins: - name: 30-logging.conf contents: | [Service] Environment="KUBELET_LOG_LEVEL=2" oc adm node-logs --role master -u kubelet oc adm node-logs --role worker -u kubelet journalctl -b -f -u kubelet.service sudo tail -f /var/log/containers/* - for n in USD(oc get node --no-headers | awk '{print USD1}'); do oc adm node-logs USDn | gzip > USDn.log.gz; done