Chapter 2. Control plane adjustments
Chapter 2. Control plane adjustments The control plane refers to the automation controller pods which contain the web and task containers that, among other things, provide the user interface and handle the scheduling and launching of jobs. On the automation controller custom resource, the number of replicas determines the number of automation controller pods in the automation controller deployment. 2.1. Requests and limits for task containers You must set a value for the task container's CPU and memory resource limits. For each job that is run in an execution node, processing must occur on the control plane to schedule, launch, and receive callback events for that job. For Operator deployments of automation controller, this control plane capacity usage is tracked on the controlplane instance group. The available capacity is determined based on the limits the user sets on the task container, using the task_resource_requirements field in the automation controller specification, or in the OpenShift UI, when creating automation controller. You can also set memory and CPU resource limits that make sense for your cluster. 2.2. Containers resource requirements You can configure the resource requirements of tasks and the web containers, at both the lower end (requests) and the upper end (limits). The execution environment control plane is used for project updates, but is normally the same as the default execution environment for jobs. Setting resource requests and limits is a best practice because a container that has both defined is given a higher Quality of Service class. This means that if the underlying node is resource constrained and the cluster has to reap a pod to prevent running memory or other failure, the control plane pod is less likely to be reaped. These requests and limits apply to the control pods for automation controller and if limits are set, determine the capacity of the instance. By default, controlling a job takes 1 unit of capacity. The memory and CPU limits of the task container are used to determine the capacity of control nodes. For more information on how this is calculated, see Resouce determination . See also Jobs scheduled on the worker nodes Name Description Default web_resource_requirements Web container resource requirements requests: {CPU: 100m, memory: 128Mi} task_resource_requirements Task container resource requirements requests: {CPU: 100m, memory: 128Mi} ee_resource_requirements EE control plane container resource requirements requests: {CPU: 100m, memory: 128Mi} redis_resource_requirements Redis control plane container resource requirements requests: {CPU:100m, memory: 128Mi} Because the use of topology_spread_constraints to maximally spread control nodes onto separate underlying kubernetes worker nodes is also recommended, a reasonable set of requests and limits would be limits whose sum is equal to the actual resources on the node. If only limits are set, then the request is automatically set to be equal to the limit. But because some variablity of resource usage between the containers in the control pod is permitted, you can set requests to a lower amount, for example to 25% of the resources available on the node. An example of container customization for a cluster where the worker nodes have 4 CPUs and 16 GB of RAM could be: spec: ... web_resource_requirements: requests: cpu: 250m memory: 1Gi limits: cpu: 1000m memory: 4Gi task_resource_requirements: requests: cpu: 250m memory: 1Gi limits: cpu: 2000m memory: 4Gi redis_resource_requirements requests: cpu: 250m memory: 1Gi limits: cpu: 1000m memory: 4Gi ee_resource_requirements: requests: cpu: 250m memory: 1Gi limits: cpu: 1000m memory: 4Gi 2.3. Alternative capacity limiting with automation controller settings The capacity of a control node in Openshift is determined by the memory and CPU limits. However, if these are not set then the capacity is determined by the memory and CPU detected by the pod on the filesystem, which are actually the CPU and memory of the underlying Kubernetes node. This can lead to issues with overwhelming the underlying Kubernetes pod if the automation controller pod is not the only pod on that node. If you do not want to set limits directly on the task container, you can use extra_settings , see Extra Settings in Custom pod timeouts section for how to configure the following SYSTEM_TASK_ABS_MEM = 3gi SYSTEM_TASK_ABS_CPU = 750m This acts as a soft limit within the application that enables automation controller to control how much work it attempts to run, while not risking any CPU throttling from Kubernetes itself, or being reaped if memory usage peaks above the desired limit. These settings accept the same format accepted by resource requests and limits in the kubernetes resource definition.
spec: web_resource_requirements: requests: cpu: 250m memory: 1Gi limits: cpu: 1000m memory: 4Gi task_resource_requirements: requests: cpu: 250m memory: 1Gi limits: cpu: 2000m memory: 4Gi redis_resource_requirements requests: cpu: 250m memory: 1Gi limits: cpu: 1000m memory: 4Gi ee_resource_requirements: requests: cpu: 250m memory: 1Gi limits: cpu: 1000m memory: 4Gi SYSTEM_TASK_ABS_MEM = 3gi SYSTEM_TASK_ABS_CPU = 750m