Chapter 19. Rebalancing clusters using Cruise Control
Chapter 19. Rebalancing clusters using Cruise Control Cruise Control is an open source system that supports the following Kafka operations: Monitoring cluster workload Rebalancing a cluster based on predefined constraints The operations help with running a more balanced Kafka cluster that uses broker pods more efficiently. A typical cluster can become unevenly loaded over time. Partitions that handle large amounts of message traffic might not be evenly distributed across the available brokers. To rebalance the cluster, administrators must monitor the load on brokers and manually reassign busy partitions to brokers with spare capacity. Cruise Control automates the cluster rebalancing process. It constructs a workload model of resource utilization for the cluster- based on CPU, disk, and network load- and generates optimization proposals (that you can approve or reject) for more balanced partition assignments. A set of configurable optimization goals is used to calculate these proposals. You can generate optimization proposals in specific modes. The default full mode rebalances partitions across all brokers. You can also use the add-brokers and remove-brokers modes to accommodate changes when scaling a cluster up or down. When you approve an optimization proposal, Cruise Control applies it to your Kafka cluster. You configure and generate optimization proposals using a KafkaRebalance resource. You can configure the resource using an annotation so that optimization proposals are approved automatically or manually. Note Streams for Apache Kafka provides example configuration files for Cruise Control . 19.1. Cruise Control components and features Cruise Control consists of four main components- the Load Monitor, the Analyzer, the Anomaly Detector, and the Executor- and a REST API for client interactions. Streams for Apache Kafka utilizes the REST API to support the following Cruise Control features: Generating optimization proposals from optimization goals. Rebalancing a Kafka cluster based on an optimization proposal. Optimization goals An optimization goal describes a specific objective to achieve from a rebalance. For example, a goal might be to distribute topic replicas across brokers more evenly. You can change what goals to include through configuration. A goal is defined as a hard goal or soft goal. You can add hard goals through Cruise Control deployment configuration. You also have main, default, and user-provided goals that fit into each of these categories. Hard goals are preset and must be satisfied for an optimization proposal to be successful. Soft goals do not need to be satisfied for an optimization proposal to be successful. They can be set aside if it means that all hard goals are met. Main goals are inherited from Cruise Control. Some are preset as hard goals. Main goals are used in optimization proposals by default. Default goals are the same as the main goals by default. You can specify your own set of default goals. User-provided goals are a subset of default goals that are configured for generating a specific optimization proposal. Optimization proposals Optimization proposals comprise the goals you want to achieve from a rebalance. You generate an optimization proposal to create a summary of proposed changes and the results that are possible with the rebalance. The goals are assessed in a specific order of priority. You can then choose to approve or reject the proposal. You can reject the proposal to run it again with an adjusted set of goals. You can generate an optimization proposal in one of three modes. full is the default mode and runs a full rebalance. add-brokers is the mode you use after adding brokers when scaling up a Kafka cluster. remove-brokers is the mode you use before removing brokers when scaling down a Kafka cluster. Other Cruise Control features are not currently supported, including self healing, notifications, write-your-own goals, and changing the topic replication factor. Additional resources Cruise Control documentation 19.2. Optimization goals overview Optimization goals are constraints on workload redistribution and resource utilization across a Kafka cluster. To rebalance a Kafka cluster, Cruise Control uses optimization goals to generate optimization proposals , which you can approve or reject. 19.2.1. Goals order of priority Streams for Apache Kafka supports most of the optimization goals developed in the Cruise Control project. The supported goals, in the default descending order of priority, are as follows: Rack-awareness Minimum number of leader replicas per broker for a set of topics Replica capacity Capacity goals Disk capacity Network inbound capacity Network outbound capacity CPU capacity Replica distribution Potential network output Resource distribution goals Disk utilization distribution Network inbound utilization distribution Network outbound utilization distribution CPU utilization distribution Leader bytes-in rate distribution Topic replica distribution Leader replica distribution Preferred leader election Intra-broker disk capacity Intra-broker disk usage distribution For more information on each optimization goal, see Goals in the Cruise Control Wiki. Note "Write your own" goals and Kafka assigner goals are not yet supported. 19.2.2. Goals configuration in Streams for Apache Kafka custom resources You configure optimization goals in Kafka and KafkaRebalance custom resources. Cruise Control has configurations for hard optimization goals that must be satisfied, as well as main, default, and user-provided optimization goals. You can specify optimization goals in the following configuration: Main goals - Kafka.spec.cruiseControl.config.goals Hard goals - Kafka.spec.cruiseControl.config.hard.goals Default goals - Kafka.spec.cruiseControl.config.default.goals User-provided goals - KafkaRebalance.spec.goals Note Resource distribution goals are subject to capacity limits on broker resources. 19.2.3. Hard and soft optimization goals Hard goals are goals that must be satisfied in optimization proposals. Goals that are not defined as hard goals in the Cruise Control code are known as soft goals . You can think of soft goals as best effort goals: they do not need to be satisfied in optimization proposals, but are included in optimization calculations. An optimization proposal that violates one or more soft goals, but satisfies all hard goals, is valid. Cruise Control will calculate optimization proposals that satisfy all the hard goals and as many soft goals as possible (in their priority order). An optimization proposal that does not satisfy all the hard goals is rejected by Cruise Control and not sent to the user for approval. Note For example, you might have a soft goal to distribute a topic's replicas evenly across the cluster (the topic replica distribution goal). Cruise Control will ignore this goal if doing so enables all the configured hard goals to be met. In Cruise Control, the following main optimization goals are hard goals: In your Cruise Control deployment configuration, you can specify which hard goals to enforce using the hard.goals property in Kafka.spec.cruiseControl.config . To enforce execution of all hard goals, simply omit the hard.goals property. To change which hard goals Cruise Control enforces, specify the required goals in the hard.goals property using their fully-qualified domain names. To prevent execution of a specific hard goal, ensure that the goal is not included in both the default.goals and hard.goals list configurations. Note It is not possible to configure which goals are considered soft or hard goals. This distinction is determined by the Cruise Control code. Example Kafka configuration for hard optimization goals apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: # ... zookeeper: # ... entityOperator: topicOperator: {} userOperator: {} cruiseControl: brokerCapacity: inboundNetwork: 10000KB/s outboundNetwork: 10000KB/s config: # Note that `default.goals` (superset) must also include all `hard.goals` (subset) default.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal hard.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal # ... Increasing the number of configured hard goals will reduce the likelihood of Cruise Control generating valid optimization proposals. If skipHardGoalCheck: true is specified in the KafkaRebalance custom resource, Cruise Control does not check that the list of user-provided optimization goals (in KafkaRebalance.spec.goals ) contains all the configured hard goals ( hard.goals ). Therefore, if some, but not all, of the user-provided optimization goals are in the hard.goals list, Cruise Control will still treat them as hard goals even if skipHardGoalCheck: true is specified. 19.2.4. Main optimization goals The main optimization goals are available to all users. Goals that are not listed in the main optimization goals are not available for use in Cruise Control operations. Unless you change the Cruise Control deployment configuration , Streams for Apache Kafka will inherit the following main optimization goals from Cruise Control, in descending priority order: Some of these goals are preset as hard goals . To reduce complexity, we recommend that you use the inherited main optimization goals, unless you need to completely exclude one or more goals from use in KafkaRebalance resources. The priority order of the main optimization goals can be modified, if desired, in the configuration for default optimization goals . You configure main optimization goals, if necessary, in the Cruise Control deployment configuration: Kafka.spec.cruiseControl.config.goals To accept the inherited main optimization goals, do not specify the goals property in Kafka.spec.cruiseControl.config . If you need to modify the inherited main optimization goals, specify a list of goals, in descending priority order, in the goals configuration option. Note To avoid errors when generating optimization proposals, make sure that any changes you make to the goals or default.goals in Kafka.spec.cruiseControl.config include all of the hard goals specified for the hard.goals property. To clarify, the hard goals must also be specified (as a subset) for the main optimization goals and default goals. 19.2.5. Default optimization goals Cruise Control uses the default optimization goals to generate the cached optimization proposal . For more information about the cached optimization proposal, see Section 19.3, "Optimization proposals overview" . You can override the default optimization goals by setting user-provided optimization goals in a KafkaRebalance custom resource. Unless you specify default.goals in the Cruise Control deployment configuration , the main optimization goals are used as the default optimization goals. In this case, the cached optimization proposal is generated using the main optimization goals. To use the main optimization goals as the default goals, do not specify the default.goals property in Kafka.spec.cruiseControl.config . To modify the default optimization goals, edit the default.goals property in Kafka.spec.cruiseControl.config . You must use a subset of the main optimization goals. Example Kafka configuration for default optimization goals apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: # ... zookeeper: # ... entityOperator: topicOperator: {} userOperator: {} cruiseControl: brokerCapacity: inboundNetwork: 10000KB/s outboundNetwork: 10000KB/s config: # Note that `default.goals` (superset) must also include all `hard.goals` (subset) default.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal hard.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal # ... If no default optimization goals are specified, the cached proposal is generated using the main optimization goals. 19.2.6. User-provided optimization goals User-provided optimization goals narrow down the configured default goals for a particular optimization proposal. You can set them, as required, in spec.goals in a KafkaRebalance custom resource: User-provided optimization goals can generate optimization proposals for different scenarios. For example, you might want to optimize leader replica distribution across the Kafka cluster without considering disk capacity or disk utilization. So, you create a KafkaRebalance custom resource containing a single user-provided goal for leader replica distribution. User-provided optimization goals must: Include all configured hard goals , or an error occurs Be a subset of the main optimization goals To ignore the configured hard goals when generating an optimization proposal, add the skipHardGoalCheck: true property to the KafkaRebalance custom resource. See Section 19.6, "Generating optimization proposals" . Additional resources Configuring and deploying Cruise Control with Kafka Configurations in the Cruise Control Wiki. 19.3. Optimization proposals overview Configure a KafkaRebalance resource to generate optimization proposals and apply the suggested changes. An optimization proposal is a summary of proposed changes that would produce a more balanced Kafka cluster, with partition workloads distributed more evenly among the brokers. Each optimization proposal is based on the set of optimization goals that was used to generate it, subject to any configured capacity limits on broker resources . All optimization proposals are estimates of the impact of a proposed rebalance. You can approve or reject a proposal. You cannot approve a cluster rebalance without first generating the optimization proposal. You can run optimization proposals in one of the following rebalancing modes: full add-brokers remove-brokers 19.3.1. Rebalancing modes You specify a rebalancing mode using the spec.mode property of the KafkaRebalance custom resource. full The full mode runs a full rebalance by moving replicas across all the brokers in the cluster. This is the default mode if the spec.mode property is not defined in the KafkaRebalance custom resource. add-brokers The add-brokers mode is used after scaling up a Kafka cluster by adding one or more brokers. Normally, after scaling up a Kafka cluster, new brokers are used to host only the partitions of newly created topics. If no new topics are created, the newly added brokers are not used and the existing brokers remain under the same load. By using the add-brokers mode immediately after adding brokers to the cluster, the rebalancing operation moves replicas from existing brokers to the newly added brokers. You specify the new brokers as a list using the spec.brokers property of the KafkaRebalance custom resource. remove-brokers The remove-brokers mode is used before scaling down a Kafka cluster by removing one or more brokers. If you scale down a Kafka cluster, brokers are shut down even if they host replicas. This can lead to under-replicated partitions and possibly result in some partitions being under their minimum ISR (in-sync replicas). To avoid this potential problem, the remove-brokers mode moves replicas off the brokers that are going to be removed. When these brokers are not hosting replicas anymore, you can safely run the scaling down operation. You specify the brokers you're removing as a list in the spec.brokers property in the KafkaRebalance custom resource. In general, use the full rebalance mode to rebalance a Kafka cluster by spreading the load across brokers. Use the add-brokers and remove-brokers modes only if you want to scale your cluster up or down and rebalance the replicas accordingly. The procedure to run a rebalance is actually the same across the three different modes. The only difference is with specifying a mode through the spec.mode property and, if needed, listing brokers that have been added or will be removed through the spec.brokers property. 19.3.2. The results of an optimization proposal When an optimization proposal is generated, a summary and broker load is returned. Summary The summary is contained in the KafkaRebalance resource. The summary provides an overview of the proposed cluster rebalance and indicates the scale of the changes involved. A summary of a successfully generated optimization proposal is contained in the Status.OptimizationResult property of the KafkaRebalance resource. The information provided is a summary of the full optimization proposal. Broker load The broker load is stored in a ConfigMap that contains data as a JSON string. The broker load shows before and after values for the proposed rebalance, so you can see the impact on each of the brokers in the cluster. 19.3.3. Manually approving or rejecting an optimization proposal An optimization proposal summary shows the proposed scope of changes. You can use the name of the KafkaRebalance resource to return a summary from the command line. Returning an optimization proposal summary oc describe kafkarebalance <kafka_rebalance_resource_name> -n <namespace> You can also use the jq command line JSON parser tool . Returning an optimization proposal summary using jq oc get kafkarebalance -o json | jq <jq_query> . Use the summary to decide whether to approve or reject an optimization proposal. Approving an optimization proposal You approve the optimization proposal by setting the strimzi.io/rebalance annotation of the KafkaRebalance resource to approve . Cruise Control applies the proposal to the Kafka cluster and starts a cluster rebalance operation. Rejecting an optimization proposal If you choose not to approve an optimization proposal, you can change the optimization goals or update any of the rebalance performance tuning options , and then generate another proposal. You can generate a new optimization proposal for a KafkaRebalance resource by setting the strimzi.io/rebalance annotation to refresh . Use optimization proposals to assess the movements required for a rebalance. For example, a summary describes inter-broker and intra-broker movements. Inter-broker rebalancing moves data between separate brokers. Intra-broker rebalancing moves data between disks on the same broker when you are using a JBOD storage configuration. Such information can be useful even if you don't go ahead and approve the proposal. You might reject an optimization proposal, or delay its approval, because of the additional load on a Kafka cluster when rebalancing. In the following example, the proposal suggests the rebalancing of data between separate brokers. The rebalance involves the movement of 55 partition replicas, totaling 12MB of data, across the brokers. Though the inter-broker movement of partition replicas has a high impact on performance, the total amount of data is not large. If the total data was much larger, you could reject the proposal, or time when to approve the rebalance to limit the impact on the performance of the Kafka cluster. Rebalance performance tuning options can help reduce the impact of data movement. If you can extend the rebalance period, you can divide the rebalance into smaller batches. Fewer data movements at a single time reduces the load on the cluster. Example optimization proposal summary Name: my-rebalance Namespace: myproject Labels: strimzi.io/cluster=my-cluster Annotations: API Version: kafka.strimzi.io/v1alpha1 Kind: KafkaRebalance Metadata: # ... Status: Conditions: Last Transition Time: 2022-04-05T14:36:11.900Z Status: ProposalReady Type: State Observed Generation: 1 Optimization Result: Data To Move MB: 0 Excluded Brokers For Leadership: Excluded Brokers For Replica Move: Excluded Topics: Intra Broker Data To Move MB: 12 Monitored Partitions Percentage: 100 Num Intra Broker Replica Movements: 0 Num Leader Movements: 24 Num Replica Movements: 55 On Demand Balancedness Score After: 82.91290759174306 On Demand Balancedness Score Before: 78.01176356230222 Recent Windows: 5 Session Id: a4f833bd-2055-4213-bfdd-ad21f95bf184 The proposal will also move 24 partition leaders to different brokers. This requires a change to the ZooKeeper configuration, which has a low impact on performance. The balancedness scores are measurements of the overall balance of the Kafka cluster before and after the optimization proposal is approved. A balancedness score is based on optimization goals. If all goals are satisfied, the score is 100. The score is reduced for each goal that will not be met. Compare the balancedness scores to see whether the Kafka cluster is less balanced than it could be following a rebalance. 19.3.4. Automatically approving an optimization proposal To save time, you can automate the process of approving optimization proposals. With automation, when you generate an optimization proposal it goes straight into a cluster rebalance. To enable the optimization proposal auto-approval mechanism, create the KafkaRebalance resource with the strimzi.io/rebalance-auto-approval annotation set to true . If the annotation is not set or set to false , the optimization proposal requires manual approval. Example rebalance request with auto-approval mechanism enabled apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster annotations: strimzi.io/rebalance-auto-approval: "true" spec: mode: # any mode # ... You can still check the status when automatically approving an optimization proposal. The status of the KafkaRebalance resource moves to Ready when the rebalance is complete. 19.3.5. Optimization proposal summary properties The following table explains the properties contained in the optimization proposal's summary section. Table 19.1. Properties contained in an optimization proposal summary JSON property Description numIntraBrokerReplicaMovements The total number of partition replicas that will be transferred between the disks of the cluster's brokers. Performance impact during rebalance operation : Relatively high, but lower than numReplicaMovements . excludedBrokersForLeadership Not yet supported. An empty list is returned. numReplicaMovements The number of partition replicas that will be moved between separate brokers. Performance impact during rebalance operation : Relatively high. onDemandBalancednessScoreBefore, onDemandBalancednessScoreAfter A measurement of the overall balancedness of a Kafka Cluster, before and after the optimization proposal was generated. The score is calculated by subtracting the sum of the BalancednessScore of each violated soft goal from 100. Cruise Control assigns a BalancednessScore to every optimization goal based on several factors, including priority- the goal's position in the list of default.goals or user-provided goals. The Before score is based on the current configuration of the Kafka cluster. The After score is based on the generated optimization proposal. intraBrokerDataToMoveMB The sum of the size of each partition replica that will be moved between disks on the same broker (see also numIntraBrokerReplicaMovements ). Performance impact during rebalance operation : Variable. The larger the number, the longer the cluster rebalance will take to complete. Moving a large amount of data between disks on the same broker has less impact than between separate brokers (see dataToMoveMB ). recentWindows The number of metrics windows upon which the optimization proposal is based. dataToMoveMB The sum of the size of each partition replica that will be moved to a separate broker (see also numReplicaMovements ). Performance impact during rebalance operation : Variable. The larger the number, the longer the cluster rebalance will take to complete. monitoredPartitionsPercentage The percentage of partitions in the Kafka cluster covered by the optimization proposal. Affected by the number of excludedTopics . excludedTopics If you specified a regular expression in the spec.excludedTopicsRegex property in the KafkaRebalance resource, all topic names matching that expression are listed here. These topics are excluded from the calculation of partition replica/leader movements in the optimization proposal. numLeaderMovements The number of partitions whose leaders will be switched to different replicas. This involves a change to ZooKeeper configuration. Performance impact during rebalance operation : Relatively low. excludedBrokersForReplicaMove Not yet supported. An empty list is returned. 19.3.6. Broker load properties The broker load is stored in a ConfigMap (with the same name as the KafkaRebalance custom resource) as a JSON formatted string. This JSON string consists of a JSON object with keys for each broker IDs linking to a number of metrics for each broker. Each metric consist of three values. The first is the metric value before the optimization proposal is applied, the second is the expected value of the metric after the proposal is applied, and the third is the difference between the first two values (after minus before). Note The ConfigMap appears when the KafkaRebalance resource is in the ProposalReady state and remains after the rebalance is complete. You can use the name of the ConfigMap to view its data from the command line. Returning ConfigMap data oc describe configmaps <my_rebalance_configmap_name> -n <namespace> You can also use the jq command line JSON parser tool to extract the JSON string from the ConfigMap. Extracting the JSON string from the ConfigMap using jq oc get configmaps <my_rebalance_configmap_name> -o json | jq '.["data"]["brokerLoad.json"]|fromjson|.' The following table explains the properties contained in the optimization proposal's broker load ConfigMap: JSON property Description leaders The number of replicas on this broker that are partition leaders. replicas The number of replicas on this broker. cpuPercentage The CPU utilization as a percentage of the defined capacity. diskUsedPercentage The disk utilization as a percentage of the defined capacity. diskUsedMB The absolute disk usage in MB. networkOutRate The total network output rate for the broker. leaderNetworkInRate The network input rate for all partition leader replicas on this broker. followerNetworkInRate The network input rate for all follower replicas on this broker. potentialMaxNetworkOutRate The hypothetical maximum network output rate that would be realized if this broker became the leader of all the replicas it currently hosts. 19.3.7. Cached optimization proposal Cruise Control maintains a cached optimization proposal based on the configured default optimization goals. Generated from the workload model, the cached optimization proposal is updated every 15 minutes to reflect the current state of the Kafka cluster. If you generate an optimization proposal using the default optimization goals, Cruise Control returns the most recent cached proposal. To change the cached optimization proposal refresh interval, edit the proposal.expiration.ms setting in the Cruise Control deployment configuration. Consider a shorter interval for fast changing clusters, although this increases the load on the Cruise Control server. Additional resources Section 19.2, "Optimization goals overview" Section 19.6, "Generating optimization proposals" Section 19.7, "Approving an optimization proposal" 19.4. Rebalance performance tuning overview You can adjust several performance tuning options for cluster rebalances. These options control how partition replicas and leadership movements in a rebalance are executed, as well as the bandwidth that is allocated to a rebalance operation. 19.4.1. Partition reassignment commands Optimization proposals are comprised of separate partition reassignment commands. When you approve a proposal, the Cruise Control server applies these commands to the Kafka cluster. A partition reassignment command consists of either of the following types of operations: Partition movement: Involves transferring the partition replica and its data to a new location. Partition movements can take one of two forms: Inter-broker movement: The partition replica is moved to a log directory on a different broker. Intra-broker movement: The partition replica is moved to a different log directory on the same broker. Leadership movement: This involves switching the leader of the partition's replicas. Cruise Control issues partition reassignment commands to the Kafka cluster in batches. The performance of the cluster during the rebalance is affected by the number of each type of movement contained in each batch. 19.4.2. Replica movement strategies Cluster rebalance performance is also influenced by the replica movement strategy that is applied to the batches of partition reassignment commands. By default, Cruise Control uses the BaseReplicaMovementStrategy , which simply applies the commands in the order they were generated. However, if there are some very large partition reassignments early in the proposal, this strategy can slow down the application of the other reassignments. Cruise Control provides four alternative replica movement strategies that can be applied to optimization proposals: PrioritizeSmallReplicaMovementStrategy : Order reassignments in order of ascending size. PrioritizeLargeReplicaMovementStrategy : Order reassignments in order of descending size. PostponeUrpReplicaMovementStrategy : Prioritize reassignments for replicas of partitions which have no out-of-sync replicas. PrioritizeMinIsrWithOfflineReplicasStrategy : Prioritize reassignments with (At/Under)MinISR partitions with offline replicas. This strategy will only work if cruiseControl.config.concurrency.adjuster.min.isr.check.enabled is set to true in the Kafka custom resource's spec. These strategies can be configured as a sequence. The first strategy attempts to compare two partition reassignments using its internal logic. If the reassignments are equivalent, then it passes them to the strategy in the sequence to decide the order, and so on. 19.4.3. Intra-broker disk balancing Moving a large amount of data between disks on the same broker has less impact than between separate brokers. If you are running a Kafka deployment that uses JBOD storage with multiple disks on the same broker, Cruise Control can balance partitions between the disks. Note If you are using JBOD storage with a single disk, intra-broker disk balancing will result in a proposal with 0 partition movements since there are no disks to balance between. To perform an intra-broker disk balance, set rebalanceDisk to true under the KafkaRebalance.spec . When setting rebalanceDisk to true , do not set a goals field in the KafkaRebalance.spec , as Cruise Control will automatically set the intra-broker goals and ignore the inter-broker goals. Cruise Control does not perform inter-broker and intra-broker balancing at the same time. 19.4.4. Rebalance tuning options Cruise Control provides several configuration options for tuning the rebalance parameters discussed above. You can set these tuning options when configuring and deploying Cruise Control with Kafka or optimization proposal levels: The Cruise Control server setting can be set in the Kafka custom resource under Kafka.spec.cruiseControl.config . The individual rebalance performance configurations can be set under KafkaRebalance.spec . The relevant configurations are summarized in the following table. Table 19.2. Rebalance performance tuning configuration Cruise Control properties KafkaRebalance properties Default Description num.concurrent.partition.movements.per.broker concurrentPartitionMovementsPerBroker 5 The maximum number of inter-broker partition movements in each partition reassignment batch num.concurrent.intra.broker.partition.movements concurrentIntraBrokerPartitionMovements 2 The maximum number of intra-broker partition movements in each partition reassignment batch num.concurrent.leader.movements concurrentLeaderMovements 1000 The maximum number of partition leadership changes in each partition reassignment batch default.replication.throttle replicationThrottle Null (no limit) The bandwidth (in bytes per second) to assign to partition reassignment default.replica.movement.strategies replicaMovementStrategies BaseReplicaMovementStrategy The list of strategies (in priority order) used to determine the order in which partition reassignment commands are executed for generated proposals. For the server setting, use a comma separated string with the fully qualified names of the strategy class (add com.linkedin.kafka.cruisecontrol.executor.strategy. to the start of each class name). For the KafkaRebalance resource setting use a YAML array of strategy class names. - rebalanceDisk false Enables intra-broker disk balancing, which balances disk space utilization between disks on the same broker. Only applies to Kafka deployments that use JBOD storage with multiple disks. Changing the default settings affects the length of time that the rebalance takes to complete, as well as the load placed on the Kafka cluster during the rebalance. Using lower values reduces the load but increases the amount of time taken, and vice versa. Additional resources CruiseControlSpec schema reference KafkaRebalanceSpec schema reference 19.5. Configuring and deploying Cruise Control with Kafka Configure a Kafka resource to deploy Cruise Control alongside a Kafka cluster. You can use the cruiseControl properties of the Kafka resource to configure the deployment. Deploy one instance of Cruise Control per Kafka cluster. Use goals configuration in the Cruise Control config to specify optimization goals for generating optimization proposals. You can use brokerCapacity to change the default capacity limits for goals related to resource distribution. If brokers are running on nodes with heterogeneous network resources, you can use overrides to set network capacity limits for each broker. If an empty object ( {} ) is used for the cruiseControl configuration, all properties use their default values. For more information on the configuration options for Cruise Control, see the Streams for Apache Kafka Custom Resource API Reference . Prerequisites An OpenShift cluster A running Cluster Operator Procedure Edit the cruiseControl property for the Kafka resource. The properties you can configure are shown in this example configuration: apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: # ... cruiseControl: brokerCapacity: 1 inboundNetwork: 10000KB/s outboundNetwork: 10000KB/s overrides: 2 - brokers: [0] inboundNetwork: 20000KiB/s outboundNetwork: 20000KiB/s - brokers: [1, 2] inboundNetwork: 30000KiB/s outboundNetwork: 30000KiB/s # ... config: 3 # Note that `default.goals` (superset) must also include all `hard.goals` (subset) default.goals: > 4 com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal # ... hard.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal # ... cpu.balance.threshold: 1.1 metadata.max.age.ms: 300000 send.buffer.bytes: 131072 webserver.http.cors.enabled: true 5 webserver.http.cors.origin: "*" webserver.http.cors.exposeheaders: "User-Task-ID,Content-Type" # ... resources: 6 requests: cpu: 1 memory: 512Mi limits: cpu: 2 memory: 2Gi logging: 7 type: inline loggers: rootLogger.level: INFO template: 8 pod: metadata: labels: label1: value1 securityContext: runAsUser: 1000001 fsGroup: 0 terminationGracePeriodSeconds: 120 readinessProbe: 9 initialDelaySeconds: 15 timeoutSeconds: 5 livenessProbe: initialDelaySeconds: 15 timeoutSeconds: 5 metricsConfig: 10 type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: cruise-control-metrics key: metrics-config.yml # ... 1 Capacity limits for broker resources. 2 Overrides set network capacity limits for specific brokers when running on nodes with heterogeneous network resources. 3 Cruise Control configuration. Standard Cruise Control configuration may be provided, restricted to those properties not managed directly by Streams for Apache Kafka. 4 Optimization goals configuration, which can include configuration for default optimization goals ( default.goals ), main optimization goals ( goals ), and hard goals ( hard.goals ). 5 CORS enabled and configured for read-only access to the Cruise Control API. 6 Requests for reservation of supported resources, currently cpu and memory , and limits to specify the maximum resources that can be consumed. 7 Cruise Control loggers and log levels added directly ( inline ) or indirectly ( external ) through a ConfigMap. A custom Log4j configuration must be placed under the log4j.properties key in the ConfigMap. Cruise Control has a single logger named rootLogger.level . You can set the log level to INFO, ERROR, WARN, TRACE, DEBUG, FATAL or OFF. 8 Template customization. Here a pod is scheduled with additional security attributes. 9 Healthchecks to know when to restart a container (liveness) and when a container can accept traffic (readiness). 10 Prometheus metrics enabled. In this example, metrics are configured for the Prometheus JMX Exporter (the default metrics exporter). Create or update the resource: oc apply -f <kafka_configuration_file> Check the status of the deployment: oc get deployments -n <my_cluster_operator_namespace> Output shows the deployment name and readiness NAME READY UP-TO-DATE AVAILABLE my-cluster-cruise-control 1/1 1 1 my-cluster is the name of the Kafka cluster. READY shows the number of replicas that are ready/expected. The deployment is successful when the AVAILABLE output shows 1 . Auto-created topics The following table shows the three topics that are automatically created when Cruise Control is deployed. These topics are required for Cruise Control to work properly and must not be deleted or changed. You can change the name of the topic using the specified configuration option. Table 19.3. Auto-created topics Auto-created topic configuration Default topic name Created by Function metric.reporter.topic strimzi.cruisecontrol.metrics Streams for Apache Kafka Metrics Reporter Stores the raw metrics from the Metrics Reporter in each Kafka broker. partition.metric.sample.store.topic strimzi.cruisecontrol.partitionmetricsamples Cruise Control Stores the derived metrics for each partition. These are created by the Metric Sample Aggregator . broker.metric.sample.store.topic strimzi.cruisecontrol.modeltrainingsamples Cruise Control Stores the metrics samples used to create the Cluster Workload Model . To prevent the removal of records that are needed by Cruise Control, log compaction is disabled in the auto-created topics. Note If the names of the auto-created topics are changed in a Kafka cluster that already has Cruise Control enabled, the old topics will not be deleted and should be manually removed. What to do After configuring and deploying Cruise Control, you can generate optimization proposals . Additional resources Optimization goals overview 19.6. Generating optimization proposals When you create or update a KafkaRebalance resource, Cruise Control generates an optimization proposal for the Kafka cluster based on the configured optimization goals . Analyze the information in the optimization proposal and decide whether to approve it. You can use the results of the optimization proposal to rebalance your Kafka cluster. You can run the optimization proposal in one of the following modes: full (default) add-brokers remove-brokers The mode you use depends on whether you are rebalancing across all the brokers already running in the Kafka cluster; or you want to rebalance after scaling up or before scaling down your Kafka cluster. For more information, see Rebalancing modes with broker scaling . Prerequisites You have deployed Cruise Control to your Streams for Apache Kafka cluster. You have configured optimization goals and, optionally, capacity limits on broker resources. For more information on configuring Cruise Control, see Section 19.5, "Configuring and deploying Cruise Control with Kafka" . Procedure Create a KafkaRebalance resource and specify the appropriate mode. full mode (default) To use the default optimization goals defined in the Kafka resource, leave the spec property empty. Cruise Control rebalances a Kafka cluster in full mode by default. Example configuration with full rebalancing by default apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: {} You can also run a full rebalance by specifying the full mode through the spec.mode property. Example configuration specifying full mode apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: mode: full add-brokers mode If you want to rebalance a Kafka cluster after scaling up, specify the add-brokers mode. In this mode, existing replicas are moved to the newly added brokers. You need to specify the brokers as a list. Example configuration specifying add-brokers mode apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: mode: add-brokers brokers: [3, 4] 1 1 List of newly added brokers added by the scale up operation. This property is mandatory. remove-brokers mode If you want to rebalance a Kafka cluster before scaling down, specify the remove-brokers mode. In this mode, replicas are moved off the brokers that are going to be removed. You need to specify the brokers that are being removed as a list. Example configuration specifying remove-brokers mode apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: mode: remove-brokers brokers: [3, 4] 1 1 List of brokers to be removed by the scale down operation. This property is mandatory. Note The following steps and the steps to approve or stop a rebalance are the same regardless of the rebalance mode you are using. To configure user-provided optimization goals instead of using the default goals, add the goals property and enter one or more goals. In the following example, rack awareness and replica capacity are configured as user-provided optimization goals: apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: goals: - RackAwareGoal - ReplicaCapacityGoal To ignore the configured hard goals, add the skipHardGoalCheck: true property: apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: goals: - RackAwareGoal - ReplicaCapacityGoal skipHardGoalCheck: true (Optional) To approve the optimization proposal automatically, set the strimzi.io/rebalance-auto-approval annotation to true : apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster annotations: strimzi.io/rebalance-auto-approval: "true" spec: goals: - RackAwareGoal - ReplicaCapacityGoal skipHardGoalCheck: true Create or update the resource: oc apply -f <kafka_rebalance_configuration_file> The Cluster Operator requests the optimization proposal from Cruise Control. This might take a few minutes depending on the size of the Kafka cluster. If you used the automatic approval mechanism, wait for the status of the optimization proposal to change to Ready . If you haven't enabled the automatic approval mechanism, wait for the status of the optimization proposal to change to ProposalReady : oc get kafkarebalance -o wide -w -n <namespace> PendingProposal A PendingProposal status means the rebalance operator is polling the Cruise Control API to check if the optimization proposal is ready. ProposalReady A ProposalReady status means the optimization proposal is ready for review and approval. When the status changes to ProposalReady , the optimization proposal is ready to approve. Review the optimization proposal. The optimization proposal is contained in the Status.Optimization Result property of the KafkaRebalance resource. oc describe kafkarebalance <kafka_rebalance_resource_name> Example optimization proposal Status: Conditions: Last Transition Time: 2020-05-19T13:50:12.533Z Status: ProposalReady Type: State Observed Generation: 1 Optimization Result: Data To Move MB: 0 Excluded Brokers For Leadership: Excluded Brokers For Replica Move: Excluded Topics: Intra Broker Data To Move MB: 0 Monitored Partitions Percentage: 100 Num Intra Broker Replica Movements: 0 Num Leader Movements: 0 Num Replica Movements: 26 On Demand Balancedness Score After: 81.8666802863978 On Demand Balancedness Score Before: 78.01176356230222 Recent Windows: 1 Session Id: 05539377-ca7b-45ef-b359-e13564f1458c The properties in the Optimization Result section describe the pending cluster rebalance operation. For descriptions of each property, see Contents of optimization proposals . Insufficient CPU capacity If a Kafka cluster is overloaded in terms of CPU utilization, you might see an insufficient CPU capacity error in the KafkaRebalance status. It's worth noting that this utilization value is unaffected by the excludedTopics configuration. Although optimization proposals will not reassign replicas of excluded topics, their load is still considered in the utilization calculation. Example CPU utilization error com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException: [CpuCapacityGoal] Insufficient capacity for cpu (Utilization 615.21, Allowed Capacity 420.00, Threshold: 0.70). Add at least 3 brokers with the same cpu capacity (100.00) as broker-0. Add at least 3 brokers with the same cpu capacity (100.00) as broker-0. Note The error shows CPU capacity as a percentage rather than the number of CPU cores. For this reason, it does not directly map to the number of CPUs configured in the Kafka custom resource. It is like having a single virtual CPU per broker, which has the cycles of the CPUs configured in Kafka.spec.kafka.resources.limits.cpu . This has no effect on the rebalance behavior, since the ratio between CPU utilization and capacity remains the same. What to do Section 19.7, "Approving an optimization proposal" Additional resources Section 19.3, "Optimization proposals overview" 19.7. Approving an optimization proposal You can approve an optimization proposal generated by Cruise Control, if its status is ProposalReady . Cruise Control will then apply the optimization proposal to the Kafka cluster, reassigning partitions to brokers and changing partition leadership. Caution This is not a dry run. Before you approve an optimization proposal, you must: Refresh the proposal in case it has become out of date. Carefully review the contents of the proposal . Prerequisites You have generated an optimization proposal from Cruise Control. The KafkaRebalance custom resource status is ProposalReady . Procedure Perform these steps for the optimization proposal that you want to approve. Unless the optimization proposal is newly generated, check that it is based on current information about the state of the Kafka cluster. To do so, refresh the optimization proposal to make sure it uses the latest cluster metrics: Annotate the KafkaRebalance resource in OpenShift with strimzi.io/rebalance=refresh : oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="refresh" Wait for the status of the optimization proposal to change to ProposalReady : oc get kafkarebalance -o wide -w -n <namespace> PendingProposal A PendingProposal status means the rebalance operator is polling the Cruise Control API to check if the optimization proposal is ready. ProposalReady A ProposalReady status means the optimization proposal is ready for review and approval. When the status changes to ProposalReady , the optimization proposal is ready to approve. Approve the optimization proposal that you want Cruise Control to apply. Annotate the KafkaRebalance resource in OpenShift with strimzi.io/rebalance=approve : oc annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="approve" The Cluster Operator detects the annotated resource and instructs Cruise Control to rebalance the Kafka cluster. Wait for the status of the optimization proposal to change to Ready : oc get kafkarebalance -o wide -w -n <namespace> Rebalancing A Rebalancing status means the rebalancing is in progress. Ready A Ready status means the rebalance is complete. NotReady A NotReady status means an error occurred- see Fixing problems with a KafkaRebalance resource . When the status changes to Ready , the rebalance is complete. To use the same KafkaRebalance custom resource to generate another optimization proposal, apply the refresh annotation to the custom resource. This moves the custom resource to the PendingProposal or ProposalReady state. You can then review the optimization proposal and approve it, if desired. Additional resources Section 19.3, "Optimization proposals overview" Section 19.8, "Stopping a cluster rebalance" 19.8. Stopping a cluster rebalance Once started, a cluster rebalance operation might take some time to complete and affect the overall performance of the Kafka cluster. If you want to stop a cluster rebalance operation that is in progress, apply the stop annotation to the KafkaRebalance custom resource. This instructs Cruise Control to finish the current batch of partition reassignments and then stop the rebalance. When the rebalance has stopped, completed partition reassignments have already been applied; therefore, the state of the Kafka cluster is different when compared to prior to the start of the rebalance operation. If further rebalancing is required, you should generate a new optimization proposal. Note The performance of the Kafka cluster in the intermediate (stopped) state might be worse than in the initial state. Prerequisites You have approved the optimization proposal by annotating the KafkaRebalance custom resource with approve . The status of the KafkaRebalance custom resource is Rebalancing . Procedure Annotate the KafkaRebalance resource in OpenShift: oc annotate kafkarebalance rebalance-cr-name strimzi.io/rebalance="stop" Check the status of the KafkaRebalance resource: oc describe kafkarebalance rebalance-cr-name Wait until the status changes to Stopped . Additional resources Section 19.3, "Optimization proposals overview" 19.9. Fixing problems with a KafkaRebalance resource If an issue occurs when creating a KafkaRebalance resource or interacting with Cruise Control, the error is reported in the resource status, along with details of how to fix it. The resource also moves to the NotReady state. To continue with the cluster rebalance operation, you must fix the problem in the KafkaRebalance resource itself or with the overall Cruise Control deployment. Problems might include the following: A misconfigured parameter in the KafkaRebalance resource. The strimzi.io/cluster label for specifying the Kafka cluster in the KafkaRebalance resource is missing. The Cruise Control server is not deployed as the cruiseControl property in the Kafka resource is missing. The Cruise Control server is not reachable. After fixing the issue, you need to add the refresh annotation to the KafkaRebalance resource. During a "refresh", a new optimization proposal is requested from the Cruise Control server. Prerequisites You have approved an optimization proposal . The status of the KafkaRebalance custom resource for the rebalance operation is NotReady . Procedure Get information about the error from the KafkaRebalance status: oc describe kafkarebalance rebalance-cr-name Attempt to resolve the issue in the KafkaRebalance resource. Annotate the KafkaRebalance resource in OpenShift: oc annotate kafkarebalance rebalance-cr-name strimzi.io/rebalance="refresh" Check the status of the KafkaRebalance resource: oc describe kafkarebalance rebalance-cr-name Wait until the status changes to PendingProposal , or directly to ProposalReady . Additional resources Section 19.3, "Optimization proposals overview"
RackAwareGoal; ReplicaCapacityGoal; DiskCapacityGoal; NetworkInboundCapacityGoal; NetworkOutboundCapacityGoal; CpuCapacityGoal apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: # zookeeper: # entityOperator: topicOperator: {} userOperator: {} cruiseControl: brokerCapacity: inboundNetwork: 10000KB/s outboundNetwork: 10000KB/s config: # Note that `default.goals` (superset) must also include all `hard.goals` (subset) default.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal hard.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkInboundCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.NetworkOutboundCapacityGoal # RackAwareGoal; MinTopicLeadersPerBrokerGoal; ReplicaCapacityGoal; DiskCapacityGoal; NetworkInboundCapacityGoal; NetworkOutboundCapacityGoal; CpuCapacityGoal; ReplicaDistributionGoal; PotentialNwOutGoal; DiskUsageDistributionGoal; NetworkInboundUsageDistributionGoal; NetworkOutboundUsageDistributionGoal; CpuUsageDistributionGoal; TopicReplicaDistributionGoal; LeaderReplicaDistributionGoal; LeaderBytesInDistributionGoal; PreferredLeaderElectionGoal apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: kafka: # zookeeper: # entityOperator: topicOperator: {} userOperator: {} cruiseControl: brokerCapacity: inboundNetwork: 10000KB/s outboundNetwork: 10000KB/s config: # Note that `default.goals` (superset) must also include all `hard.goals` (subset) default.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal hard.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal # KafkaRebalance.spec.goals describe kafkarebalance <kafka_rebalance_resource_name> -n <namespace> get kafkarebalance -o json | jq <jq_query> . Name: my-rebalance Namespace: myproject Labels: strimzi.io/cluster=my-cluster Annotations: API Version: kafka.strimzi.io/v1alpha1 Kind: KafkaRebalance Metadata: Status: Conditions: Last Transition Time: 2022-04-05T14:36:11.900Z Status: ProposalReady Type: State Observed Generation: 1 Optimization Result: Data To Move MB: 0 Excluded Brokers For Leadership: Excluded Brokers For Replica Move: Excluded Topics: Intra Broker Data To Move MB: 12 Monitored Partitions Percentage: 100 Num Intra Broker Replica Movements: 0 Num Leader Movements: 24 Num Replica Movements: 55 On Demand Balancedness Score After: 82.91290759174306 On Demand Balancedness Score Before: 78.01176356230222 Recent Windows: 5 Session Id: a4f833bd-2055-4213-bfdd-ad21f95bf184 apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster annotations: strimzi.io/rebalance-auto-approval: "true" spec: mode: # any mode # describe configmaps <my_rebalance_configmap_name> -n <namespace> get configmaps <my_rebalance_configmap_name> -o json | jq '.["data"]["brokerLoad.json"]|fromjson|.' apiVersion: kafka.strimzi.io/v1beta2 kind: Kafka metadata: name: my-cluster spec: # cruiseControl: brokerCapacity: 1 inboundNetwork: 10000KB/s outboundNetwork: 10000KB/s overrides: 2 - brokers: [0] inboundNetwork: 20000KiB/s outboundNetwork: 20000KiB/s - brokers: [1, 2] inboundNetwork: 30000KiB/s outboundNetwork: 30000KiB/s # config: 3 # Note that `default.goals` (superset) must also include all `hard.goals` (subset) default.goals: > 4 com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.ReplicaCapacityGoal, com.linkedin.kafka.cruisecontrol.analyzer.goals.DiskCapacityGoal # hard.goals: > com.linkedin.kafka.cruisecontrol.analyzer.goals.RackAwareGoal # cpu.balance.threshold: 1.1 metadata.max.age.ms: 300000 send.buffer.bytes: 131072 webserver.http.cors.enabled: true 5 webserver.http.cors.origin: "*" webserver.http.cors.exposeheaders: "User-Task-ID,Content-Type" # resources: 6 requests: cpu: 1 memory: 512Mi limits: cpu: 2 memory: 2Gi logging: 7 type: inline loggers: rootLogger.level: INFO template: 8 pod: metadata: labels: label1: value1 securityContext: runAsUser: 1000001 fsGroup: 0 terminationGracePeriodSeconds: 120 readinessProbe: 9 initialDelaySeconds: 15 timeoutSeconds: 5 livenessProbe: initialDelaySeconds: 15 timeoutSeconds: 5 metricsConfig: 10 type: jmxPrometheusExporter valueFrom: configMapKeyRef: name: cruise-control-metrics key: metrics-config.yml apply -f <kafka_configuration_file> get deployments -n <my_cluster_operator_namespace> NAME READY UP-TO-DATE AVAILABLE my-cluster-cruise-control 1/1 1 1 apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: {} apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: mode: full apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: mode: add-brokers brokers: [3, 4] 1 apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: mode: remove-brokers brokers: [3, 4] 1 apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: goals: - RackAwareGoal - ReplicaCapacityGoal apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster spec: goals: - RackAwareGoal - ReplicaCapacityGoal skipHardGoalCheck: true apiVersion: kafka.strimzi.io/v1beta2 kind: KafkaRebalance metadata: name: my-rebalance labels: strimzi.io/cluster: my-cluster annotations: strimzi.io/rebalance-auto-approval: "true" spec: goals: - RackAwareGoal - ReplicaCapacityGoal skipHardGoalCheck: true apply -f <kafka_rebalance_configuration_file> get kafkarebalance -o wide -w -n <namespace> describe kafkarebalance <kafka_rebalance_resource_name> Status: Conditions: Last Transition Time: 2020-05-19T13:50:12.533Z Status: ProposalReady Type: State Observed Generation: 1 Optimization Result: Data To Move MB: 0 Excluded Brokers For Leadership: Excluded Brokers For Replica Move: Excluded Topics: Intra Broker Data To Move MB: 0 Monitored Partitions Percentage: 100 Num Intra Broker Replica Movements: 0 Num Leader Movements: 0 Num Replica Movements: 26 On Demand Balancedness Score After: 81.8666802863978 On Demand Balancedness Score Before: 78.01176356230222 Recent Windows: 1 Session Id: 05539377-ca7b-45ef-b359-e13564f1458c com.linkedin.kafka.cruisecontrol.exception.OptimizationFailureException: [CpuCapacityGoal] Insufficient capacity for cpu (Utilization 615.21, Allowed Capacity 420.00, Threshold: 0.70). Add at least 3 brokers with the same cpu capacity (100.00) as broker-0. Add at least 3 brokers with the same cpu capacity (100.00) as broker-0. annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="refresh" get kafkarebalance -o wide -w -n <namespace> annotate kafkarebalance <kafka_rebalance_resource_name> strimzi.io/rebalance="approve" get kafkarebalance -o wide -w -n <namespace> annotate kafkarebalance rebalance-cr-name strimzi.io/rebalance="stop" describe kafkarebalance rebalance-cr-name describe kafkarebalance rebalance-cr-name annotate kafkarebalance rebalance-cr-name strimzi.io/rebalance="refresh" describe kafkarebalance rebalance-cr-name