title: Operator SDK controller failed to update Custom Resource status
body: <p>I'm following <a href="https://learn.openshift.com/operatorframework/go-operator-podset/" rel="nofollow noreferrer">this tutorial</a> to create my first Custom Resource named PodSet and currently at step 6 of 7 to test my CR.</p>
  <p>Here is my Operator SDK controller Go code:</p>
  <pre><code>package controllers
  
  import (
      &quot;context&quot;
      &quot;reflect&quot;
  
      &quot;github.com/go-logr/logr&quot;
      &quot;k8s.io/apimachinery/pkg/labels&quot;
      &quot;k8s.io/apimachinery/pkg/runtime&quot;
      ctrl &quot;sigs.k8s.io/controller-runtime&quot;
      &quot;sigs.k8s.io/controller-runtime/pkg/client&quot;
      &quot;sigs.k8s.io/controller-runtime/pkg/controller/controllerutil&quot;
      &quot;sigs.k8s.io/controller-runtime/pkg/reconcile&quot;
  
      appv1alpha1 &quot;github.com/redhat/podset-operator/api/v1alpha1&quot;
      corev1 &quot;k8s.io/api/core/v1&quot;
      &quot;k8s.io/apimachinery/pkg/api/errors&quot;
      metav1 &quot;k8s.io/apimachinery/pkg/apis/meta/v1&quot;
  )
  
  // PodSetReconciler reconciles a PodSet object
  type PodSetReconciler struct {
      client.Client
      Log    logr.Logger
      Scheme *runtime.Scheme
  }
  
  // +kubebuilder:rbac:groups=app.example.com,resources=podsets,verbs=get;list;watch;create;update;patch;delete
  // +kubebuilder:rbac:groups=app.example.com,resources=podsets/status,verbs=get;update;patch
  // +kubebuilder:rbac:groups=v1,resources=pods,verbs=get;list;watch;create;update;patch;delete
  
  // Reconcile is the core logic of controller
  func (r *PodSetReconciler) Reconcile(req ctrl.Request) (ctrl.Result, error) {
      _ = context.Background()
      _ = r.Log.WithValues(&quot;podset&quot;, req.NamespacedName)
  
      // Fetch the PodSet instance (the parent of the pods)
      instance := &amp;appv1alpha1.PodSet{}
      err := r.Get(context.Background(), req.NamespacedName, instance)
      if err != nil {
          if errors.IsNotFound(err) {
              // Request object not found, could have been deleted after reconcile request.
              // Owned objects are automatically garbage collected. For additional cleanup logic use finalizers.
              // Return and don't requeue
              return reconcile.Result{}, nil
          }
          // Error reading the object - requeue the request
          return reconcile.Result{}, err
      }
  
      // List all pods owned by this PodSet instance
      podSet := instance
      podList := &amp;corev1.PodList{}
      labelz := map[string]string{
          &quot;app&quot;:     podSet.Name, // the metadata.name field from user's CR PodSet YAML file
          &quot;version&quot;: &quot;v0.1&quot;,
      }
      labelSelector := labels.SelectorFromSet(labelz)
      listOpts := &amp;client.ListOptions{Namespace: podSet.Namespace, LabelSelector: labelSelector}
      if err = r.List(context.Background(), podList, listOpts); err != nil {
          return reconcile.Result{}, err
      }
  
      // Count the pods that are pending or running and add them to available array
      var available []corev1.Pod
      for _, pod := range podList.Items {
          if pod.ObjectMeta.DeletionTimestamp != nil {
              continue
          }
          if pod.Status.Phase == corev1.PodRunning || pod.Status.Phase == corev1.PodPending {
              available = append(available, pod)
          }
      }
      numAvailable := int32(len(available))
      availableNames := []string{}
      for _, pod := range available {
          availableNames = append(availableNames, pod.ObjectMeta.Name)
      }
  
      // Update the status if necessary
      status := appv1alpha1.PodSetStatus{
          PodNames:          availableNames,
          AvailableReplicas: numAvailable,
      }
      if !reflect.DeepEqual(podSet.Status, status) {
          podSet.Status = status
          err = r.Status().Update(context.Background(), podSet)
          if err != nil {
              r.Log.Error(err, &quot;Failed to update PodSet status&quot;)
              return reconcile.Result{}, err
          }
      }
  
      // When the number of pods in the cluster is bigger that what we want, scale down
      if numAvailable &gt; podSet.Spec.Replicas {
          r.Log.Info(&quot;Scaling down pods&quot;, &quot;Currently available&quot;, numAvailable, &quot;Required replicas&quot;, podSet.Spec.Replicas)
          diff := numAvailable - podSet.Spec.Replicas
          toDeletePods := available[:diff] // Syntax help: https://play.golang.org/p/SHAMCdd12sp
          for _, toDeletePod := range toDeletePods {
              err = r.Delete(context.Background(), &amp;toDeletePod)
              if err != nil {
                  r.Log.Error(err, &quot;Failed to delete pod&quot;, &quot;pod.name&quot;, toDeletePod.Name)
                  return reconcile.Result{}, err
              }
          }
          return reconcile.Result{Requeue: true}, nil
      }
  
      // When the number of pods in the cluster is smaller that what we want, scale up
      if numAvailable &lt; podSet.Spec.Replicas {
          r.Log.Info(&quot;Scaling up pods&quot;, &quot;Currently available&quot;, numAvailable, &quot;Required replicas&quot;, podSet.Spec.Replicas)
          // Define a new Pod object
          pod := newPodForCR(podSet)
          // Set PodSet instance as the owner of the Pod
          if err := controllerutil.SetControllerReference(podSet, pod, r.Scheme); err != nil {
              return reconcile.Result{}, err
          }
          err = r.Create(context.Background(), pod)
          if err != nil {
              r.Log.Error(err, &quot;Failed to create pod&quot;, &quot;pod.name&quot;, pod.Name)
              return reconcile.Result{}, err
          }
          return reconcile.Result{Requeue: true}, nil
      }
  
      return ctrl.Result{}, nil
  }
  
  // newPodForCR returns a busybox pod with the same name/namespace as the cr
  func newPodForCR(cr *appv1alpha1.PodSet) *corev1.Pod {
      labels := map[string]string{
          &quot;app&quot;:     cr.Name, // the metadata.name field from user's CR PodSet YAML file
          &quot;version&quot;: &quot;v0.1&quot;,
      }
      return &amp;corev1.Pod{
          ObjectMeta: metav1.ObjectMeta{
              GenerateName: cr.Name + &quot;-pod&quot;,
              Namespace:    cr.Namespace,
              Labels:       labels,
          },
          Spec: corev1.PodSpec{
              Containers: []corev1.Container{
                  {
                      Name:    &quot;busybox&quot;,
                      Image:   &quot;busybox&quot;,
                      Command: []string{&quot;sleep&quot;, &quot;3600&quot;},
                  },
              },
          },
      }
  }
  
  // SetupWithManager defines how the controller will watch for resources
  func (r *PodSetReconciler) SetupWithManager(mgr ctrl.Manager) error {
      return ctrl.NewControllerManagedBy(mgr).
          For(&amp;appv1alpha1.PodSet{}).
          Owns(&amp;corev1.Pod{}).
          Complete(r)
  }
  </code></pre>
  <p>When I apply below YAML file, I saw strange behaviour of the pods. They were struggling in the first few seconds — some of them get up and running for a while and quickly get into terminating state. When I leave them untouched for few more seconds, the CR reached the desired state just fine.</p>
  <pre><code>apiVersion: app.example.com/v1alpha1
  kind: PodSet
  metadata:
    name: podset-sample
  spec:
    replicas: 5
  </code></pre>
  <p>I captured the deployment scene above in <a href="https://youtu.be/bK5_2uiDuhE" rel="nofollow noreferrer">this video</a>. And <a href="https://pastebin.com/raw/md0QYx23" rel="nofollow noreferrer">here</a> are the full logs from my local terminal running <code>WATCH_NAMESPACE=podset-operator make run</code> command (sorry, I have to use Pastebin because SO didn't allow me to paste the full logs here because they are too long).</p>
  <p>So, my questions here are:</p>
  <ol>
  <li>What does the <code>Failed to update PodSet status    {&quot;error&quot;: &quot;Operation cannot be fulfilled on podsets.app.example.com \&quot;podset-sample\&quot;: the object has been modified; please apply your changes to the latest version and try again&quot;}</code> actually means?</li>
  <li>Why this happened?</li>
  <li>What can I do to get rid of these errors?</li>
  </ol>