golang error "object has been modified" on k8s operator

php editor Strawberry encountered a common error when exploring the use of golang to develop kubernetes operators: "The object has been modified". This error usually occurs when multiple goroutines modify the same object at the same time, resulting in data inconsistency or race conditions. In this article, we will explore the causes and solutions of this error to help developers better understand and deal with similar problems.

Problem content

The "object has been modified" error appears on the k8s operator

import "sigs.k8s.io/controller-runtime"

There is a lot of discussion about this error. The main answer is "This problem occurs because I have an old version of the object when I try to update." But I also have some questions. In my operator, for some scenarios, I need to update a pod's annotation 2 times during a single "coordinate" call. Of course I often get the "object has been modified" error.

Question: I want to know where 'r.Get()' and 'r.Update()' get/update objects? From local cache, or API server?

1: I think 'r.Get()' is getting the object from 'cache' and 'r.Update()' is updating the object to cache, right? If so, why am I getting this error? If the pod object changes due to reasons other than the operator, will I never be able to update my pod object during the current "reconciliation"? (Since the cached object is local, it is already out of sync with the API server.) Why "retry" sometimes to get the latest object?

import "sigs.k8s.io/controller-runtime"

var pod corev1.Pod
if err := r.Get(ctx, req.NamespacedName, &pod); err != nil {
    if !apierrors.IsNotFound(err) {
        log.Error(err, "unable to get pod")
        return ctrl.Result{}, err
    }
}

if err := r.Update(ctx, &pod); err != nil {
    log.Error(err, "unable to update chaosctl status")
    return ctrl.Result{}, err
}

2: If 'r.Get()' is to get the object from the API server, and 'r.Update()' also updates the API server. Why do I need to retry updating the object?

Workaround

When you use r.Get() and r.Update() in a Kubernetes controller, the interaction with the API server involves the local cache and the API server itself.

r.Get():

The r.Get() function retrieves the object from the local cache if it exists; otherwise, it gets it from the API server. If the object exists in the local cache, it is returned immediately. If not, request the API server to obtain the object, and store the obtained object in the local cache for subsequent use. r.Update():

r.Update() The function updates objects in the local cache and API server. Update operations to the API server may fail with a "conflict" error if the object has been modified in the local cache since it was originally retrieved. This occurs when the version of an object in the cache does not match the version on the API server, indicating that someone else modified the object in the meantime.

Strategies can be adopted to deal with it -

1. Optimistic Concurrency Control (OCC): - Update to match previous version.
2. retry -

retryErr := retry.RetryOnConflict(retry.DefaultRetry, func() error {
        return r.Update(ctx, pod)
    })
    if retryErr != nil {
        return retryErr
    }

The above is the detailed content of golang error "object has been modified" on k8s operator. For more information, please follow other related articles on the PHP Chinese website!