WorksOnMyCluster

Project 2: Build a CRD & Controller

Extend Kubernetes with your own resource type and the reconciliation loop behind it — including the finalizer, the status conditions and the idempotence that separate an operator from a script.

advanced 70 min lesson hands-on task included

An operator is the reconciliation loop from lesson 1 applied to a resource you invented. This project builds a small but complete one — the value is in the four properties that make it safe, not in the feature.

What You Are Building

A ManagedCache CRD that provisions a Redis-style cache: a StatefulSet, a headless Service, a password Secret, and a PDB. Small enough to finish; complete enough to exercise everything real operators do.

apiVersion: cache.example.com/v1
kind: ManagedCache
metadata:
  name: sessions
  namespace: shop
spec:
  size: 3
  version: "7.4"
  storageGi: 10
  evictionPolicy: allkeys-lru

Step 1: Scaffold

mkdir cache-operator && cd cache-operator
kubebuilder init --domain example.com --repo example.com/cache-operator
kubebuilder create api --group cache --version v1 --kind ManagedCache --resource --controller

controller-runtime underneath handles caching, watches, workqueues, rate limiting and leader election. Writing those by hand is a large amount of subtle concurrency work.

Step 2: The API Type

// api/v1/managedcache_types.go
type ManagedCacheSpec struct {
    // +kubebuilder:validation:Minimum=1
    // +kubebuilder:validation:Maximum=9
    // +kubebuilder:default=1
    Size int32 `json:"size"`

    // +kubebuilder:validation:Pattern=`^[0-9]+\.[0-9]+$`
    Version string `json:"version"`

    // +kubebuilder:default=8
    StorageGi int32 `json:"storageGi,omitempty"`

    // +kubebuilder:validation:Enum=noeviction;allkeys-lru;volatile-lru
    // +kubebuilder:default=noeviction
    EvictionPolicy string `json:"evictionPolicy,omitempty"`
}

type ManagedCacheStatus struct {
    Conditions    []metav1.Condition `json:"conditions,omitempty"`
    ReadyReplicas int32              `json:"readyReplicas"`
    Endpoint      string             `json:"endpoint,omitempty"`
}

// +kubebuilder:object:root=true
// +kubebuilder:subresource:status
// +kubebuilder:subresource:scale:specpath=.spec.size,statuspath=.status.readyReplicas
// +kubebuilder:printcolumn:name="Size",type=integer,JSONPath=`.spec.size`
// +kubebuilder:printcolumn:name="Ready",type=integer,JSONPath=`.status.readyReplicas`
// +kubebuilder:printcolumn:name="Status",type=string,JSONPath=`.status.conditions[?(@.type=="Ready")].status`
// +kubebuilder:printcolumn:name="Age",type=date,JSONPath=`.metadata.creationTimestamp`
type ManagedCache struct { /* ... */ }

make manifests generate

The markers matter more than they look:

  • Validation markers become an OpenAPI schema the API server enforces — so your controller never receives invalid input and users get a clear error at kubectl apply time.
  • +kubebuilder:subresource:status separates spec and status endpoints. Without it, your controller writing status can clobber a concurrent user edit to spec.
  • +kubebuilder:subresource:scale makes kubectl scale managedcache/sessions --replicas=5 work, and lets an HPA target your CRD.
  • printcolumn makes kubectl get managedcache useful rather than just name-and-age.

Step 3: The Reconcile Loop

func (r *ManagedCacheReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    log := log.FromContext(ctx)

    var mc cachev1.ManagedCache
    if err := r.Get(ctx, req.NamespacedName, &mc); err != nil {
        // NotFound is normal — the object was deleted. Do not requeue.
        return ctrl.Result{}, client.IgnoreNotFound(err)
    }

    // ---- deletion path: run the finalizer, then release ----
    if !mc.DeletionTimestamp.IsZero() {
        if controllerutil.ContainsFinalizer(&mc, cacheFinalizer) {
            if err := r.cleanupExternal(ctx, &mc); err != nil {
                return ctrl.Result{}, err          // retry; object stays until it succeeds
            }
            controllerutil.RemoveFinalizer(&mc, cacheFinalizer)
            return ctrl.Result{}, r.Update(ctx, &mc)
        }
        return ctrl.Result{}, nil
    }

    // ---- ensure the finalizer is present BEFORE creating anything external ----
    if !controllerutil.ContainsFinalizer(&mc, cacheFinalizer) {
        controllerutil.AddFinalizer(&mc, cacheFinalizer)
        if err := r.Update(ctx, &mc); err != nil {
            return ctrl.Result{}, err
        }
    }

    // ---- reconcile children, idempotently ----
    if err := r.reconcileSecret(ctx, &mc); err != nil {
        return r.fail(ctx, &mc, "SecretFailed", err)
    }
    if err := r.reconcileService(ctx, &mc); err != nil {
        return r.fail(ctx, &mc, "ServiceFailed", err)
    }
    sts, err := r.reconcileStatefulSet(ctx, &mc)
    if err != nil {
        return r.fail(ctx, &mc, "StatefulSetFailed", err)
    }

    // ---- report OBSERVED state ----
    mc.Status.ReadyReplicas = sts.Status.ReadyReplicas
    mc.Status.Endpoint = fmt.Sprintf("%s.%s.svc.cluster.local:6379", mc.Name, mc.Namespace)
    ready := sts.Status.ReadyReplicas == mc.Spec.Size
    meta.SetStatusCondition(&mc.Status.Conditions, metav1.Condition{
        Type:    "Ready",
        Status:  boolToCondition(ready),
        Reason:  ternary(ready, "AllReplicasReady", "WaitingForReplicas"),
        Message: fmt.Sprintf("%d/%d replicas ready", sts.Status.ReadyReplicas, mc.Spec.Size),
    })
    if err := r.Status().Update(ctx, &mc); err != nil {
        return ctrl.Result{}, err
    }

    if !ready {
        return ctrl.Result{RequeueAfter: 10 * time.Second}, nil
    }
    return ctrl.Result{RequeueAfter: 5 * time.Minute}, nil    // periodic resync
}

The idempotent child pattern:

func (r *ManagedCacheReconciler) reconcileStatefulSet(
    ctx context.Context, mc *cachev1.ManagedCache,
) (*appsv1.StatefulSet, error) {
    sts := &appsv1.StatefulSet{
        ObjectMeta: metav1.ObjectMeta{Name: mc.Name, Namespace: mc.Namespace},
    }

    _, err := controllerutil.CreateOrUpdate(ctx, r.Client, sts, func() error {
        // This closure runs on BOTH create and update. Set only what you own.
        sts.Spec.Replicas    = &mc.Spec.Size
        sts.Spec.ServiceName = mc.Name
        sts.Spec.Selector    = &metav1.LabelSelector{MatchLabels: labelsFor(mc)}
        sts.Spec.Template    = podTemplateFor(mc)
        // Ownership → garbage collection deletes this when the CR is deleted
        return controllerutil.SetControllerReference(mc, sts, r.Scheme)
    })
    return sts, err
}

CreateOrUpdate fetches, applies your mutation, and writes only if something changed. That is what makes reconcile safe to run a thousand times — the property lesson 1 called idempotence.

Set only the fields you own inside the closure. Overwriting the whole spec fights other controllers (an HPA managing replicas, a mutating webhook injecting a sidecar) and produces an infinite update loop between you and them.

Watching children:

func (r *ManagedCacheReconciler) SetupWithManager(mgr ctrl.Manager) error {
    return ctrl.NewControllerManagedBy(mgr).
        For(&cachev1.ManagedCache{}).
        Owns(&appsv1.StatefulSet{}).      // child changes trigger the OWNER's reconcile
        Owns(&corev1.Service{}).
        Owns(&corev1.Secret{}).
        Complete(r)
}

Owns is what makes self-healing work: delete the StatefulSet and the owner reference maps it back to your ManagedCache, which reconciles and recreates it.

Step 4: Finalizers, Carefully

func (r *ManagedCacheReconciler) cleanupExternal(ctx context.Context, mc *cachev1.ManagedCache) error {
    // Delete things Kubernetes garbage collection does NOT own:
    // cloud resources, DNS records, external registrations.
    // Children with ownerReferences need no cleanup here.
    return r.deleteBackupBucket(ctx, mc)
}

Only use a finalizer for state outside the cluster. Owned children are handled by garbage collection.

The finalizer footgun: if your operator is uninstalled while ManagedCache objects exist, those objects become undeletable — and they block deletion of their entire namespace, which is the stuck-Terminating problem from lesson 5. Ship the escape hatch in your README:

kubectl patch managedcache sessions -p '{"metadata":{"finalizers":[]}}' --type=merge

Step 5: RBAC and Deploy

// +kubebuilder:rbac:groups=cache.example.com,resources=managedcaches,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups=cache.example.com,resources=managedcaches/status,verbs=get;update;patch
// +kubebuilder:rbac:groups=cache.example.com,resources=managedcaches/finalizers,verbs=update
// +kubebuilder:rbac:groups=apps,resources=statefulsets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=services;secrets,verbs=get;list;watch;create;update;patch;delete
// +kubebuilder:rbac:groups="",resources=events,verbs=create;patch

make manifests
make install                                          # CRDs only
make run                                              # run locally against your kubeconfig
make docker-build docker-push IMG=registry.example.com/cache-operator:0.1.0
make deploy IMG=registry.example.com/cache-operator:0.1.0

make run is the fast development loop — the controller runs on your laptop against a real cluster, with a debugger attached if you want one.

Step 6: Verification — the Four Properties

kubectl apply -f config/samples/cache_v1_managedcache.yaml
kubectl get managedcache
# NAME       SIZE   READY   STATUS   AGE
# sessions   3      3       True     2m

1. Idempotence

kubectl annotate managedcache sessions reconcile=force --overwrite
kubectl get sts sessions -o jsonpath='{.metadata.resourceVersion}{"\n"}'
# repeat — resourceVersion must NOT change. If it does, your reconcile writes unconditionally.

2. Self-healing

kubectl delete sts sessions
kubectl get sts sessions -w        # returns within seconds

3. Garbage collection

kubectl delete managedcache sessions
kubectl get sts,svc,secret -l app.kubernetes.io/name=managedcache
# all gone — via ownerReferences, not via code you wrote

4. Crash convergence

kubectl apply -f config/samples/cache_v1_managedcache.yaml
kubectl delete pod -n cache-operator-system -l control-plane=controller-manager   # kill mid-reconcile
kubectl get managedcache -w
# converges after restart, with NO duplicate children

Property 4 is the one that separates a controller from a script: it must reconcile from current state, never from a remembered diff.

Extensions worth building: a validating webhook rejecting a size change from 3 to 1 (quorum loss); a conversion webhook for v1 → v2; Prometheus metrics on reconcile duration and error count; envtest-based unit tests (make test).

The lesson to take away: the CRD is an afternoon’s work. Idempotence, ownership, status conditions and finalizer hygiene are what make it safe to run unattended — and each maps directly to a failure you can demonstrate in five minutes.