Admission Control, CRDs & Operators

Kubernetes was designed to be extended. Custom resources give you new object types; admission webhooks let you intervene in every write; operators combine both into software that does what an experienced human operator would.

Topic 1: The Admission Chain

The order and what each stage can do:

1. Authentication — who are you? (certificate, bearer token, OIDC)
2. Authorisation — may you? (RBAC)
3. Mutating admission — built-in plugins, then MutatingWebhookConfigurations. May change the object.
4. Schema validation — does it match the OpenAPI schema?
5. Validating admission — built-in plugins, then ValidatingWebhookConfigurations, then ValidatingAdmissionPolicy. May only accept or reject.
6. Persist to etcd.

Mutation runs before validation, deliberately: a sidecar injector adds a container, and the validators then see the final object.

Topic 2: Admission Webhooks and the Way They Bite

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: require-team-label
webhooks:
  - name: validate.example.com
    admissionReviewVersions: ["v1"]
    sideEffects: None
    failurePolicy: Fail                 # Fail | Ignore
    timeoutSeconds: 5
    clientConfig:
      service:
        name: policy-webhook
        namespace: policy-system
        path: /validate
      caBundle: <base64 CA>
    rules:
      - apiGroups: ["apps"]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["deployments"]
        scope: Namespaced
    namespaceSelector:
      matchExpressions:
        - key: kubernetes.io/metadata.name
          operator: NotIn
          values: [kube-system, policy-system]     # ← CRITICAL

The failure mode that takes out a cluster:

failurePolicy: Fail means “if the webhook is unreachable, reject the request”. Now consider: the webhook runs as pods. Those pods are down. Any write matching the rules is rejected — including the write that would restart the webhook. The cluster cannot recover itself, and the fix is to delete the ValidatingWebhookConfiguration by hand, which requires knowing that is the problem.

Three defences, and you want all three:

1. namespaceSelector excluding kube-system and the webhook’s own namespace. Non-negotiable.
2. Narrow rules. Match only the resources you actually need. A webhook matching */* is a cluster-wide single point of failure.
3. Run the webhook HA with a PDB, and consider failurePolicy: Ignore for anything not security-critical.

kubectl get validatingwebhookconfigurations
kubectl get mutatingwebhookconfigurations
# The emergency escape hatch:
kubectl delete validatingwebhookconfiguration require-team-label

ValidatingAdmissionPolicy — no webhook at all

GA in 1.30, and it removes this entire class of risk by evaluating CEL expressions in-process:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: require-limits
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: "object.spec.containers.all(c, has(c.resources.limits))"
      message: "all containers must set resource limits"
      reason: Invalid
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: require-limits-binding
spec:
  policyName: require-limits
  validationActions: [Deny, Audit]
  matchResources:
    namespaceSelector:
      matchLabels: { enforce-limits: "true" }

No pods to run, no certificates to rotate, no availability risk. For validation-only policies this should now be your default; reach for a webhook when you need mutation or logic CEL cannot express.

Topic 3: Custom Resource Definitions

apiVersion: apiextensions.k8s.io/v1
kind: CustomResourceDefinition
metadata:
  name: databases.example.com
spec:
  group: example.com
  scope: Namespaced
  names:
    plural: databases
    singular: database
    kind: Database
    shortNames: [db]
  versions:
    - name: v1
      served: true
      storage: true
      subresources:
        status: {}              # separate status endpoint — see below
        scale:
          specReplicasPath: .spec.replicas
          statusReplicasPath: .status.replicas
      additionalPrinterColumns:
        - name: Engine
          type: string
          jsonPath: .spec.engine
        - name: Ready
          type: string
          jsonPath: .status.conditions[?(@.type=="Ready")].status
        - name: Age
          type: date
          jsonPath: .metadata.creationTimestamp
      schema:
        openAPIV3Schema:
          type: object
          required: [spec]
          properties:
            spec:
              type: object
              required: [engine, version]
              properties:
                engine:
                  type: string
                  enum: [postgres, mysql]
                version:
                  type: string
                  pattern: '^[0-9]+\.[0-9]+$'
                replicas:
                  type: integer
                  minimum: 1
                  maximum: 9
                  default: 1
                storageGi:
                  type: integer
                  default: 20
              x-kubernetes-validations:
                - rule: "self.replicas % 2 == 1"
                  message: "replicas must be odd for quorum"
            status:
              type: object
              properties:
                conditions:
                  type: array
                  items: { type: object, x-kubernetes-preserve-unknown-fields: true }

kubectl apply -f crd.yaml
kubectl get crd databases.example.com
kubectl explain database.spec        # your schema, served by the API server
kubectl get db                       # your printer columns

Details worth getting right:

Always write a full schema. The API server enforces it, so your controller receives validated input and users get useful errors at apply time instead of confusing behaviour later.

x-kubernetes-validations brings CEL into the CRD itself — cross-field rules the OpenAPI schema cannot express, evaluated by the API server.

The status subresource is not cosmetic. With it, spec and status have separate endpoints and separate RBAC: users update spec, the controller updates status, and neither can clobber the other. Without it, a controller writing status can overwrite a concurrent spec change.

additionalPrinterColumns is a small thing that makes a CRD feel finished — kubectl get db showing engine and readiness rather than just name and age.

Versioning: exactly one version has storage: true. Serving several requires a conversion webhook to translate between them.

Topic 4: Operators

An operator is the reconciliation loop from lesson 1, applied to your CRD. Nothing about it is privileged — it is a Deployment with RBAC and a watch.

func (r *DatabaseReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    var db examplev1.Database
    if err := r.Get(ctx, req.NamespacedName, &db); err != nil {
        return ctrl.Result{}, client.IgnoreNotFound(err)   // deleted — nothing to do
    }

    // Reconcile toward the desired state. This MUST be idempotent —
    // it will run many times for the same object.
    sts := buildStatefulSet(&db)
    if err := ctrl.SetControllerReference(&db, sts, r.Scheme); err != nil {
        return ctrl.Result{}, err
    }
    if _, err := ctrl.CreateOrUpdate(ctx, r.Client, sts, func() error {
        sts.Spec.Replicas = &db.Spec.Replicas
        return nil
    }); err != nil {
        return ctrl.Result{}, err
    }

    // Report what you observed
    meta.SetStatusCondition(&db.Status.Conditions, metav1.Condition{
        Type: "Ready", Status: metav1.ConditionTrue, Reason: "StatefulSetReady",
    })
    return ctrl.Result{RequeueAfter: 5 * time.Minute}, r.Status().Update(ctx, &db)
}

Principles that separate a working operator from a dangerous one:

Idempotence above all. Reconcile runs repeatedly for the same object — on resync, on any watched change, on restart. Every run must converge to the same result.

Never assume you saw every event. Reconcile from current state, not from a diff. This is the level-triggered design from lesson 1, and it is why an operator can be restarted safely.

Set owner references so garbage collection cleans up children when the parent is deleted.

Report status honestly with conditions. kubectl describe should tell an operator-of-the-operator what is wrong.

Use finalizers for external cleanup — deleting a Database CR should delete the cloud resource. But a finalizer whose controller is uninstalled makes the object undeletable and blocks namespace deletion, which is the namespace-stuck-Terminating problem from the labels lesson. Always ship a documented way to remove your finalizers.

Rate-limit and back off. A tight reconcile loop against the API server is a self-inflicted denial of service. Kubebuilder’s default workqueue handles this; hand-rolled controllers frequently do not.

Building one:

kubebuilder init --domain example.com --repo example.com/db-operator
kubebuilder create api --group example --version v1 --kind Database
make manifests generate
make docker-build docker-push IMG=registry.example.com/db-operator:0.1.0
make deploy IMG=registry.example.com/db-operator:0.1.0

Kubebuilder and the Operator SDK both build on controller-runtime, which handles caching, watches, workqueues and leader election. Writing those yourself is a large amount of subtle work.

Topic 5: When an Operator Is the Wrong Answer

The pattern is fashionable and frequently over-applied. An operator is justified when there is domain knowledge a human would otherwise apply: taking a backup before a version upgrade, promoting a replica during failover, resharding, or coordinating a rolling restart in a specific order.

It is not justified for:

• Templating YAML — that is Helm or Kustomize.
• Simple deployment — that is a Deployment.
• Wrapping a Helm chart in a CRD so it “looks like an operator”.

Every operator is a privileged, always-running controller you now maintain, with RBAC broad enough to manage its domain. That is a real ongoing cost. Prefer an existing, mature operator (CloudNativePG, Strimzi, cert-manager) over writing one, and prefer no operator over a thin one.

Topic 6: Debugging Extensions

# Is a webhook rejecting things?
kubectl get validatingwebhookconfigurations,mutatingwebhookconfigurations
kubectl apply -f pod.yaml
# Error from server: admission webhook "validate.example.com" denied the request: ...

# Is the webhook service even reachable?
kubectl get endpointslices -n policy-system -l kubernetes.io/service-name=policy-webhook
kubectl logs -n policy-system deploy/policy-webhook --tail=100

# CRD problems
kubectl get crd databases.example.com -o jsonpath='{.status.conditions}' | jq
kubectl explain database.spec --recursive

# Operator problems
kubectl logs -n db-system deploy/db-operator --tail=200
kubectl get db mydb -o jsonpath='{.status.conditions}' | jq
kubectl get events --field-selector involvedObject.kind=Database

# Is the operator's RBAC sufficient?
kubectl auth can-i --list --as=system:serviceaccount:db-system:db-operator

# Emergency: remove a finalizer from an orphaned CR
kubectl patch database mydb -p '{"metadata":{"finalizers":[]}}' --type=merge

Try it yourself: Create a ValidatingWebhookConfiguration with failurePolicy: Fail pointing at a Service with no backing pods, matching only Deployments in a test namespace. Try to create a Deployment there and read the error. Then delete the webhook config to recover — and note how you would have found that at 3am.

Common mistake: Shipping a webhook that matches all namespaces including kube-system and its own. When it goes down, nothing can be created anywhere — including its own replacement pods. Always exclude system namespaces and your own with a namespaceSelector, and test the webhook-is-down case deliberately before production.