The API Object Model & kubectl

kubectl is a thin HTTP client. Everything it does, you could do with curl — and knowing that turns it from a set of memorised incantations into something you can reason about.

Topic 1: Everything Is a REST Resource

The API is organised into groups, versions and kinds, and every object’s address follows from those three:

/api/v1/namespaces/default/pods/my-pod                    ← core group (empty name, legacy path)
/apis/apps/v1/namespaces/default/deployments/api          ← apps group
/apis/networking.k8s.io/v1/namespaces/default/ingresses   ← networking group

That is why manifests start the way they do:

apiVersion: apps/v1        # group/version
kind: Deployment           # the resource type
metadata:
  name: api
  namespace: default
spec: {}

apiVersion: v1 (no slash) means the core group, which predates grouping. Everything newer is namespaced under a group so that the API can evolve independently per area.

kubectl api-resources                    # every kind, its short name, group, and scope
kubectl api-resources --namespaced=false # cluster-scoped kinds: nodes, PVs, ClusterRoles
kubectl api-versions                     # every group/version served

kubectl api-resources is the honest answer to “what can I even create here” — on a cluster with operators installed it will list far more than the built-ins.

Version suffixes mean stability, and they are a promise:

An object stored as v1beta1 and served as v1 is the same object — the API server converts between versions on read. This is what makes upgrades survivable, and why kubectl get -o yaml may show a different apiVersion than you applied.

Topic 2: What kubectl apply Actually Does

The steps in words:

1. kubectl reads your YAML, converts it to JSON, and sends an HTTP request to the API server.
2. Authentication (who are you — cert, token, OIDC), then authorisation (RBAC: may you do this), then admission (mutating webhooks may change the object; validating webhooks may reject it). Covered fully in the admission lesson.
3. etcd write. The spec is now durable. kubectl apply returns here.
4. A controller notices the difference through a watch — not a poll — and acts.
5. The scheduler assigns a node, if the object produced pods.
6. The kubelet on that node starts containers and reports status.

Because your prompt returns at step 3, a CI job that runs kubectl apply and exits has verified only that the YAML was accepted. To actually wait:

kubectl apply -f deploy.yaml
kubectl rollout status deployment/api --timeout=5m     # blocks until rolled out or fails
kubectl wait --for=condition=available deploy/api --timeout=5m
kubectl wait --for=condition=ready pod -l app=api --timeout=2m

kubectl rollout status returning non-zero is the check your pipeline needs. Without it, a deploy that crash-loops forever is a green build.

Topic 3: apply vs create vs replace vs patch

Why apply is not replace:

apply performs a three-way merge between your manifest, the live object, and a record of what you applied last time. That is what lets a Deployment you manage in Git coexist with an HPA that owns spec.replicas — the HPA’s change is not in your manifest, so apply leaves it alone.

Since 1.22 this is server-side apply, and the record lives in metadata.managedFields:

kubectl get deploy api -o yaml | grep -A20 managedFields

Each field is tagged with the manager that set it. When two managers fight over the same field you get an explicit conflict rather than silent flapping:

kubectl apply --server-side --field-manager=ci --force-conflicts -f deploy.yaml

Common mistake: Setting replicas in a Git-managed manifest while an HPA is also managing it. Every apply resets the count, the HPA scales it back, and the Deployment oscillates. Remove replicas from the manifest entirely when an HPA owns it.

Topic 4: Reading Any Object Without the Docs

kubectl explain reads the OpenAPI schema the cluster itself serves, so it is always correct for your version — including CRDs installed by operators.

kubectl explain pod.spec.containers
kubectl explain deployment.spec.strategy.rollingUpdate
kubectl explain pod.spec --recursive | head -60      # the whole tree
kubectl explain networkpolicy.spec.egress --recursive

This is more reliable than a web search, which will happily hand you a field from a different version.

Getting exactly what you want out of get:

kubectl get pods -o wide                     # + node, IP, nominated node
kubectl get pods -o yaml                     # everything
kubectl get pods -o json | jq '.items[].metadata.name'

# JSONPath — no jq required
kubectl get pods -o jsonpath='{range .items[*]}{.metadata.name}{"\t"}{.status.phase}{"\n"}{end}'

# Custom columns — the readable option
kubectl get pods -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,QOS:.status.qosClass'

# Sorting and filtering
kubectl get pods --sort-by=.status.startTime
kubectl get pods --field-selector status.phase=Running
kubectl get pods -l 'app in (api,web),env!=dev'
kubectl get pods -A                          # every namespace

Two that repay learning properly: custom-columns turns any field into a table without jq, and field-selector filters server-side, which matters on a cluster with 50,000 pods where | grep transfers all of them first.

The four commands that answer most questions:

kubectl describe <kind> <name>      # human summary + EVENTS at the bottom
kubectl get events --sort-by=.lastTimestamp
kubectl logs <pod> -c <container> --previous     # --previous = the crashed instance
kubectl get <kind> <name> -o yaml   # ground truth

kubectl logs --previous is the one people forget. When a container is crash-looping, kubectl logs shows the current attempt, which has usually produced nothing yet. --previous shows the instance that actually died, and that is where the error is.

Topic 5: Namespaces, Contexts and Not Deploying to Prod by Accident

kubectl config get-contexts
kubectl config current-context
kubectl config use-context staging
kubectl config set-context --current --namespace=payments

Every kubectl command runs against whatever context happens to be current — which is a loaded gun in any environment with more than one cluster. Two habits worth adopting:

# 1. Put the context in your shell prompt (kube-ps1, starship, oh-my-zsh plugin)
# 2. Be explicit in anything scripted or dangerous:
kubectl --context=staging --namespace=payments delete deploy api

kubectx and kubens are worth installing purely for the reduction in cognitive load.

Topic 6: Debugging kubectl Itself

When a command does something you did not expect, make it show you the wire:

kubectl get pods -v=6      # URLs and status codes
kubectl get pods -v=8      # full request and response bodies
kubectl get pods -v=9      # + curl equivalent you can paste

-v=8 settles arguments about what a command actually sent. And to talk to the API directly:

kubectl proxy --port=8001 &
curl -s localhost:8001/api/v1/namespaces/default/pods | jq '.items | length'
curl -s localhost:8001/apis/apps/v1/namespaces/default/deployments/api | jq .spec.replicas

kubectl get --raw='/api/v1/namespaces/default/pods?limit=5' | jq .

Two checks worth knowing before you need them:

kubectl auth can-i create deployments --namespace payments
kubectl auth can-i '*' '*' --all-namespaces
kubectl auth can-i delete pods --as=system:serviceaccount:default:my-sa

kubectl auth can-i --as= answers “does this ServiceAccount have permission” without deploying anything, and turns most RBAC debugging into a single command.

Try it yourself: Run kubectl get pods -v=9 and copy the printed curl command. Run it directly and confirm you get identical JSON. That is the whole of kubectl — an HTTP client with good ergonomics.

Common mistake: Using kubectl edit to fix something in a cluster managed by GitOps or a Deployment. The change survives until the next reconcile and then vanishes, which produces the maddening experience of a fix that “works for ten seconds”. Change the source of truth, not the live object.