Labels, Selectors & Namespaces

Nothing in Kubernetes references anything else by name. A Service does not know which pods it fronts; a Deployment does not track its pods by ID. They all use labels and selectors, and that indirection is what makes the system composable — and what makes a one-character typo silently disconnect two objects.

Topic 1: Labels Are Queryable; Annotations Are Not

metadata:
  labels:                                    # identifying — SELECTABLE
    app.kubernetes.io/name: api
    app.kubernetes.io/instance: api-prod
    app.kubernetes.io/version: "1.4.2"
    app.kubernetes.io/component: backend
    environment: production
    team: payments
  annotations:                               # descriptive — NOT selectable
    kubernetes.io/change-cause: "rollout 1.4.2"
    prometheus.io/scrape: "true"
    checksum/config: "a3f9c1d8"

The rule of thumb: if something needs to find the object, it is a label. If something needs to know a fact about it, it is an annotation. A git SHA is an annotation. A release channel you route on is a label.

The recommended label set:

The app.kubernetes.io/* prefix is a documented convention, and using it means Helm, dashboards, and most tooling understand your objects without configuration:

app.kubernetes.io/name         api            the application
app.kubernetes.io/instance     api-prod       this deployment of it
app.kubernetes.io/version      1.4.2          the version
app.kubernetes.io/component    backend        the role within the app
app.kubernetes.io/part-of      checkout       the wider system
app.kubernetes.io/managed-by   helm           what deploys it

Do not put version in a Service selector. It seems reasonable and it breaks every rollout: during the update half the pods carry the old version label and drop out of the Service, so you deliberately black-hole traffic mid-deploy.

Topic 2: Selectors

# equality-based
kubectl get pods -l app=api
kubectl get pods -l app=api,env=prod              # comma = AND
kubectl get pods -l 'env!=dev'

# set-based
kubectl get pods -l 'app in (api,web)'
kubectl get pods -l 'env notin (dev,staging)'
kubectl get pods -l 'app'                          # key EXISTS, any value
kubectl get pods -l '!app'                         # key does NOT exist

That last one is genuinely useful — kubectl get pods -A -l '!app.kubernetes.io/name' finds everything in the cluster that does not follow your labelling convention.

In YAML the two forms are:

selector:
  matchLabels:                 # equality, ANDed
    app: api
    tier: backend
  matchExpressions:            # set-based
    - key: environment
      operator: In
      values: [production, staging]
    - key: deprecated
      operator: DoesNotExist

matchLabels and matchExpressions are ANDed together. Operators are In, NotIn, Exists, DoesNotExist.

Services only support equality selectors (matchLabels-style, in the older flat form). Set-based expressions work for Deployments, ReplicaSets, NetworkPolicies and Jobs, but a Service selector is a plain map — a real asymmetry that catches people writing their first NetworkPolicy after only ever writing Services.

Topic 3: How Objects Actually Connect

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  selector:
    matchLabels:
      app: api              # ← 1. the Deployment adopts pods with this label
  template:
    metadata:
      labels:
        app: api            # ← 2. and stamps that label on the pods it creates
---
apiVersion: v1
kind: Service
metadata:
  name: api
spec:
  selector:
    app: api                # ← 3. the Service independently selects the same label
  ports:
    - port: 80
      targetPort: 8080

Three objects, no direct references. The Service has never heard of the Deployment. If you deleted the Deployment and created a bare pod with app: api, the Service would route to it happily.

The rule that prevents most of the pain: spec.selector.matchLabels must be a subset of spec.template.metadata.labels. If they disagree, the Deployment creates pods it does not consider its own, notices it still has zero, and creates more — forever. Kubernetes validates this at admission for Deployments, which is why the error is caught early; it is not validated for the Service, which is why a Service with no endpoints is such a common silent failure.

# The one-line check for "why does my Service have no endpoints"
kubectl get svc api -o jsonpath='{.spec.selector}{"\n"}'
kubectl get pods -l app=api --show-labels
kubectl get endpointslices -l kubernetes.io/service-name=api

If the selector is right and pods exist but endpoints are empty, the pods are not ready — that is a probe problem, not a label problem.

Selectors are immutable on Deployments:

kubectl apply -f deploy.yaml
# The Deployment "api" is invalid: spec.selector: Invalid value: ...
#   field is immutable

Changing a selector would orphan every existing pod, so the API refuses. To change it you delete and recreate the Deployment — which is precisely why you should choose selector labels that will never need to change (app: api, not version: 1.4).

Topic 4: Ownership and Cascading Deletion

Objects created by controllers carry an ownerReferences entry:

kubectl get pod api-7d9f-x2k4 -o jsonpath='{.metadata.ownerReferences}' | jq
# [{"apiVersion":"apps/v1","kind":"ReplicaSet","name":"api-7d9f","uid":"...","controller":true}]

This forms a chain: Deployment → ReplicaSet → Pod. Deleting the Deployment garbage-collects the whole tree.

kubectl delete deploy api                          # cascades: RS and pods go too
kubectl delete deploy api --cascade=orphan         # leaves pods running, unmanaged
kubectl delete deploy api --cascade=foreground     # waits for children first

--cascade=orphan is an occasionally invaluable trick: it lets you replace a controller without dropping traffic, because the pods keep serving while unowned. It is also a good way to leak resources nothing will ever clean up, so use it deliberately.

Topic 5: Namespaces

A namespace is a name scope, not a security boundary by itself.

kubectl get namespaces
kubectl create namespace payments
kubectl config set-context --current --namespace=payments

What namespaces give you:

• Name uniqueness — a Service called api can exist in ten namespaces.
• A DNS segment — api.payments.svc.cluster.local.
• A scope for RBAC — Roles and RoleBindings are namespaced.
• A scope for ResourceQuota and LimitRange.
• A target for NetworkPolicy via namespaceSelector.

What they do not give you:

• Network isolation. By default every pod can reach every other pod in the cluster, across namespaces. That requires NetworkPolicy.
• Node isolation. Pods from different namespaces share nodes and their kernels.
• A hard security boundary. For untrusted tenants you need separate clusters or a sandboxed runtime (gVisor, Kata).

Namespaced vs cluster-scoped:

kubectl api-resources --namespaced=true    | head
kubectl api-resources --namespaced=false   # Nodes, PVs, StorageClasses, ClusterRoles, CRDs, Namespaces

Trying to put a namespace on a cluster-scoped object is silently ignored, which produces the “I deleted it from my namespace and it is still there” confusion.

Deleting a namespace that will not delete:

kubectl delete namespace payments
# hangs in Terminating forever
kubectl get namespace payments -o jsonpath='{.spec.finalizers}{"\n"}{.status.conditions}' | jq

A namespace stuck Terminating almost always means a finalizer on some object inside it cannot complete — commonly a CRD whose operator has already been uninstalled, so nothing is left to run the finalizer. Find the offending resource rather than force-removing the finalizer, which leaks whatever the finalizer existed to clean up.

Topic 6: A Labelling Scheme That Survives

Decide these once, at the start:

labels:
  app.kubernetes.io/name: api          # NEVER changes — safe for selectors
  app.kubernetes.io/instance: api-prod # NEVER changes — safe for selectors
  app.kubernetes.io/version: 1.4.2     # changes every release — NEVER in a Service selector
  app.kubernetes.io/component: backend
  app.kubernetes.io/part-of: checkout
  team: payments                       # ownership — for cost allocation and paging
  environment: production

Two rules carry the whole scheme:

1. Selector labels must be immutable. Anything that changes per release goes in a non-selector label or an annotation.
2. Label for the questions you will ask at 3am. “Show me everything the payments team owns in production” should be one command. If it is not, add the label now.

kubectl get pods -A -l team=payments,environment=production
kubectl get all -A -l app.kubernetes.io/part-of=checkout

Consistent team labels are also what make cost-allocation tooling work at all — retrofitting them across a live cluster is far harder than deciding them up front.

Try it yourself: Pick any workload in your cluster and run kubectl get pod <name> --show-labels. Ask whether you could answer “who owns this and what release is it” from labels alone. If not, that is the gap to close.

Common mistake: Putting version in a Service selector. During a rolling update the Service selects only pods carrying the old version label, so as the rollout proceeds your endpoint count falls toward zero and then jumps back — a self-inflicted outage that looks exactly like a readiness problem.