Pods are mortal and their IPs are recycled. A Service is the stable thing you point at instead — a virtual IP and a DNS name that always resolve to whichever pods are currently healthy.
Topic 1: Service → EndpointSlice → Pod
The chain:
- You create a Service with a
selector. - The EndpointSlice controller watches for pods matching that selector that are ready, and maintains a list of their IPs.
- kube-proxy on every node programs kernel rules mapping the Service’s ClusterIP to those pod IPs.
- CoreDNS serves the Service name → ClusterIP.
kubectl get svc api
kubectl get endpointslices -l kubernetes.io/service-name=api
kubectl get endpointslices -l kubernetes.io/service-name=api -o yaml | grep -A4 addresses
EndpointSlices replaced the older Endpoints object (which put every backend in a single resource and did not scale past a few thousand). kubectl get endpoints still works via a compatibility shim; EndpointSlice is what controllers actually use.
The only two reasons a Service has no endpoints:
kubectl get endpointslices -l kubernetes.io/service-name=api
# NAME ADDRESSTYPE PORTS ENDPOINTS AGE
# api-8fk2p IPv4 8080 <unset> 5m ← empty
1. The selector matches no pods.
kubectl get svc api -o jsonpath='{.spec.selector}{"\n"}'
# {"app":"api-server"}
kubectl get pods --show-labels | grep api
# api-7d9f-x2k4 1/1 Running app=api ← "api" ≠ "api-server"
2. Pods match but are not ready.
kubectl get pods -l app=api
# NAME READY STATUS
# api-7d9f-x2k4 0/1 Running ← Running, but NOT ready
A pod is added to the endpoint list only when every container passes its readiness probe. Running is not enough. This is by design and it is the mechanism that makes rolling updates safe.
Check the selector first — it is one command and it eliminates half the possibilities.
Topic 2: The Service Types
ClusterIP — the default
A virtual IP reachable only inside the cluster.
apiVersion: v1
kind: Service
metadata:
name: api
spec:
type: ClusterIP
selector:
app: api
ports:
- name: http
port: 80 # the Service's port
targetPort: 8080 # the CONTAINER's port
protocol: TCP
port and targetPort differ constantly and cause constant confusion. port is what clients dial; targetPort is where the pod listens. Prefer naming the container port and referencing it:
ports:
- name: http
containerPort: 8080
# ...then in the Service:
targetPort: http # by NAME — survives the container port changing
The ClusterIP is not a real interface. Nothing has that address; it exists only as a set of iptables/IPVS rules on each node. You cannot ping it meaningfully, and tcpdump will never show it as a destination past the first hop.
NodePort
Opens the same port (30000–32767 by default) on every node, forwarding to the Service.
spec:
type: NodePort
ports:
- port: 80
targetPort: 8080
nodePort: 30080 # optional; assigned if omitted
Useful for bare-metal and local clusters. Rarely right for production: the port range is unfriendly, you need something in front to load-balance across nodes, and every node opens the port whether or not it runs the pod.
LoadBalancer
A NodePort plus an external load balancer provisioned by the cloud-controller-manager.
kubectl get svc api
# NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
# api LoadBalancer 10.96.4.7 a1b2c3.elb.eu-west-1.amazonaws.com 80:31234/TCP
Stuck in <pending> means no controller is provisioning it — normal on kind/minikube/bare metal, and a signal that something is wrong on a managed cluster.
The cost trap: one cloud load balancer per Service adds up fast. Ten type: LoadBalancer Services is ten load balancers on the bill. This is the main reason Ingress and Gateway API exist — one entry point fronting many Services.
ExternalName
spec:
type: ExternalName
externalName: db.rds.amazonaws.com
Pure DNS: a CNAME, no proxying, no endpoints. Handy for pointing an in-cluster name at an external dependency so applications never learn the real hostname.
Headless — clusterIP: None
spec:
clusterIP: None
selector:
app: postgres
No virtual IP and no proxying. DNS returns the pod IPs directly, one A record per ready pod. Clients do their own selection.
This is what StatefulSets use to give every pod a stable DNS name, and what you want for a database driver that manages its own connection pool per replica, or any client that needs to reach a specific member rather than a random one.
nslookup api.default.svc.cluster.local # ClusterIP: one virtual IP
nslookup postgres.default.svc.cluster.local # headless: N pod IPs
nslookup postgres-0.postgres.default.svc.cluster.local # a specific pod
Topic 3: Cluster DNS
CoreDNS runs as a Deployment in kube-system and every pod is configured to use it:
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl exec -it debug -- cat /etc/resolv.conf
# nameserver 10.96.0.10
# search default.svc.cluster.local svc.cluster.local cluster.local
# options ndots:5
The naming scheme:
<service>.<namespace>.svc.cluster.local Services
<pod-ip-dashes>.<namespace>.pod.cluster.local Pods (10-1-2-9.default.pod.cluster.local)
<pod>.<service>.<namespace>.svc.cluster.local StatefulSet pods via a headless Service
Short names work because of the search list. From a pod in default:
api → tries api.default.svc.cluster.local ✓
api.payments → tries api.payments.svc.cluster.local ✓
api.payments.svc → ✓
ndots:5 — the setting behind most “DNS is slow” reports
ndots:5 means: if a name contains fewer than 5 dots, try every search domain first before treating it as absolute.
Looking up api.example.com (2 dots) therefore produces:
api.example.com.default.svc.cluster.local NXDOMAIN
api.example.com.svc.cluster.local NXDOMAIN
api.example.com.cluster.local NXDOMAIN
api.example.com ✓ (4th attempt)
Four queries — and with IPv4+IPv6 lookups, eight. For a service making thousands of external calls this is a measurable load on CoreDNS and measurable latency in your app.
Two fixes:
# 1. Fully-qualify external names with a TRAILING DOT — skips the search list entirely
# "api.example.com."
# 2. Lower ndots for pods that mostly call outward:
spec:
dnsConfig:
options:
- name: ndots
value: "2"
Debugging DNS:
kubectl run -it --rm dnsutils --image=registry.k8s.io/e2e-test-images/agnhost:2.47 --restart=Never -- sh
nslookup api.default.svc.cluster.local
nslookup kubernetes.default
kubectl logs -n kube-system -l k8s-app=kube-dns --tail=50
kubectl get cm coredns -n kube-system -o yaml # the Corefile
If kubernetes.default resolves but your Service does not, DNS is healthy and the problem is the Service. That single test splits the search space in half.
Topic 4: Session Affinity, Topology and Traffic Policy
spec:
sessionAffinity: ClientIP
sessionAffinityConfig:
clientIP:
timeoutSeconds: 10800
Pins a client IP to one backend. Coarse — behind a NAT gateway every client shares one source IP — so prefer application-level sessions or a proper ingress with cookie affinity.
spec:
internalTrafficPolicy: Local # in-cluster traffic stays on the node
externalTrafficPolicy: Local # external traffic is not re-routed between nodes
externalTrafficPolicy: Local matters for two reasons:
- It preserves the client source IP. The default (
Cluster) SNATs, so your app sees a node IP and every access log and rate limiter is wrong. - It removes an extra network hop.
The cost: traffic arriving at a node with no local pod is dropped, not forwarded. So you need pods spread across the nodes the load balancer targets — which is what health checks on the node port are for.
spec:
trafficDistribution: PreferClose # 1.31+, GA in 1.33
Prefers topologically-close endpoints (same zone), which cuts cross-AZ data transfer cost. It replaces the older topologyKeys and the service.kubernetes.io/topology-mode annotation.
Topic 5: Multi-Port and Named Ports
spec:
ports:
- name: http # name is REQUIRED once there is more than one port
port: 80
targetPort: http
- name: metrics
port: 9090
targetPort: metrics
- name: grpc
port: 50051
targetPort: grpc
appProtocol: grpc # hints for ingress controllers and meshes
Naming ports everywhere pays off later: an Ingress or Gateway can reference port: http rather than a number that changes, and appProtocol lets a proxy pick HTTP/2 for gRPC without guessing.
Topic 6: A Debugging Ladder for “I Cannot Reach the Service”
Work upward; each rung eliminates a layer.
# 1. Does the Service exist and have the selector you think?
kubectl get svc api -o jsonpath='{.spec.selector} {.spec.ports}{"\n"}'
# 2. Are there endpoints? (selector problem vs readiness problem)
kubectl get endpointslices -l kubernetes.io/service-name=api
# 3. Are the pods actually ready?
kubectl get pods -l app=api
# 4. Does the POD answer directly? (bypasses Service entirely)
kubectl port-forward pod/api-7d9f-x2k4 8080:8080
curl localhost:8080/healthz
# 5. Does the SERVICE answer from inside the cluster?
kubectl run -it --rm t --image=nicolaka/netshoot --restart=Never -- \
curl -sS -m5 http://api.default.svc.cluster.local
# 6. Does DNS resolve at all?
kubectl run -it --rm t --image=nicolaka/netshoot --restart=Never -- \
nslookup api.default.svc.cluster.local
# 7. Is a NetworkPolicy blocking it?
kubectl get networkpolicy -A
The high-value split is step 4 vs step 5. If the pod answers on a port-forward but the Service does not, the application is fine and the problem is Service, endpoints, kube-proxy or NetworkPolicy. If the pod does not answer either, stop looking at Kubernetes — it is the app.
Try it yourself: Create a Service whose targetPort is wrong (say 8081 when the app listens on 8080). Note that it has endpoints and looks perfectly healthy, and that only an actual connection attempt fails. Endpoints existing does not mean traffic works.
Common mistake: Assuming a Service load-balances per request. It load-balances per connection. A client using HTTP keep-alive or gRPC opens one connection and pins to one pod for its lifetime — which is why a scaled-up Deployment can leave all traffic on the original pods until clients reconnect. gRPC in particular needs client-side load balancing or a proxy that understands it.