The Pod Network: CNI, kube-proxy & conntrack

Kubernetes networking has exactly one rule and a great deal of implementation. The rule: every pod gets its own IP, and every pod can reach every other pod without NAT. Everything else is how a CNI plugin makes that true on your infrastructure.

Topic 1: The Four Requirements

The network model demands:

1. Every pod has a unique IP, cluster-wide.
2. Pods can communicate with all pods without NAT.
3. Agents on a node (kubelet, daemons) can reach all pods on that node.
4. A pod sees its own IP as the same address others use to reach it.

Requirement 4 matters more than it looks: it is why applications can register their own address with a service registry and have it work. Docker’s default bridge networking violates it, which is one of several reasons Kubernetes does not use it.

Kubernetes implements none of this itself. It defines the contract and delegates to a CNI plugin — Calico, Cilium, Flannel, AWS VPC CNI, Azure CNI. This is why a cluster with no CNI installed has every pod stuck in ContainerCreating forever: the kubelet cannot complete pod setup without something to assign an IP.

Topic 2: The Packet Path

Step by step, for curl http://api from a pod:

1. DNS. api → api.default.svc.cluster.local → CoreDNS returns the ClusterIP 10.96.4.7.

2. The pod’s netns. The packet leaves the container’s eth0, which is one end of a veth pair. The other end lives in the node’s root namespace, usually attached to a bridge or handled directly by the CNI.

3. kube-proxy’s rules — the important hop. In the node’s kernel, iptables or IPVS rules match the ClusterIP and perform DNAT, rewriting the destination to one of the ready pod IPs, chosen at random (or by IPVS’s scheduler).

4. Routing to the destination node. If the target pod is on another node, the CNI’s data path carries it — an overlay (VXLAN/Geneve encapsulation), native routing (the VPC knows the pod CIDRs), or eBPF.

5. Into the destination pod’s netns via its veth pair.

6. The reply goes back and the conntrack entry created at step 3 reverses the DNAT, so the client sees the reply coming from the ClusterIP it dialled — as it must, or the connection would be rejected.

The ClusterIP never exists as an interface anywhere. It is purely a match target in kernel rules. That is why you cannot ping it usefully and why it never appears past the first hop in a capture.

Topic 3: kube-proxy Modes

kubectl -n kube-system get cm kube-proxy -o jsonpath='{.data.config\.conf}' | grep mode
sudo iptables -t nat -L KUBE-SERVICES -n | head -20
sudo ipvsadm -Ln | head -20

The practical difference appears at scale: with iptables, adding one Service means rewriting the whole ruleset, so a cluster with 5,000 Services can take seconds to converge after a change. During that window, endpoints are stale. IPVS and eBPF do incremental updates.

Topic 4: Conntrack — the Silent Killer

Every DNAT’d connection needs a conntrack entry so the return packet can be un-NAT’d. The table is finite.

cat /proc/sys/net/netfilter/nf_conntrack_max      # e.g. 262144
cat /proc/sys/net/netfilter/nf_conntrack_count    # current
sudo conntrack -L | wc -l
sudo conntrack -S                                 # per-CPU stats, including drops

When it fills:

nf_conntrack: table full, dropping packet

The symptoms are what make this hard: intermittent connection failures, correlated with load, affecting random connections, with nothing in application logs — the packet was dropped in the kernel before anything in userspace saw it. Retries usually succeed, so it looks like flakiness rather than a limit.

Causes worth knowing:

• High connection churn — short-lived connections leave entries in TIME_WAIT for 120s by default.
• A load test, or a client without connection pooling.
• A node running far more pods than planned.

# Raise the ceiling (and make it persist)
sysctl -w net.netfilter.nf_conntrack_max=1048576
# Shorten how long closed entries linger
sysctl -w net.netfilter.nf_conntrack_tcp_timeout_time_wait=30

# Alert BEFORE it fills
node_nf_conntrack_entries / node_nf_conntrack_entries_limit > 0.8

That alert is worth adding to any cluster today. The failure mode is invisible until it is severe, and the metric is already exported by node-exporter.

Topic 5: MTU — the Other Silent Killer

Overlay networks encapsulate packets, and encapsulation costs bytes:

If the pod’s MTU is larger than the real path MTU, large packets must fragment — and if DF is set (as TCP does) they are dropped, with an ICMP “fragmentation needed” reply that cloud security groups very often filter.

The signature is distinctive and confusing: small requests work, large ones hang. A TLS handshake completes (small packets), then the first full-size response stalls forever. curl hangs after printing headers. Health checks pass; real traffic does not.

kubectl exec -it netshoot -- ip link show eth0        # pod MTU
ip link show ens5                                     # node MTU

# Find the real path MTU — send unfragmentable packets of decreasing size
kubectl exec -it netshoot -- ping -M do -s 1472 -c 2 <other-pod-ip>
# 1472 + 28 (ICMP+IP headers) = 1500

If 1472 fails and 1422 succeeds, your path MTU is 1450 and the pod MTU is wrong.

This bites hardest on jumbo-frame networks (AWS instances default to 9001) where a CNI configured for 1500 silently caps you, and on VPN/WireGuard paths where the overhead is larger than the CNI assumed.

Topic 6: Choosing and Debugging a CNI

The AWS VPC CNI constraint is worth calling out because it surprises people: each instance type supports a fixed number of ENIs and IPs per ENI, so the pod-per-node limit is an IP limit, not a CPU limit. An m5.large supports 29 pods regardless of how idle it is. Prefix delegation raises this substantially and is worth enabling.

The debugging ladder:

# 0. Is the CNI even healthy?
kubectl get pods -n kube-system -l k8s-app=cilium   # or calico-node, aws-node
kubectl logs -n kube-system ds/aws-node --tail=50

# 1. Pod-to-pod DIRECTLY (bypasses Service entirely)
kubectl exec -it netshoot -- curl -sS -m5 http://10.1.7.4:8080/healthz

# 2. Pod-to-Service (adds kube-proxy)
kubectl exec -it netshoot -- curl -sS -m5 http://api.default.svc.cluster.local

# 3. DNS alone
kubectl exec -it netshoot -- nslookup api.default.svc.cluster.local
kubectl exec -it netshoot -- nslookup kubernetes.default

# 4. Is a NetworkPolicy responsible?
kubectl get networkpolicy -A

# 5. The rules themselves, on the node
sudo iptables -t nat -L KUBE-SERVICES -n | grep 10.96.4.7
sudo ipvsadm -Ln | grep -A3 10.96.4.7

# 6. The wire
sudo tcpdump -i any -nn "host 10.1.7.4 and port 8080"

The split that saves the most time is step 1 vs step 2. Direct pod IP works but the Service does not → kube-proxy, endpoints, or policy. Neither works → CNI or the application.

Try it yourself: From a netshoot pod, run curl against a Service while capturing on the node with tcpdump -i any -nn port 8080. Find the packet where the destination is the ClusterIP, and the one where it is a pod IP. There is no packet in between — the rewrite is done in the kernel, in place.

Common mistake: Blaming the application for intermittent 5xx errors under load without checking conntrack. The kernel drops the packet before your app sees the connection, so there is nothing to log — the absence of evidence is itself the clue. Check nf_conntrack_count against nf_conntrack_max on the affected nodes before spending a day in application traces.