SRE Interview Preparation

STAR Narrative

Situation: I was the senior DevOps engineer on an airline operations platform running on GKE — real-time flight, booking, and operational systems processing millions of transactions daily. One afternoon, API latency across multiple services spiked simultaneously, with no single service reporting errors — which is exactly the kind of symptom pattern that sends you down the wrong path if you don't know what you're looking for. Alerts started firing for elevated response times, but nothing pointed at an obvious culprit. No recent deploys, no config changes.

Task: I needed to find the actual root cause fast — this wasn't a "look into it tomorrow" situation, it was actively degrading the operational platform in production — and fix it in a way that wouldn't just come back the next time load spiked.

Action: My first instinct was to check node and pod health across the cluster — everything came back green, which was the first red flag, not a reassurance. A cluster that's "healthy" by every standard dashboard while services are still failing almost always means the problem is somewhere silent: DNS, or something between components rather than inside any one of them. CoreDNS specifically doesn't crash or throw obvious errors under CPU pressure — it just gets slower, and slow DNS resolution shows up as generic timeouts and latency everywhere else in the system, not as a DNS error anywhere.

I pulled up Datadog and correlated DNS query volume against the CoreDNS pods' resource usage and found query throughput climbing well past what the current pod count could handle — there was no HPA configured on CoreDNS at all, so it had a fixed capacity ceiling that request volume had simply outgrown. I confirmed this wasn't just an external connectivity issue by checking internal service-to-service DNS resolution specifically, not just whether the pods could reach the public internet — those are two different health signals, and one being fine doesn't mean the other is.

Once I had the root cause, I did an emergency kubectl scale to horizontally scale CoreDNS immediately and stop the bleeding. Then, because a manual scale-up doesn't survive a redeploy or protect you the next time load spikes, I went back and codified the real fix into our Terraform-managed manifests: a proper HPA policy on CoreDNS tied to query-rate thresholds, tuned cache TTLs so repeat lookups didn't keep re-hitting CoreDNS unnecessarily, and adjusted resource requests/limits so the pods weren't starting from an under-provisioned baseline. Last thing — I added DNS resolution latency at P99 as a first-class SLI in Datadog with proactive alerting on query-rate thresholds, because the whole incident happened in the first place due to a blind spot: nobody was watching DNS latency as its own signal.

Result: Full service was restored in under 20 minutes. DNS resolution latency dropped 65% after the fix. We hit 99.99% availability on that SLO going forward with zero recurrence over the next 12 months, and I standardized the same DNS SLO dashboard across every GKE cluster we ran, not just the one that had the incident — closing that observability gap everywhere at once instead of waiting for it to bite us again somewhere else.

STAR Narrative

Situation: In the middle of investigating an unrelated production issue, my team realized we had no logs — anywhere. The centralized logging pipeline across the entire GKE cluster had gone silent. This is about the worst timing a failure like this can have: we needed logs specifically because something else was going on, and now we were debugging blind on two fronts at once.

Task: Restore full log visibility as fast as possible, and — separately — make sure this specific kind of silent failure couldn't happen again undetected, since the scariest part wasn't that logging broke, it was that nothing told us it had.

Action: I treated this as a pipeline with distinct stages and checked each one independently rather than guessing: application stdout, the Fluentd/FluentBit DaemonSet running on every node, the aggregation layer, and Datadog log ingestion itself. Using kubectl logs against the DaemonSet pods and FluentBit's own internal metrics, I found the actual break: a TCP connection in the forwarding pipeline had failed, and it was silently dropping events on the floor instead of erroring loudly — from the infrastructure's point of view, everything still looked healthy, because the DaemonSet pods themselves were running fine; they just weren't successfully shipping anything downstream. That's the trap with this class of failure — a broken forwarding connection doesn't crash the pod holding it, so standard health checks don't catch it.

I reconfigured the FluentBit forward plugin with proper retry logic and buffer tuning to restore the connection and handle transient failures going forward, and I did it without restarting any actual workloads — the fix lived entirely in the logging layer, so there was no reason to touch anything running application traffic. Once the pipeline was back, I didn't stop there — I built end-to-end validation using synthetic log injection with assertion monitors in Datadog, so the system would actively confirm the pipeline was working by injecting a known test log and verifying it arrived, rather than just assuming health from the absence of errors.

Result: Full log visibility was restored across all services within 35 minutes. Because we now had reliable, always-on observability going forward, MTTR on subsequent incidents dropped by 40% — teams weren't losing time re-discovering "wait, do we even have logs right now" mid-incident anymore. And the blind-debugging scenario itself was permanently eliminated, since log pipeline health became a continuously monitored first-class signal instead of something we only found out about when it had already failed.

STAR Narrative

Situation: On an EKS cluster running critical product workloads, deployments started getting stuck — some in Pending, some in Terminating — and this wasn't isolated to one team's service, it was blocking release pipelines across multiple product teams simultaneously. The worst part: every monitoring dashboard we had said the cluster was healthy. Nodes green, standard metrics green — what I've since come to think of as the "green illusion," where the dashboards you'd normally trust are exactly what's giving you false confidence.

Task: Figure out what was actually happening at the control-plane level — since dashboards were useless here — restore the ability to deploy, and close whatever monitoring gap let this go undetected in the first place.

Action: Because the symptom was deployments stuck rather than any application misbehaving, I treated this as a control-plane problem from the start, not a data-plane one — that distinction matters, because it tells you where to even start looking. I compared kubectl cluster-info output and pod states directly across the control-plane components, which is what actually surfaced inconsistencies that the aggregated dashboards were smoothing over — individual component state told a different story than the summary view did. From there I went into the API server's own audit logs, and that's where I found it: admission webhook calls were consistently timing out at exactly 5 seconds. Every write request that had to pass through that webhook was queuing up waiting on a timeout that kept happening, and that queuing was enough to effectively deadlock the controller-manager — it wasn't crashed, it was just permanently backed up behind requests that would never complete in time.

I patched the admission webhook's timeout configuration and its failure policy so a slow or failing webhook call couldn't block the entire write path going forward, then verified etcd latency normalized back to baseline as confirmation the fix actually resolved the underlying pressure, not just the symptom. I also cross-checked Jenkins pipeline logs specifically to confirm the blocked deployments were a downstream consequence of this control-plane state, not a separate application-config problem that happened to be coinciding with it — you don't want to fix one thing and assume it explains everything without checking. Last step: I instrumented admission webhook latency as its own dedicated Prometheus metric with a Grafana alerting threshold, because the entire incident was possible in the first place due to a monitoring blind spot — control-plane health summary dashboards existed, but nothing was watching webhook latency specifically, which is exactly the layer that actually broke.

Result: Cluster write operations and deployment pipelines were restored within 45 minutes. The "green illusion" itself got eliminated as a risk going forward — control-plane health, specifically at the webhook layer, is now a first-class monitored metric, not something inferred from a summary dashboard. We introduced a formal admission webhook SLO off the back of this, and had zero similar incidents in the following 18 months.