SRE Labs (Advanced Track) — Titan Grid (Session 2): Configuration Automation, Self-Service Applications & CI/CD Pipeline Design

Structured educational resource covering sre labs (advanced track) — titan grid (session 2): configuration automation, self-service applications & ci/cd pipeline design.

senior 45 min read 11 sections

#kubernetes#cloud-k8s#aws#gcp

Immutable Rotational Patching, the “DevSpace” Access-Automation Platform, and Progressive Multi-Region Deployment

2. Table of Contents

Executive Summary
Table of Contents
Detailed Structured Notes
- 3.1 Session Context & Recap
- 3.2 Opening War Story — Classic vs. YAML Pipelines and the Cost of Non-Modularity
- 3.3 The Automation Category Taxonomy (Revisited)
- 3.4 Self-Service Application vs. Configuration Automation — Why They’re Kept Separate
- 3.5 Patching Automation — The Manual Process First
- 3.6 Patching Automation — Immutable Rotational Strategy & Script Walkthrough
- 3.7 Patching Automation — GitHub Actions Pipeline & Manual Trigger Gates
- 3.8 The “DevSpace” Self-Service Application — Full Walkthrough
- 3.9 Third-Party Automation Platforms (Flight Control, QBR.AI)
- 3.10 Participant Pushback — The “We Need to See This in Action” Moment
- 3.11 CI/CD — Repository Structure & the Three-Phase Pipeline
- 3.12 CI/CD — Progressive Multi-Region Deployment & Deployment Strategies
- 3.13 CI/CD — Database Schema Migration Under Canary Deployment
- 3.14 Program Logistics
Key Concepts Table
Architecture & Workflow Analysis
Commands & Configurations
Tools & Technologies
Real-World Production Usage
Interview Preparation (Beginner / Intermediate / Advanced)
Exam & Certification Notes
Cheat Sheet
Gaps & Assumptions

3. Detailed Structured Notes

3.1 Session Context & Recap

Brief recap of the prior session: business context (Phase 1) and full system architecture (Phase 2) were completed — the multi-region request flow, domain decomposition, data layer, event layer, and home-region write-ownership pattern. This session picks up with Phase 3 (automation) and begins Phase 4 (CI/CD), with the explicit goal of covering both phases (and Terraform, time permitting) in this call.

3.2 Opening War Story — Classic vs. YAML Pipelines and the Cost of Non-Modularity

Co-instructor Ravi opened with a detailed, real account, preserved because it directly motivates the CI/CD structuring principles covered later in the session:

A US client, early in cloud adoption, used Azure Classic (GUI-based) pipelines rather than YAML-based pipelines — a common early-stage choice because visual/GUI-based configuration is more approachable for teams coming from a systems-administration background rather than a scripting background.
The failure mode: as the client’s microservice count grew past ~100, the pipelines had no modularity or reusability. A single organization-wide requirement (e.g., “add a SonarQube scan to every pipeline”) required manually editing every single one of the 100+ classic pipelines individually.
Compounding failure mode: onboarding a new microservice meant copy-pasting an existing working pipeline and manually editing parameters — a manual, repetitive process with a high error rate. Real, concrete consequence cited: databases got mixed up, and services that should never have touched production did connect to production due to copy-paste/parameter errors.
Resolution: the client eventually migrated to YAML-based pipelines, enabling shared modules, common build/quality-gate logic, and single-point-of-change updates — and has since progressed further toward GitOps, since most of their codebase already lived in Git.
Explicit lesson stated: it’s acceptable to take the easiest implementation path early on, but maintenance cost compounds over time — the earlier you modularize and automate pipeline structure, the less pain you accumulate as the system scales.

3.3 The Automation Category Taxonomy (Revisited)

The seven automation categories from the earlier program sessions were reaffirmed and organized into two higher-level buckets:

Bucket 1 — Self-Service Application scope:

Access automation
Secrets/credentials automation
Observability automation
Incident automation
Provisioning automation (infrastructure creation)

Bucket 2 — Configuration Automation (kept separate, explained in 3.4):

Patching automation
Security hardening / OS-level configuration automation

Organizational team structure reaffirmed (from a real prior organization the instructor worked at): a BAU (Business-As-Usual) team handling day-to-day tickets, a project team handling scaling/cost/security/migration initiatives, and an automation team specifically dedicated to reducing the BAU team’s ticket workload through automation — described as the single most leveraged team in a mature DevOps organization.

A real, cited audit consequence of poor access automation: at a prior heavily-regulated organization, an internal security audit found that many developers and test engineers had standing access to production databases containing PII and SPI data — caught internally (avoiding external penalty), but explicitly cited as the kind of finding that can result in significant regulatory fines if caught by an external auditor instead.

3.4 Self-Service Application vs. Configuration Automation — Why They’re Kept Separate

A specific, explicit, hard-won piece of architectural guidance:

Rule stated directly: do not fold configuration automation (patching, security hardening) into the same general-purpose self-service application used for routine access/infrastructure requests.
Reasoning: based on the instructor’s own observation across “the last 3 to 5 years,” teams that included configuration/patching automation inside a broader self-service application repeatedly ran into operational issues and audit problems.
Practical implication: security-sensitive, blast-radius-heavy automation (patching, hardening) should live in its own, separately governed pipeline with stricter manual-trigger gating (detailed in Section 3.7), distinct from the lower-risk, higher-volume self-service platform covering access requests, secret rotation requests, and similar routine work.

3.5 Patching Automation — The Manual Process First

Before automating, the instructor walked through the manual patching process that most organizations start from, to establish what’s actually being automated:

A security scanning tool (e.g., Snyk or similar) throws a vulnerability alert (“bit”) to the DevOps team’s dashboard.
The security team classifies the alert by severity — critical, warning, or normal.
A change ticket is created and requires manager/upper-management approval before any action.
Only after approval does the actual patching begin — executed region by region, sequentially (not in parallel), to control blast radius.
Health is validated after each region’s patch before proceeding to the next.
A global completion confirmation and an audit log entry close out the activity.

3.6 Patching Automation — Immutable Rotational Strategy & Script Walkthrough

Core principle: never patch in-place. In-place patching causes configuration drift and is very difficult to debug when a step fails partway through a multi-step process. The correct pattern is immutable, rotational patching: build new (patched) infrastructure alongside the old, validate it, and only then retire the old infrastructure — never mutate a live resource directly.

Inventory context used for the worked example: 3 regions (AP South 1, EU, US), ~20 node pools, ~500 microservices — explicitly chosen to illustrate why manual patching at this scale is untenable.

Script-driven patch orchestration flow, as demonstrated (a simplified educational version, with the real production version also shared separately):

Notify the team (via Slack, using a Slack API token) that a patching window has started.
Process regions sequentially, never in parallel — explicitly to control blast radius: “if anything is going to break in one environment, the same thing will exactly break in all the environments” if run in parallel.
For each region: provision new node groups using an updated, security-patched AMI.
Drain the old node (respecting PodDisruptionBudgets — explicitly flagged as a lesson learned from a past production outage where the process got stuck at this exact stage, discussed further below).
Validate pod health on the new nodes before proceeding.
Delete the old node only after validation succeeds.
Run a full cluster health validation — covering multiple distinct dimensions, not just basic pod status:
- Infrastructure-level: API server health, absence of CrashLoopBackOff errors.
- Application-level (synthetic monitoring): a dedicated script using a synthetic test user that actually logs into the payment application and performs a real test transaction — explicitly justified because crash-loop/readiness-probe checks alone cannot confirm a payment-critical service is actually functionally correct; “if your pods are healthy and the application is not working, that can be the case.”
- Observability-level: scraping the Prometheus endpoint directly to check SLO compliance, error rate, and P95 latency (must remain under the platform’s 200ms target) — with a deliberate 2-minute stabilization wait before evaluating live traffic, to avoid false positives from momentary post-deployment noise.
Only after full validation succeeds does the rollout promote to the next region.
If any region’s validation fails, the entire rollout halts immediately — explicitly, no promotion to subsequent regions is permitted after any regional failure: “if anything is failing right there, just perform exit directly… that’s the most important thing you have to understand if you are playing with multi-region execution in any kind of automation.”

A specific, real lesson incorporated live, mid-explanation: the instructor noted, live, that an earlier iteration of this exact script was missing an explicit PDB check before draining nodes — directly connecting back to a real past production outage (referenced elsewhere in this series) that got stuck specifically at the node-draining stage due to a Pod Disruption Budget conflict. The instructor used this as a teaching moment: automation scripts should be continuously refined based on real incidents, and additional checks (PDB status, HPA minimum replica confirmation, database connection validation, background job status) should be added as an organization’s understanding of its own failure modes matures.

Roll-back principle, stated explicitly: “the upgrades and the patching… can take a lot of time, but that rollback should be as quick as possible” — asymmetric time budgets are appropriate: slow, careful forward progress, but near-instant rollback capability.

3.7 Patching Automation — GitHub Actions Pipeline & Manual Trigger Gates

The scripted process above was then wrapped in an actual GitHub Actions pipeline — but with a deliberately non-fully-automated trigger:

The pipeline explicitly requires a manual trigger with required input fields: change reason, change ticket ID, CVE reference, and patch type.
Explicit justification for keeping this manual: “we don’t allow automated patching… it is somehow related to security and that needs to be validated while execution” — this manual gate is framed as intentional friction that prevents accidental/unauthorized production changes, not a gap in automation maturity.
Pipeline steps mirror the script logic: install Ansible → notify Slack per region → execute → validate health → roll back on failure → promote to next region only on success → final success notification.
A parallel-deployment example was also shown deliberately as a negative example — to make the contrast between sequential (correct, blast-radius-controlled) and parallel (dangerous) rollout patterns concrete rather than abstract.
Suggested further maturity step: automating the change-ticket creation itself for critical-severity findings specifically (auto-creating and routing a ticket to the relevant manager for approval), while keeping the actual patch execution behind the same manual trigger gate.

3.8 The “DevSpace” Self-Service Application — Full Walkthrough

A detailed architectural and UI walkthrough of a real internal platform (referred to as “DevSpace”) built to serve a ~3,000-person development organization.

The motivating business problem, stated with real numbers: a 4-5 person BAU team was receiving 50-60 tickets per day, against SLAs of 24 hours for normal tasks (48-72 hours for some normal tasks depending on type) and 8 hours for critical tasks — an unsustainable load, driving the decision to build self-service tooling.

Ticket categories identified and targeted for automation: access requests, secret rotation, user onboarding/offboarding, infrastructure creation, infrastructure deletion, incident-related automation, and deployment automation.

Full user flow, as demonstrated (developer perspective):

Developer logs into DevSpace (credentials provisioned by their manager, per business domain/team).
Lands on a category selection screen: AWS, GCP, VCS (version control), Monitoring, CI/CD, and similar top-level categories, reflecting the organization’s actual internal tooling landscape.
Selecting a cloud provider (e.g., AWS) opens a sub-menu: create resources, delete resources, or request access.
Selecting “access” → a resource-type list (Lambda, S3, EC2, EKS, ECS, etc.).
Selecting a specific resource type (e.g., Lambda) opens a structured form: requester name, email, business unit (drives a dependent manager-name dropdown), the specific resource name/role/tags being requested.
Submitting the form generates a request ID and triggers a notification (email + Slack) to the selected manager — using GCP Pub/Sub as the underlying event-triggering mechanism in the real implementation.
Manager reviews and either approves or rejects the request (with a rejection reason communicated back to the requester via email).
On approval, a trigger fires to Jenkins, which holds the actual execution credentials and runs the underlying automation script.
The script performs the actual cloud action (e.g., checking whether the target IAM user already exists; creating it if not; attaching a dynamically generated policy matching the requested access scope; verifying the policy attachment succeeded).
Terraform/Terragrunt manages the resulting infrastructure state throughout — explicitly called out as the most operationally important part to get right when building this kind of platform.
A final success notification is sent to the original requester.

Simplified backend script logic demonstrated live (an Ansible-based simplified version, standing in for the real Python + Terraform implementation): set up environment → authenticate to AWS → check if the target IAM user exists → create if not → generate and attach a dynamic policy (hardcoded in the simplified demo version for clarity) → verify attachment → notify.

Explicit clarification on team ownership: the front-end UI/forms were built by front-end engineers; the entire backend — state management, script execution, notification logic — was owned by the DevOps team.

A specific, granular example given (S3 bucket-level access): the self-service form for S3 access specifically requests the exact folder/path scope being requested, so the resulting IAM policy can be scoped precisely rather than granting broad bucket-level access.

Extensibility note, framed as a genuine open assignment for participants: the instructor explicitly did not provide a complete state-management architecture for a from-scratch self-service application build, instead posing it as something for participants to think through themselves, with reference code to be shared afterward.

3.9 Third-Party Automation Platforms (Flight Control, QBR.AI)

For organizations that don’t want to build custom self-service tooling from scratch:

Flight Control: a third-party tool that can either be granted access to an AWS environment directly (via a CloudFormation-based authentication/access stack) or self-hosted within the organization’s own environment for tighter security. Operates via natural-language prompts inside a chat tool (Slack/Teams) — e.g., “deploy this repo to ECS” — with configurable underlying workflows (build image → push to artifact registry → optional DevSecOps scanning → deploy to ECS).
QBR.AI: described as a more comprehensive, end-to-end DevOps workflow automation platform covering a broad range of DevOps tasks (including observability tool installation, from simple to complex setups) without requiring deep technical configuration from the user.
Framing: both are offered as legitimate alternatives to building custom tooling — the underlying goal (reducing manual DevOps workload, progressing toward a self-healing infrastructure ideal) is the same regardless of whether an organization builds its own platform or adopts a third-party one; the choice depends on organizational risk tolerance, budget, and in-house engineering capacity.

3.10 Participant Pushback — The “We Need to See This in Action” Moment

A genuinely valuable, candid exchange, preserved because it reflects real, constructive learner feedback mid-session:

A participant directly pushed back that the session had so far provided scripts and high-level explanation, but not live, working demonstrations or the underlying system-design reasoning for why specific choices were made versus plausible alternatives — specifically requesting to see actual live deployment behavior (e.g., in ArgoCD/GitOps) and rollback-in-action, not just static script review.
The instructor’s response was direct and non-defensive: acknowledged the gap, explained that reproducing the entire real production environment live wasn’t practical, but committed to recording a dedicated screen-capture video from an actual working environment demonstrating live deployment and rollback behavior, to be shared via the drive.
A related, more specific question (about how database schema differences are handled across two simultaneously-running canary versions) was answered thoroughly and well within this same exchange — see Section 3.13.
Why this exchange is worth preserving: it models a healthy dynamic between instructor and learners — direct critical feedback given respectfully, received without defensiveness, and resulting in a concrete commitment to close the gap, rather than the feedback being deflected or the session moving on without addressing it.

3.11 CI/CD — Repository Structure & the Three-Phase Pipeline

Repository structure principle: no monorepo for the 500 microservices — each microservice has its own independent repository, containing its own Dockerfile, GitHub Actions workflow definitions, and Helm templates. Explicit justification: independent versioning, independent rollback, and parallel pipeline execution across teams (since many domain teams work simultaneously, centralizing CI/CD into one repo/pipeline would create an unacceptable bottleneck) — plus fault isolation, since a single microservice’s pipeline failure has zero impact on any other microservice’s pipeline.

Every microservice’s pipeline is structured into three distinct phases:

CI Phase (“code integrity”): linting, unit tests, SAST (static application security testing), and dependency/SCA scanning (e.g., via Snyk).
Artifact Hardening Phase: builds the production container image; tags it with both a Git SHA and semantic version — explicitly warning against using a latest tag, described as “a red flag” that has caused real production issues; pushes the tagged image to the artifact registry; signs the image using a tool called Cosign, making the artifact cryptographically verifiable/immutable.
CD Phase: deploys the signed, hardened artifact across environments and regions, using the progressive multi-region strategy detailed in Section 3.12.

3.12 CI/CD — Progressive Multi-Region Deployment & Deployment Strategies

Multi-region deployment is explicitly sequential, never parallel — described as a “progressive promotional deployment,” implemented using GitHub Actions’ matrix deployment feature, with regions ordered by business priority (e.g., AP South 1 → EU → US, matching the earlier-established regional priority).

Validation gate between each region — the same multi-dimensional health check philosophy as the patching automation (Section 3.6): error rate, SLO compliance, P95 latency, memory/CPU utilization, node health, and database connection status must all be confirmed healthy in the just-deployed region before promotion to the next region is permitted. If any of these metrics degrades, promotion halts entirely — explicitly framed as the mechanism that prevents most CI/CD-driven production outages: “we are not validating only based on some random stuff, we are actually validating based on the observability hood and the use case of those applications.”

A dedicated 2-minute traffic-stabilization wait is built in before evaluating live-traffic metrics post-deployment, to avoid false positives from transient post-deploy noise.

Deployment pattern options, all demonstrated in the shared codebase:

Rolling update — the default, described as safe for most services.
Canary deployment — for higher-risk changes: traffic is shifted progressively (10% → 25% → 50% → 100%), with the same full health-check gate applied at each step before increasing traffic share; implemented via the service mesh.
Blue-green deployment — reserved for the most business-critical services, since it requires running and paying for two full parallel environments simultaneously; explicitly framed as a cost/safety trade-off decision made per-service based on business criticality, not a default.

Quality gates: SonarCloud integrated as an automated quality gate (e.g., rejecting a deployment if a code-quality/coverage threshold like 80% isn’t met), alongside the same real-time observability-based checks (pod readiness, error rate, latency, crash-loop status) used throughout.

Rollback: a dedicated rollback script supports rolling back to a specific version, with per-region granularity — e.g., rolling back only in EU and US while explicitly excluding AP South 1 from the rollback if that region doesn’t need it.

3.13 CI/CD — Database Schema Migration Under Canary Deployment

A direct, well-answered participant question: when two application versions (e.g., v1 and v2) run simultaneously during a canary rollout, and v2 requires new/different database schema or data types, how is this handled without breaking v1?

The answer — the expand → migrate → contract pattern, explained as a strict, non-negotiable rule (“no exception on that”):

Rule stated first: database schema changes and application code changes must never be deployed in the same release — doing so blindly “can cause a lot of issues.”
Expand phase: add the new column(s)/schema elements first, as a separate, preceding change — old code (v1) safely ignores unknown/new columns, since they’re optional/unused from v1’s perspective.
Migrate phase: only after the schema expansion is live and stable, begin the canary rollout of v2 (e.g., starting at 10% traffic) — v2 writes to the new columns; v1 (still serving the remaining traffic) continues to ignore them safely. If backfilling of existing data into the new columns is required, this can be done at this stage for consistency.
Contract phase: only after v2 has been promoted to 100% of traffic and confirmed stable, the old, now-unused schema elements are removed as a final, separate cleanup step.
Explicit connection to zero-downtime migration principles: this pattern is what enables a genuinely zero-downtime schema migration under a live canary rollout, since at no point does either running version encounter a schema it can’t safely handle.

4. Key Concepts Table

Concept	Explanation	Example	Why It Matters
Classic (GUI) vs. YAML-based CI/CD pipelines	GUI pipelines are easier for beginners but don’t support modularity/reuse; YAML pipelines support shared modules and single-point-of-change updates	Adding a security scan required editing 100+ pipelines individually under a classic-pipeline setup	A concrete illustration of why pipeline architecture choices compound in cost as an organization scales
Immutable rotational patching	Building new, patched infrastructure alongside old, validating it, then retiring the old — never mutating a live resource directly	New AMI-based node group created, validated, and swapped in; old node deleted only after validation	Avoids configuration drift and makes failures at any step cleanly debuggable, unlike in-place patching
Sequential (never parallel) multi-region rollout	Regions are patched/deployed one at a time, with full health validation between each, and any failure halts the entire rollout	AP South 1 → validate → EU → validate → US	The single most important blast-radius-control mechanism used throughout this entire infrastructure
Multi-dimensional health validation (beyond pod status)	Validating infra health (API server, crash loops), application health (synthetic transaction tests), and observability health (P95 latency, error rate, SLO) together	A synthetic user actually completing a test payment as part of automated validation	Crash-loop/readiness checks alone can miss a functionally broken but “technically healthy” service
Self-service application vs. configuration automation separation	Deliberately keeping security-sensitive automation (patching, hardening) out of the general self-service platform used for routine access requests	Patching lives in its own manually-gated pipeline, not in DevSpace	A hard-won, real-world lesson about audit and operational risk from combining these two automation categories
Manual trigger gates on security-sensitive automation	Requiring explicit human-provided context (change reason, ticket ID, CVE reference) before executing automated patching, even though the execution itself is scripted	Patching pipeline requires manual trigger with required fields, never runs unattended	Automation of execution doesn’t have to mean automation of authorization — a deliberate, security-conscious design choice
Expand → Migrate → Contract (DB schema evolution)	A strict sequencing pattern for schema changes under canary/progressive deployment: add new schema first, let both versions coexist safely, remove old schema only after full rollout	Adding a column before deploying the code version that uses it, removing old columns only after 100% rollout	The standard, correct pattern for zero-downtime schema migration — prevents two simultaneously-running app versions from breaking each other
Cosign-based artifact signing	Cryptographically signing container images as part of the CI/CD artifact-hardening phase	Every production image signed before being deployable	Makes deployed artifacts verifiable and immutable — a meaningful supply-chain security practice
Progressive canary traffic shifting	Gradually increasing a new version’s traffic share (10% → 25% → 50% → 100%) with health validation gating each increase	Managed via the service mesh	Limits the blast radius of a bad release to a small fraction of real traffic before full exposure
Quantified self-service ROI case	A concrete, numbers-based justification for building self-service automation (team size, ticket volume, SLA pressure)	4-5 person team, 50-60 tickets/day, 8-72 hour SLAs	A genuinely reusable business case template for justifying this class of investment to stakeholders

5. Architecture & Workflow Analysis

5.1 Immutable Rotational Patching Flow

Security tool flags vulnerability ("bit")
        |
        v
Security team classifies severity (critical/warning/normal)
        |
        v
Change ticket created -> manager approval required
        |
        v
[AUTOMATED FROM HERE, behind a manual trigger gate]
        |
        v
FOR EACH REGION (sequential, e.g. AP -> EU -> US):
        |
   1. Notify Slack: patching window started
        |
   2. Provision NEW node group (patched AMI)
        |
   3. Drain OLD node (respecting PDB!)
        |
   4. Validate NEW node pod health
        |
   5. Delete OLD node (only after validation)
        |
   6. FULL health validation:
        - Infra: API server health, no CrashLoopBackOff
        - Application: synthetic user completes test payment
        - Observability: P95 <200ms, error rate, SLO (after 2-min stabilization)
        |
        v
   PASS? --> promote to NEXT region
   FAIL? --> HALT ENTIRE ROLLOUT, roll back immediately
        |
        v
Global completion -> audit log entry

5.2 DevSpace Self-Service Application — Full Event Flow

Developer logs into DevSpace
        |
        v
Selects category (AWS / GCP / VCS / Monitoring / CI-CD)
        |
        v
Selects action (Create / Delete / Access resources)
        |
        v
Selects resource type (e.g. Lambda) -> fills structured form
   (name, email, business unit -> manager dropdown, resource details)
        |
        v
SUBMIT -> generates Request ID
        |
        v
Event triggered (GCP Pub/Sub in real implementation)
        |
   -----------------------------
   |                           |
Email to manager          Slack notification to manager
   -----------------------------
        |
        v
Manager: APPROVE or REJECT
        |
   -----------------------------
   |                           |
REJECT                     APPROVE
   |                           |
Email to requester         Trigger fires to Jenkins
w/ reason                  (holds execution credentials)
                                |
                                v
                          Script executes:
                          - Check if IAM user exists
                          - Create if not
                          - Generate + attach dynamic policy
                          - Verify attachment
                                |
                                v
                          Terraform/Terragrunt updates state
                                |
                                v
                          Success notification to requester

5.3 Three-Phase CI/CD Pipeline (Per Microservice)

PHASE 1: CI ("code integrity")
   Linting -> Unit Tests -> SAST -> Dependency/SCA scan (e.g. Snyk)
        |
        v
PHASE 2: Artifact Hardening
   Build image -> Tag (SHA + semver, NEVER "latest")
   -> Push to artifact registry -> Sign with Cosign
        |
        v
PHASE 3: CD (progressive multi-region deployment)
   Region 1 (e.g. AP South 1):
      Deploy -> 2-min stabilization wait -> Validate
      (error rate, SLO, P95 latency, CPU/mem, node health, DB conns)
        |
        v
   PASS? --> Region 2 (EU): same validation gate
   FAIL? --> HALT, rollback
        |
        v
   PASS? --> Region 3 (US): same validation gate
   FAIL? --> HALT, rollback
        |
        v
   Deployment pattern applied per service criticality:
      Rolling update (default) | Canary (10->25->50->100%) | Blue-Green (critical only)

5.4 Expand-Migrate-Contract Schema Evolution

STEP 1: EXPAND (separate, preceding release)
   Add new column(s) to schema
   v1 (old code) safely ignores new/unknown columns
        |
        v
STEP 2: MIGRATE (canary rollout begins)
   v2 deployed at 10% traffic -> writes to new columns
   v1 (90% traffic) continues ignoring new columns safely
   [optional: backfill existing data into new columns]
        |
        v
   Canary progresses: 10% -> 25% -> 50% -> 100%
   (same health-check gates as any other canary rollout)
        |
        v
STEP 3: CONTRACT (only after 100% rollout + stability confirmed)
   Remove old/unused schema elements
        |
        v
   Zero-downtime schema migration complete

6. Commands & Configurations

Command / Config	Purpose	Explanation
GitHub Actions manual trigger with required inputs (`change_reason`, `change_id`, `cve_reference`, `patch_type`)	Gate security-sensitive patching automation behind explicit human-provided context	Ensures patching execution is scripted, but authorization remains manual and auditable
Slack API token integration	Notify a channel at the start/end of a patching window, per region	Standard operational visibility practice woven directly into the automation script
Node/AMI rotation pattern (create new → drain old, respecting PDB → validate → delete old)	The mechanical steps of immutable rotational patching	The core sequence demonstrated in the patching script; PDB check explicitly flagged as previously missing and since added
Synthetic user script (test login + test transaction)	Application-level health validation beyond infrastructure checks	Specifically built for the payment microservice; logs in as a dedicated synthetic test user and completes a real test transaction
Prometheus endpoint scrape for SLO/error-rate/P95 validation	Observability-driven health gating for both patching and CI/CD promotion decisions	Includes a deliberate 2-minute stabilization wait before evaluating live-traffic metrics
Image tagging: Git SHA + semantic version (never `latest`)	Artifact immutability and traceability	Explicitly called out as a common, real source of production issues when done incorrectly
Cosign image signing	Cryptographic signing of container artifacts during the artifact-hardening CI/CD phase	Makes deployed images verifiable and tamper-evident
GitHub Actions matrix deployment (region-ordered, sequential)	Deploy across multiple regions from a single pipeline definition, in a controlled order	Used for both patching and application CD, always sequential, never parallel
Canary traffic-shift steps (10% → 25% → 50% → 100%), managed via service mesh	Progressive exposure of a new version to real traffic	Each step gated by the same full health-check validation
Rollback script with version + per-region targeting	Roll back to a specific prior version, scoped to specific regions only if needed	E.g., roll back only EU and US, leaving AP South 1 on the current version

7. Tools & Technologies

Ansible (in the simplified demo scripts)

Purpose: Configuration/automation scripting, used in the session’s simplified educational examples.
Note: the real production implementation uses Python + Terraform; Ansible was used purely to make the underlying logic easier to follow in a teaching context.

Terraform / Terragrunt

Purpose: Infrastructure provisioning and state management underlying the self-service application’s actual resource creation/access-granting actions.
When to use it: Explicitly called out as the most operationally critical part of building a self-service platform correctly — state management discipline is what prevents the platform itself from becoming a source of infrastructure drift.

Jenkins

Purpose: Holds execution credentials and runs the actual backend automation scripts triggered by the DevSpace self-service application after manager approval.
When to use it: As the secure execution layer separating the user-facing request/approval workflow from the actual privileged cloud actions.

Cosign

Purpose: Container image signing tool, used in the artifact-hardening CI/CD phase.
When to use it: As part of a supply-chain security practice, ensuring deployed artifacts are cryptographically verifiable.

SonarCloud

Purpose: Automated code-quality and security quality-gate tool integrated into the CI phase.
When to use it: As an automated gate that can reject a deployment if code-quality/coverage thresholds aren’t met.

Flight Control (third-party automation platform)

Purpose: Natural-language, chat-driven infrastructure automation for AWS (and similar clouds).
When to use it: As a lower-effort alternative to building custom self-service tooling from scratch, for organizations without heavy in-house platform engineering capacity.

QBR.AI (third-party automation platform)

Purpose: Broader, end-to-end DevOps workflow automation platform.
When to use it: Similar positioning to Flight Control — an alternative to custom tooling for organizations wanting to reduce manual DevOps workload without building everything in-house.

GCP Pub/Sub

Purpose: Event-triggering mechanism underlying the DevSpace self-service application’s approval/notification workflow in the real implementation.
When to use it: For decoupling the request-submission event from the downstream notification/execution actions in an event-driven internal platform.

8. Real-World Production Usage

The classic-vs-YAML-pipeline war story is a genuinely common, realistic pattern across organizations at a certain growth stage — the specific, concrete consequence cited (databases mixed up, services accidentally connecting to production due to copy-paste pipeline errors) is exactly the kind of incident that motivates real organizations to invest in pipeline modularity before it’s forced on them by a crisis.
The immutable rotational patching pattern, and the specific mid-session correction (adding a PDB check based on a real past outage), models exactly how mature automation actually evolves in practice — not designed perfectly upfront, but iteratively hardened based on real incidents, with each incident’s lesson explicitly folded back into the automation.
The multi-dimensional health validation approach (infra + application + observability) reflects genuinely mature SRE practice — many teams stop at “the pod is running and passing its readiness probe,” which this session explicitly and correctly identifies as insufficient for business-critical services.
The self-service-application-vs-configuration-automation separation is a specific, non-obvious, real-world architectural lesson that isn’t commonly discussed in generic automation tutorials — it’s the kind of guidance that comes only from having actually run into the audit/operational consequences of getting this wrong.
The participant pushback moment (Section 3.10) reflects a genuinely healthy, real-world instructor-learner dynamic worth internalizing as a professional skill in its own right — giving and receiving direct, specific critical feedback constructively, and following through on a concrete commitment rather than deflecting, is itself a valuable modeled behavior beyond the technical content.
The expand-migrate-contract pattern is the industry-standard, correct approach to this exact problem, used at virtually every organization running progressive/canary deployments against a shared relational database — a genuinely important pattern for any engineer working with continuous deployment against stateful systems.

9. Interview Preparation

Beginner Questions

Q1: What’s the difference between in-place patching and immutable rotational patching? A: In-place patching modifies existing, live infrastructure directly — applying updates to a running server or node. This risks configuration drift and is difficult to debug if a step fails partway through. Immutable rotational patching instead builds entirely new, already-patched infrastructure alongside the old, validates that the new infrastructure is healthy, and only then retires the old infrastructure — the live resource itself is never directly mutated.

Q2: Why should multi-region deployments or patching operations always be sequential rather than parallel? A: Sequential rollout contains the blast radius of any bad change to a single region at a time — if something goes wrong, only one region is affected, and the rollout halts before any other region is touched. Parallel rollout means a single bad change is applied to every region simultaneously, risking a full, global outage from one mistake.

Q3: Why might checking only that a pod’s status is “Running” and passing its readiness probe be insufficient to confirm a service is actually healthy after a deployment? A: A pod can be technically running and passing basic health checks while the application itself is functionally broken in a way those checks don’t detect — for example, a payment service could report healthy pod status while actually failing to process real transactions correctly. This is why more thorough validation (like a synthetic test transaction, or checking real error rates and latency from observability data) is needed for business-critical services.

Intermediate Questions

Q4: Explain the expand-migrate-contract pattern for database schema changes, and why the order matters. A: This pattern sequences schema evolution into three distinct steps to support zero-downtime deployment when two application versions may run simultaneously (as in a canary rollout). First, expand the schema by adding new columns/elements as their own release — old application code safely ignores columns it doesn’t know about. Second, migrate — deploy the new application version (which uses the new schema elements) progressively via canary rollout, while the old version continues running safely against the same expanded schema, ignoring the new elements. Third, contract — only after the new version has been fully rolled out and confirmed stable, remove the old, now-unused schema elements. The order matters because deploying code and schema changes together, or removing old schema elements too early, risks breaking whichever version is still running against the old assumptions.

Q5: Why did this organization deliberately avoid including patching/security-hardening automation in the same self-service application used for routine access requests? A: Based on real, repeated operational and audit issues observed across multiple organizations, combining security-sensitive automation (which carries significant blast radius if misused or misconfigured) with routine, high-volume, lower-risk self-service requests (like access grants) creates governance and audit complications. Keeping configuration/patching automation in its own separately-governed, manually-triggered pipeline preserves stricter control and auditability specifically where the risk is highest, without slowing down the much higher-volume, lower-risk routine requests handled by the general self-service platform.

Q6: A self-service application allows developers to request cloud resource access, which is then approved by their manager and automatically provisioned. What are the key architectural components needed to build this safely? A: A front-end request interface capturing structured, complete request details (requester identity, business unit, specific resource and scope requested); an approval workflow that notifies the correct approver (typically derived from the requester’s business unit/team) and captures an explicit approve/reject decision; a secure execution layer (like a dedicated CI/CD tool such as Jenkins) that holds the actual privileged credentials separately from the user-facing request system, executing the actual cloud action only after approval; and critically, robust infrastructure state management (e.g., via Terraform/Terragrunt) so that resources created or modified through this platform remain consistent with infrastructure-as-code state, preventing drift between what the platform has provisioned and what’s actually tracked in code.

Advanced Questions

Q7: Design a patching automation system for a 3-region, 500-microservice Kubernetes-based infrastructure that minimizes the risk of a patching-induced production outage. A: Structure the rollout as immutable and rotational — provision new, patched node groups alongside existing ones rather than patching in place, and never mutate live nodes directly. Sequence the rollout strictly by region (never in parallel), ordered by business priority, with a full multi-dimensional health validation gate between each region: infrastructure-level checks (API server health, absence of crash-loop errors, respecting PodDisruptionBudgets during node draining), application-level checks (synthetic transaction testing for business-critical services, not just probe status), and observability-level checks (P95 latency, error rate, SLO compliance, evaluated only after a brief stabilization window to avoid transient false positives). Any validation failure in any region must halt the entire rollout immediately — no promotion to subsequent regions should be permitted after a failure. Gate the pipeline’s execution behind a manual trigger requiring explicit change-ticket/CVE/reason metadata, since patching is a security-sensitive operation warranting human authorization even when execution itself is automated. Ensure rollback is fast and well-tested, since forward progress can reasonably be slow and careful, but recovery from a failed patch must be near-instant.

Q8: A team wants to run a canary deployment where a new application version requires a database column that doesn’t exist yet, and the old version must keep running correctly during the rollout. Walk through exactly how you’d sequence this change to avoid an outage. A: First, deploy a schema-only change (the “expand” step) that adds the new column, as its own separate release — this is safe because the currently-running old application version simply ignores a column it doesn’t reference. Confirm this schema change is stable before proceeding. Second, begin the canary rollout of the new application version (the “migrate” step) — starting at a small traffic percentage (e.g., 10%), with the new version now writing to the new column while the remaining traffic continues to be served by the old version, which continues ignoring that column safely; if existing data needs to be backfilled into the new column for consistency, this can be done during this phase. Progress the canary through increasing traffic percentages (e.g., 25%, 50%, 100%), applying the same health-validation gates as any other canary rollout at each step. Only after the new version has been fully promoted to 100% of traffic and confirmed stable over an appropriate observation window should the old, now-unused schema elements actually be removed (the “contract” step) — at no point in this sequence does either running application version ever encounter a schema state it can’t handle safely.

Q9: A participant in a training session directly tells the instructor that the material has stayed too high-level and hasn’t explained the system-design reasoning behind key architectural choices. How should this kind of feedback be handled, and what does the instructor’s actual response in this session illustrate about good practice? A: Direct, specific critical feedback — especially feedback naming exactly what’s missing (live demonstration, comparative reasoning against alternative approaches) rather than vague dissatisfaction — should be treated as valuable, actionable input rather than a challenge to be deflected or minimized. In this session, the instructor’s response modeled good practice: acknowledging the gap directly without defensiveness, explaining the practical constraint that motivated the original approach (fully reproducing a production environment live isn’t always feasible), and converting the feedback into a concrete, specific commitment (recording an actual live-environment demonstration video) rather than a vague promise to “do better.” This kind of responsive, non-defensive handling of direct feedback — especially feedback given in front of a group — is itself a transferable professional skill relevant well beyond a training context, applicable to how a technical lead or mentor should respond to critical feedback from their own team.

10. Exam & Certification Notes

(Relevant to Kubernetes (CKA/CKAD) certifications for the patching/PDB content, and increasingly to DevOps/SRE-focused certifications covering CI/CD and database migration patterns.)

PodDisruptionBudget interaction with node draining: Directly reinforced again in this session — draining a node respects PDB constraints, and a patching/maintenance automation that doesn’t explicitly account for this can hang or fail exactly as described in the referenced past outage.
Blue-green vs. canary vs. rolling deployment strategies: A commonly tested deployment-strategy distinction — know that blue-green requires running two full parallel environments (highest cost, fastest full cutover/rollback), canary progressively shifts a percentage of traffic (moderate cost, gradual risk exposure), and rolling update incrementally replaces instances of a single environment (lowest cost, standard default).
Container image tagging best practices: Know that using a mutable tag like latest in production is considered a significant anti-pattern — production deployments should reference immutable, specific tags (commonly a Git SHA and/or semantic version) to guarantee reproducibility and avoid unexpected drift.
Expand-migrate-contract (also known as “parallel change”) pattern: A well-established, named pattern in database migration and continuous-delivery literature — worth knowing by name, as it may appear directly in system-design interview questions about zero-downtime deployments.
Artifact signing / supply-chain security (Cosign, Sigstore ecosystem): An increasingly relevant topic in DevSecOps-focused certification content, reflecting the industry’s growing emphasis on software supply-chain integrity.

11. Cheat Sheet

Immutable Rotational Patching — 6 Steps:

Notify (Slack)
Provision new (patched) node
Drain old node (respect PDB!)
Validate new node health
Delete old node
Full health check → promote or halt

Multi-Dimensional Health Check (use for BOTH patching and CI/CD):

Infra: API server, no CrashLoopBackOff
Application: synthetic transaction test
Observability: P95 <200ms, error rate, SLO (after 2-min stabilization)

Golden Rule: Sequential regions, NEVER parallel. Any regional failure = HALT ENTIRE ROLLOUT.

Self-Service vs. Configuration Automation: Keep patching/hardening OUT of the general self-service app — separate, manually-gated pipeline.

CI/CD — 3 Phases:

CI (lint, unit test, SAST, SCA)
Artifact Hardening (build, tag SHA+semver, sign with Cosign)
CD (progressive multi-region deploy)

Deployment Strategy Cost/Risk Trade-off:

Strategy	Cost	Risk exposure
Rolling (default)	Low	Standard
Canary (10→25→50→100%)	Moderate	Gradual, controlled
Blue-Green	High (2 full envs)	Fastest cutover/rollback

DB Schema Migration — Expand → Migrate → Contract:

Expand: add new schema (old code ignores it)
Migrate: canary rollout, new version writes to new schema
Contract: remove old schema (only after 100% + stable) Never deploy schema + code changes in the same release.

Image Tagging Rule: Git SHA + semantic version. NEVER latest in production.

12. Gaps & Assumptions

This session ends before completing the CI/CD phase — Helm-at-scale and the full observability deep-dive are explicitly deferred to a follow-up session not captured in this transcript. This document reflects only the automation and CI/CD content actually covered live.
The promised live-demo recording (committed to directly in response to participant pushback, Section 3.10) is not part of this transcript — it’s a forward commitment, not yet delivered content.
Exact script syntax for several components (the Slack integration, the dynamic IAM policy generator, the Prometheus SLO-scraping logic) was shown live on screen but described narratively in this transcript rather than dictated verbatim — this document describes the logic and sequence of these scripts accurately, but exact code syntax should be verified against the actual shared codebase (promised for upload) rather than treated as a verbatim transcription.
”DevSpace” is presented as the real name of an actual internal platform from the instructor’s own past organization (distinct from the anonymized “Titan Grid” project name) — treated here as a real, named example based on how it was presented live, though (consistent with the rest of this series) specific identifying organizational details are not independently verifiable from the transcript alone.
The manager-approval SLA figures (24 hours normal, 48-72 hours for some normal tasks, 8 hours critical) were given somewhat loosely in the live narration with a brief self-correction mid-sentence — presented here as stated, reflecting the real approximate figures rather than a precisely defined SLA table.
Third-party tool positioning (Flight Control, QBR.AI) is presented as described live by the instructor; this document does not independently verify current pricing, feature sets, or positioning for these tools beyond what was stated in the session.
This document consolidates a long session that included a real-time, unplanned pivot (the participant pushback exchange in Section 3.10) — content has been reorganized topically for clarity rather than presented in strict chronological order, consistent with the approach used for prior packages in this series, while preserving the substance and directness of that exchange since it’s a valuable part of the session in its own right.