SRE Labs (Advanced Track) — Project 4: Titan Grid — Session 2

Structured educational resource covering sre labs (advanced track) — project 4: titan grid — session 2.

senior 45 min read 16 sections

#kubernetes#cloud-k8s#aws#gcp

Automation (Phase 3) + CI/CD Release Engineering (Phase 5)

Complete Learning Package with Real Code

Sources: 2026-02-28-19-05-48.md (session transcript) + real project files from Real_Infra_Projects.zip. The session narrated over the patching scripts and CI/CD workflows already in the zip — this package integrates both.

Companion reading: Read alongside the Session 1 package (Project 4 Session 1 — Phases 1–3 + Real Code), which covers the system architecture, self-service application architecture, and the CI/CD pipeline code in depth. This session adds the conceptual reasoning behind those decisions plus the database migration pattern for canary deployments and the blue-green / rolling update / canary comparison.

Honest note on session completeness: The session ran over time and reached approximately 70% of planned content. Helm and Observability (Phases 6 and 7) were deferred again. Attendees explicitly flagged that the explanation was too high-level — the instructor acknowledged this and committed to recording a live deployment demo. Several questions were answered only at the end. These are flagged throughout.

2. Table of Contents

Executive Summary
Table of Contents
Why Automation Matters at Scale — Real-World Examples
The Automation Maturity Model
The 7 Automation Categories
Two Buckets: SSA vs. Configuration Automation
Patching Automation — Concepts
- 7.1 Types of Patching
- 7.2 Manual Patching Workflow
- 7.3 In-Place vs. Immutable (Rotational) Patching
- 7.4 The Immutable Patching Execution Model
Patching Automation — Real Code (Scripts)
- 8.1 GitHub Actions Pipeline (patching.yml)
- 8.2 Ansible Orchestrator (patch_orchestrator.yml)
- 8.3 Node Rotation (rotate_nodegroup.yml)
- 8.4 Cluster Validation (validate_cluster.yml)
- 8.5 Rollback (rollback_nodegroup.yml)
- 8.6 Integration Checklist (PDB, HPA, DB connections)
Self-Service Application (SSA) — Architecture and Workflow
- 9.1 Problem Statement
- 9.2 Automation Categories Handled by SSA
- 9.3 SSA Architecture (UI → Pub/Sub → Jenkins → Ansible/Terraform → State)
- 9.4 User Journey (from form submission to access granted)
- 9.5 Why Configuration Automation is NOT in SSA
- 9.6 State Management
- 9.7 Third-Party Tools (FlightControl, QBR.AI)
CI/CD — Phase 3 (Release Engineering)
- 10.1 Repo Structure: Decentralized Deployments
- 10.2 The 3 Pipeline Phases
- 10.3 Phase 1 — CI (Code Integrity)
- 10.4 Phase 2 — Artifact Hardening
- 10.5 Phase 3 — CD (Progressive Promotional Deployment)
- 10.6 Sequential Deployment Logic
- 10.7 Validation Gates (SLO + Synthetic + Observability)
- 10.8 Deployment Strategies — Rolling / Canary / Blue-Green
- 10.9 DB Schema Migration for Canary Deployments
- 10.10 Rollback (Manual to Specific Version)
- 10.11 The Complete pre-prod.yml Walkthrough
Architecture & Workflow Analysis
Key Concepts Table
Tools & Technologies
Interview Preparation
Exam & Certification Notes
Cheat Sheet
Gaps, Deferred Content & What’s Coming Next

3. Why Automation Matters at Scale — Real-World Examples

Example 1: Azure Classic Pipelines (Ravi)

A US fintech client had 100+ Java/Python microservices with GUI-based (“Classic”) Azure DevOps pipelines — not YAML. When a mandate came to add SonarQube scanning to all pipelines, the DevOps team had to manually edit every single pipeline because there was no shared module to update.

This is the precise cost of non-modular, non-YAML automation: any cross-cutting change requires N manual edits, where N = number of services. At 100 services, you get human errors. Databases were cross-connected to wrong environments. Non-prod services accidentally reached prod. The root cause: copy-paste pipeline creation + manual parameter editing.

Resolution: Migrated to YAML-based pipelines with shared modules. Cross-cutting concerns (SonarQube, security gates) became single-line references. Eventually moved to GitOps. The lesson: start with modular, reusable automation from day one. The “easy” GUI path becomes exponentially harder to maintain as scale grows.

Example 2: Access Management Chaos (Ravi)

Same client. Access provisioning was manual. During an internal security audit: dozens of developers and QA engineers had direct production database access. The DB contained PII and SPI (financial data). If found in an external audit, fines would have been severe. Only found internally.

Resolution: Automated access provisioning with group-membership-based policies. Permissions tied to roles, not individuals. Automated offboarding revokes on role change or termination.

Example 3: DevOps Ticket Overload (Lead SRE)

A startup with ~100 crore ARR (near-MNC scale) had 4–5 BAU DevOps engineers receiving 50–60 tickets per day. SLAs: 72 hours for normal, 8 hours for critical. Tickets were dominated by: access requests (S3 buckets, Lambda functions, RDS), secret rotation requests, onboarding/offboarding. None of these required deep technical judgment — they required time.

Solution: Self-service application (“DevSpace”) built in React for the internal team. 3,000 developers self-served access via forms. DevOps BAU load dropped dramatically. Only exception-handling tickets remained.

4. The Automation Maturity Model

Stage 1: SCRIPT-BASED AUTOMATION
  Manual trigger + shell scripts
  Reduces human effort per execution
  Still requires someone to run the script
  Most teams start here
  ↓
Stage 2: WORKFLOW AUTOMATION
  Scheduled or event-triggered pipelines (GitHub Actions, Jenkins)
  Reduces manual trigger requirement
  Adds logging, notification, retry
  Where most mature teams operate
  ↓
Stage 3: POLICY-DRIVEN AUTOMATION
  Automation governed by policy rules (branch protection, access policies, approval gates)
  Prevents non-compliant executions
  Audit trail built in
  Required for fintech / healthcare
  ↓
Stage 4: SELF-HEALING INFRASTRUCTURE (the goal)
  AI/knowledge-base-driven incident response
  Alert fires → system diagnoses → raises PR with fix → engineer reviews at 2 AM
  Not just automation of known tasks; autonomous resolution of unknown incidents
  Most organizations are working toward this

Key principle: You don’t skip stages. A team that tries to build self-healing infrastructure without stable workflow automation and policy controls will create more incidents than it solves.

5. The 7 Automation Categories

#	Category	What it automates	Priority
1	Provisioning	Creating any cloud resource (EC2, GKE cluster, S3 bucket, Lambda) via Terraform	High
2	Configuration	OS hardening, package installation, agent deployment (Datadog, Fluent Bit), CVE patching	High (keep separate from SSA)
3	Access (User Management)	IAM user creation, role attachment, RBAC bindings, offboarding, access revocation	Critical
4	Secrets	Secret rotation, propagation to Kubernetes Secrets / Vault, cross-service secret updates	Critical
5	Observability	Auto-create dashboards, alerts, SLO monitors when a new service is deployed	High
6	Incident (Self-Healing)	Scale on alert, restart unhealthy pod, open Jira ticket, notify Slack, raise fix PR	Dream stage
7	FinOps	Start/stop non-prod on schedule, right-size instances, integrate FinOps tools	Medium

Not listed but important (raised by Ravi): Branch policies and organisational CI/CD scaffolding — when a new microservice is onboarded, a pipeline runs to create the repo structure, branch protections, PR rules, and quality gate configuration automatically. This is the “shift left” scaffolding strategy.

6. Two Buckets: SSA vs. Configuration Automation

The 7 categories split into two buckets:

SELF-SERVICE APPLICATION (SSA)
  Categories: Access, Secrets, Observability, Incident, FinOps, Provisioning
  Who triggers it: Developers and managers self-service via web UI
  Approval gate: Manager approves before execution
  Backend: Pub/Sub → Jenkins → Ansible/Terraform
  State: Managed (Terraform state)
  Risk: Controlled (policy-governed, approval-gated)

CONFIGURATION AUTOMATION (Kept Separate from SSA)
  Categories: Patching (node image rotation, OS packages, K8s version upgrades)
  Who triggers it: DevOps/SRE engineers with specific CVE/change approval
  Approval gate: Security team classification + change ticket + management approval
  Backend: GitHub Actions → Ansible playbooks
  State: Manual oversight required during execution
  Risk: High (touches production nodes; wrong patch can cause cluster-wide outage)

Why the separation? Configuration automation (especially patching) interacts with security and production at execution time, not just approval time. If you embed it in an SSA, developers could self-trigger production node rotations. Additionally, audit trails for patching must satisfy security audits — self-service tooling makes this harder to demonstrate.

”I highly suggest you to not include configuration automation in SSA. As far as I have seen in the last 3–5 years, most of those who tried to include configuration automation as part of their SSA faced issues in their infrastructure and in audits.”

7. Patching Automation — Concepts

7.1 Types of Patching

Type	What it patches	Trigger
Node image rotation	AMI/node image for EKS worker nodes; replaces old AMI with hardened version containing CVE fix	CVE alert from security scanner (Snyk, Inspector, SonarCloud)
OS package update	apt/yum package updates on EC2 VMs or K8s nodes	CVE alert; scheduled quarterly
Application patching	Rebuilds Docker image with fixed base image (e.g. CVE in `node:18-alpine`) and redeploys	CVE in base image detected by Trivy
K8s version upgrade	Upgrades control plane and worker nodes to new K8s version	K8s support lifecycle; recommended for non-prod

Caution on K8s version upgrades: Automate for non-prod. For production: automate only if your automation is mature enough to handle the specific failure modes of a Kubernetes upgrade (etcd backup, control plane version skew, admission webhook compatibility).

7.2 Manual Patching Workflow

Before automation, this is how it works:

1. Security scanner (Snyk, AWS Inspector, SonarCloud) generates a finding/ticket ("bit")
2. Security team classifies severity: Critical / Warning / Informational
3. Manager/change board approves the patch
4. DevOps engineer executes patch sequentially: AP → EU → US
5. For node image rotation:
      a. Build new AMI with CVE fix
      b. Drain old node (evict pods)
      c. Bring up new node with new AMI
      d. Validate cluster health
      e. If healthy → promote to next region
      f. If not → rollback immediately
6. Store audit log in S3 bucket

At Titan Grid scale: 20+ node pools, 500 microservices, 3 regions. Manual execution = days of work per CVE. Automation reduces this to: classify → approve → input CVE ID into pipeline → monitor.

7.3 In-Place vs. Immutable (Rotational) Patching

Approach	How it works	Problems
In-place patching	SSH into running node, run `apt-get upgrade`, patch in-place	Configuration drift (node state diverges from the original image); hard to debug if step N of M fails; inconsistent state across nodes
Immutable (rotational) patching	Build a new hardened AMI → launch new node with new AMI → drain old node → delete old node	Clean state always; any failure triggers rollback to a known-good image; audit trail of which AMI each node runs

Rule: Never patch in place in a production Kubernetes environment. Always rotate nodes to a new AMI.

7.4 The Immutable Patching Execution Model

FOR EACH REGION (AP → EU → US, sequential):
  1. Build new AMI with CVE fix (build_new_node_image.sh)
  2. Update node group launch template to new AMI (update_nodegroup.sh)
  3. Drain old node: kubectl drain --ignore-daemonsets --delete-emptydir-data
  4. Wait for pods to reschedule
  5. Delete old node (delete_old_node.sh)
  6. Validate cluster health (validate_cluster.yml)
       - API server responsive (kubectl get nodes)
       - No CrashLoopBackOff
       - P95 latency < 200ms
       - Synthetic payment transaction successful
  7. IF HEALTHY → promote to next region
     IF UNHEALTHY → rollback (restore_previous_image.sh) → STOP, do not promote
8. After all regions: store audit log

Additional checks to add (instructor noted as missing from simplified scripts):

Check PDB (PodDisruptionBudget) before draining — if PDB blocks eviction, drain will hang
Verify HPA minimum replicas are respected (min 2) before draining a node
Check DB connection counts — ensure apps reconnect properly after node replacement

8. Patching Automation — Real Code

8.1 GitHub Actions Pipeline (`patching.yml`)

name: TitanGrid Production Patching

on:
  workflow_dispatch:            # MANUAL TRIGGER ONLY — critical for security-related changes
    inputs:
      cve_id:
        description: "CVE Reference (e.g. CVE-2026-12345)"
        required: true
      patch_type:
        description: "Type of patch (node | os | kubernetes)"
        required: true
        default: "node"

env:
  ANSIBLE_FORCE_COLOR: "true"

jobs:
  patch-ap:
    name: Patch AP-SOUTH
    runs-on: ubuntu-latest
    environment: production-patching   # GitHub Environment = requires manual approval gate
    steps:
      - uses: actions/checkout@v4
      - name: Install Ansible
        run: sudo apt update && sudo apt install -y ansible

      - name: Notify Slack - AP Start
        run: echo "Starting AP patch for ${{ github.event.inputs.cve_id }}"

      - name: Execute Patching - AP
        run: |
          ansible-playbook ansible360/patching/patch_orchestrator.yml \
            -e region=ap_south \
            -e patch_type=${{ github.event.inputs.patch_type }}

      - name: Validate Cluster Health - AP
        run: |
          ansible-playbook ansible360/patching/validate_cluster.yml \
            -e region=ap_south

      - name: Rollback AP if Failed          # runs ONLY if previous steps failed
        if: failure()
        run: |
          ansible-playbook ansible360/patching/rollback_nodegroup.yml \
            -e region=ap_south

  patch-eu:
    needs: patch-ap               # EU only starts if AP passed — sequential blast-radius control
    # ... identical steps with region=eu_central ...

  patch-us:
    needs: patch-eu               # US only starts if EU passed
    # ... identical steps with region=us_east ...

  notify-complete:
    needs: patch-us
    steps:
      - run: echo "TitanGrid patching for ${{ github.event.inputs.cve_id }} completed"

Why workflow_dispatch only (no automated trigger):

Patching is a security-sensitive operation — it modifies running production nodes
An automated trigger could be exploited (e.g., inject a CVE alert that triggers an attacker-controlled patch)
Required inputs (cve_id, patch_type) force the engineer to consciously confirm what they’re patching
Combined with GitHub Environment approval, this creates a two-human verification chain (security team approves change ticket, DevOps engineer confirms inputs)

Why environment: production-patching: This maps to a GitHub Environment with protection rules requiring a named reviewer to approve before the job runs. Provides a second human checkpoint within the automation itself.

8.2 Ansible Orchestrator (`patch_orchestrator.yml`)

- name: TitanGrid Multi-Region Patching
  hosts: localhost
  gather_facts: false
  vars:
    regions:
      - ap_south
      - eu_central
      - us_east

  tasks:
    - name: Notify Slack - Patching Started
      debug:
        msg: "Patching window started"
        # In production: replace with actual Slack webhook call

    - name: Loop through regions sequentially
      include_tasks: rotate_nodegroup.yml
      loop: "{{ regions }}"
      loop_control:
        loop_var: region
        # Sequential loop — eu_central only starts after ap_south completes
        # If ap_south fails: Ansible stops the loop; eu_central and us_east never run

The loop_control: loop_var: region pattern: Each iteration passes the current region name as region to rotate_nodegroup.yml. This single orchestrator file controls the entire multi-region patching sequence.

Blast radius control: The sequential loop is the entire blast radius mechanism. If the first region’s rotation fails, include_tasks stops executing. No additional logic needed — Ansible’s native error handling does the work.

8.3 Node Rotation (`rotate_nodegroup.yml`)

- name: Rotate Nodegroup for {{ region }}
  hosts: "{{ region }}"       # targets the specific region's inventory
  serial: 1                   # process ONE node at a time — never drain all simultaneously
  become: yes

  tasks:
    - name: Create new patched node image
      shell: ./build_new_node_image.sh {{ region }}
      # Builds new AMI with CVE fix applied; returns new AMI ID

    - name: Update nodegroup with new image
      shell: ./update_nodegroup.sh {{ region }}
      # Updates EKS managed node group or ASG launch template to new AMI

    - name: Drain old node
      shell: |
        kubectl drain {{ inventory_hostname }} \
          --ignore-daemonsets \
          --delete-emptydir-data
      # Evicts all non-DaemonSet pods; marks node SchedulingDisabled
      # --ignore-daemonsets: DaemonSets (monitoring, CNI) stay — they'll die with the node
      # --delete-emptydir-data: removes pods using emptyDir (ephemeral storage) — fine for stateless

    - name: Validate pods rescheduled
      shell: kubectl get pods -A
      # Confirm pods have moved to other nodes before deleting the old one

    - name: Delete old node
      shell: ./delete_old_node.sh {{ inventory_hostname }}
      # Terminates the EC2 instance running the old AMI

serial: 1 — the most important setting: Without this, Ansible would try to drain and delete all nodes in the host group simultaneously. serial: 1 forces one-at-a-time execution — the cluster never has all nodes drained at once.

What build_new_node_image.sh does: Creates a new AMI based on the current hardened base image with the specific CVE patch applied. Uses AWS EC2 Image Builder or a Packer workflow to produce an immutable, versioned AMI. The AMI ID is captured as output.

What update_nodegroup.sh does: Updates the EKS managed node group’s launch template to the new AMI ID. The actual node replacement happens via the drain + delete steps — NOT via AWS node group rolling update, because the Ansible scripts give more granular control over validation between nodes.

8.4 Cluster Validation (`validate_cluster.yml`)

- name: Validate Cluster Health
  hosts: localhost
  gather_facts: false

  tasks:
    - name: Check API server health
      shell: kubectl get nodes
      # If kubectl fails to reach the API server, the control plane is unhealthy

    - name: Check pod crashloops
      shell: kubectl get pods -A | grep CrashLoopBackOff
      # Any CrashLoopBackOff after node rotation = application compatibility issue with new AMI

    - name: Check latency spike via metrics API
      shell: ./check_latency.sh
      register: latency_result

    - name: Fail if latency exceeds threshold
      fail:
        msg: "Health check failed — latency {{ latency_result.stdout }}ms exceeds 200ms threshold"
      when: latency_result.stdout > 200
      # If this task fails → GitHub Actions `if: failure()` triggers rollback

Three levels of validation (the instructor emphasised this explicitly):

Infrastructure level: kubectl get nodes — is the control plane reachable?
Kubernetes level: kubectl get pods -A | grep CrashLoopBackOff — are workloads healthy?
Application level: ./check_latency.sh + Prometheus SLO query — is the application serving traffic correctly?

Why 3 levels matter: Pods can be in Running state while the application inside them is broken (startup error, bad config, dependency issue). A new AMI might have a kernel version that breaks a specific system call used by one microservice. This would show as a latency spike or error rate increase, not as a CrashLoopBackOff. Without the application-level check, you would promote a broken patch to the next region.

Additional checks to add (instructor noted during session):

# Before draining: check PDB won't block
- name: Check PodDisruptionBudget allows drain
  shell: |
    kubectl get pdb -A -o json | jq \
      '.items[] | select(.status.disruptionsAllowed == 0) | .metadata.name'
  register: blocking_pdbs
  failed_when: blocking_pdbs.stdout != ""

# Verify HPA min replicas
- name: Verify minimum replicas available
  shell: |
    kubectl get hpa -n services -o json | jq \
      '.items[] | select(.status.currentReplicas < 2) | .metadata.name'

# Check DB connection counts
- name: Validate DB connections post-rotation
  shell: ./check_db_connections.sh

8.5 Rollback (`rollback_nodegroup.yml`)

- name: Rollback Nodegroup
  hosts: "{{ region }}"
  become: yes

  tasks:
    - name: Restore previous node image
      shell: ./restore_previous_image.sh {{ region }}
      # Reverts the EKS node group launch template to the previous AMI ID
      # The previous AMI ID must be stored before rotation begins

    - name: Validate restoration
      shell: kubectl get nodes

What restore_previous_image.sh must do:

Retrieve the previous AMI ID (stored as a variable/file before rotation began)
Update the node group launch template to the previous AMI
Drain new (broken) nodes
Launch replacement nodes with the old AMI
Delete new (broken) nodes

Critical principle: Rollback must be faster than forward patching. Forward patching validates thoroughly (5–10 minutes per node). Rollback must execute in under 2 minutes. This requires the previous AMI ID to be stored and immediately accessible — not looked up dynamically.

9. Self-Service Application (SSA) — Architecture and Workflow

9.1 Problem Statement

At Titan Grid’s scale (~3,000 developers across 150+ Business Units):

4–5 BAU DevOps engineers receiving 50–60 tickets/day
SLAs: 72 hours (normal) / 8 hours (critical)
Ticket categories: access requests (80%), secret rotation, onboarding/offboarding, infra provisioning, infra deletion, incident support, deployment help
Problem: Access and secrets tickets require no deep technical judgment — they require time and are consuming the majority of DevOps bandwidth

9.2 Automation Categories Handled by SSA

Ticket type	SSA capability
AWS resource access (Lambda, S3, EC2, EKS, RDS, ECS, SQS)	Form-based request → Ansible playbook creates IAM user + attaches least-privilege policy
GCP resource access (GKE, GCS, Cloud Run, Cloud SQL, Pub/Sub, Cloud Functions, Compute Engine)	Form-based request → gcloud IAM policy binding
Secrets rotation	Form triggers secret rotation playbook → propagates to Kubernetes Secrets
Onboarding	Form triggers: IAM user creation + RBAC bindings + tool access (Jenkins, Datadog, Bitbucket)
Offboarding	Form triggers: revoke all resource access + disable IAM user + remove from groups
Infra provisioning	Form triggers Terraform module to create the requested resource
Infra deletion	Form (with additional approval gate) triggers Terraform destroy
CI/CD pipeline creation	Form creates Helm folder structure, pipeline YAML template, pushes to repo

9.3 SSA Architecture

Developer submits form (React UI, "DevSpace")
  │
  ▼ Validation
  Form validates inputs (lambda name, role, email, BU, manager)
  │
  ▼ Pub/Sub trigger (GCP)
  Event published to internal topic with request payload
  Request ID generated + stored
  │
  ▼ Manager notification
  Email + Slack DM to selected manager: "Approve / Reject"
  Rejection: developer notified with reason + request closed
  Approval: continue ↓
  │
  ▼ Jenkins trigger
  Approval fires Jenkins job with request payload
  Jenkins holds credentials (AWS/GCP STS tokens or service account JSON)
  │
  ▼ Ansible playbook execution
  e.g. eks_access.yml:
    authenticate → check user exists → create if not → attach policy
  │
  ▼ Terraform state management
  Changes recorded in Terraform state (all resource modifications are tracked)
  │
  ▼ Notification
  Developer + manager notified: "Access granted" or "Action failed + reason"

Why Jenkins (not Lambda) for execution:

Jenkins already holds all infrastructure credentials (no need to distribute them to another service)
Jenkins provides built-in build history and log retention for audit
The infra team already manages Jenkins — no additional tool to operate

Why Pub/Sub (not direct webhook):

Decouples form submission from manager response timeline
If the manager responds 2 hours later, the event is still in the topic
Provides at-least-once delivery guarantee

9.4 User Journey (Example: Lambda Access)

Developer logs in to DevSpace with BU credentials (created by their manager)
Selects: AWS → Access → Lambda
Fills form: Name, Email, Business Unit (dropdown), Manager (auto-populates from BU), Lambda name, Lambda ARN, Access type (read/invoke), Request reason
Clicks “Send”
Immediate: Request ID generated; manager receives Slack notification + email with Approve/Reject buttons
On Rejection: Developer gets email with reason; ticket closed
On Approval:
- Pub/Sub event triggers Jenkins job access-lambda-{request-id}
- Jenkins runs lambda_access.yml Ansible playbook with form values as variables
- Playbook: authenticate STS → check IAM user exists → create if not → aws iam put-user-policy with least-privilege JSON (read-only: lambda:GetFunction, lambda:InvokeFunction)
- Terraform records the IAM change in state
Completion: Developer + manager get Slack notification: “Lambda access for arn:aws:lambda:... granted to user@company.com”

9.5 Why Configuration Automation is NOT in SSA

Three reasons:

Security oversight: Patching production nodes should require the DevOps engineer to be present during execution — not just approve a form and walk away. The engineer needs to watch the drain, watch the validation, be ready to rollback manually if automation fails.
Audit trail complexity: Regulators expect to see that a qualified engineer oversaw security-related infrastructure changes. SSA-triggered changes can look like “developer triggered a patch.”
Blast radius risk: A misconfigured patch triggered via SSA by a developer (or an attacker who compromised a developer account) could drain and replace all production nodes. Configuration automation’s blast radius is too high for the SSA trust model.

9.6 State Management

Every change made through SSA must be reflected in Terraform state. This is the instructor’s explicit assignment:

“This is a bit of an assignment for you — create a document on how you would manage state if you were creating a self-service application. How exactly will you manage the state out there.”

The challenge: If the SSA runs Ansible to create an IAM user, that user is now in AWS but NOT in Terraform state. If someone later runs terraform plan, Terraform will see an unmanaged resource and may plan to destroy or leave it untracked.

Solutions:

Option A: Terraform-only SSA backend. All SSA operations run Terraform modules (not Ansible directly). Every resource change generates a Terraform plan → apply, updating state.
Option B: State import. Ansible creates the resource; a post-step runs terraform import resource_type resource_id to bring it into state.
Option C: Terraform Cloud / Atlantis. All SSA-triggered changes go through a Terraform Cloud run, giving full state management, plan approval, and run history.

The instructor mentioned that the actual implementation used “Python + Terraform” (not Ansible alone) for the production SSA — the Ansible scripts in the zip are simplified demonstrations.

9.7 Third-Party Tools (No-Code/Low-Code Alternatives)

For teams that cannot build a custom SSA:

FlightControl (flightcontrol.dev):

AWS-native infrastructure management platform
Integrates with your Slack/Teams; accepts plain English prompts
”Deploy this repo to ECS” → creates workflow (build → push → ECS deploy)
Self-hostable (for data residency compliance)
Generates CloudFormation for access scoping

QBR.AI:

End-to-end DevOps workflow platform
Pre-built workflows for common DevOps tasks (install observability tools, create K8s resources, etc.)
Multi-cloud; HIPAA and compliance-friendly options

10. CI/CD — Phase 5 (Release Engineering)

10.1 Repo Structure: Decentralized Deployments

One repo per microservice (NOT a monorepo):
  payments-service/
  ├── .docker/Dockerfile
  ├── .github/workflows/
  │   ├── pre-prod.yml        ← full CI + artifact hardening + sequential prod deploy
  │   ├── Development.yml     ← canary deploy: AP → EU → US
  │   ├── Staging.yml         ← single-region staging deploy
  │   ├── Production.yml      ← matrix prod deploy with IgnoreRegion
  │   ├── rollback.yml        ← rollback to specific version, any region
  │   ├── quality.yml         ← SonarQube PR gate
  │   ├── reuse.yml           ← workflow_call: reusable build
  │   ├── dev_gw.yml          ← gateway-only deploy (dev)
  │   ├── stage_gw.yml        ← gateway-only deploy (staging)
  │   └── prod_gw.yml         ← gateway-only deploy (prod)
  ├── .k8s/
  │   ├── prod/{server.yml, consumer.yml, gateway.yml, secrets.yml}
  │   ├── stage/{...}
  │   └── dev/{...}
  └── payments/               ← Helm chart
      ├── Chart.yaml
      ├── templates/
      └── {qa,stage}/values.yaml

Why not monorepo at 500 services:

Independent versioning per service (v1.2.3 for payment, v2.0.1 for fraud — no coupling)
Independent rollback per service (rollback payment without touching fraud)
Parallel CI/CD (500 services can all run pipelines simultaneously — no central queue)
Reduced blast radius (a broken pipeline in one service doesn’t block others)

10.2 The 3 Pipeline Phases

Phase 1: CI (Code Integrity)
  └── Linting + Unit Tests + SAST (CodeQL) + Dependency Scan (Snyk)
  Purpose: Verify code quality and security BEFORE building an artifact
  Gate: Any failure = build blocked; PR cannot merge

Phase 2: Artifact Hardening
  └── Docker build + semantic version tag + SHA tag + push to Artifact Registry
  └── Container scan (Trivy) — CRITICAL/HIGH CVEs = pipeline fails
  └── Image signing (Cosign) — immutable, verifiable artifact
  Purpose: Produce a verified, immutable, versioned artifact
  Gate: Trivy CRITICAL/HIGH = pipeline fails; latest tag = forbidden

Phase 3: CD (Progressive Promotional Deployment)
  └── Deploy to AP → validate → deploy to EU → validate → deploy to US → validate
  └── Validation: SLO gates (Prometheus) + synthetic transaction + observability checks
  Purpose: Deploy verified artifact to production with live traffic validation
  Gate: SLO failure or synthetic test failure = rollback + stop promotion

10.3 Phase 1 — CI (Code Integrity)

CI:
  steps:
    - run: npm ci              # reproducible dependency install (lockfile-based)
    - run: npm test            # unit tests
    - run: npm run lint        # code style enforcement
    - uses: github/codeql-action/init@v3
      with: { languages: javascript }    # SAST scan
    - run: npm run build       # verify build compiles successfully

Why npm ci over npm install: npm ci uses the lockfile exactly — no version drift between developer machines and CI. npm install can resolve to newer patch versions, introducing undiscovered breakage.

SonarQube quality gate (quality.yml): Runs on every PR open/sync/reopen. Uses fetch-depth: 0 for full git history (required for accurate diff-scope analysis). Blocks merge until gate passes (configurable threshold, e.g. 80% coverage, 0 critical issues).

10.4 Phase 2 — Artifact Hardening

Build:
  needs: CI
  steps:
    - uses: docker/setup-buildx-action@v3
    - uses: docker/build-push-action@v5
      with:
        file: .docker/Dockerfile
        push: true
        tags: |
          ${{ env.REGISTRY }}/.../seller:${{ env.VERSION }}    # semantic version
          ${{ env.REGISTRY }}/.../seller:${{ github.sha }}     # git SHA
        cache-from: type=gha
        cache-to: type=gha,mode=max

    - uses: aquasecurity/trivy-action@master
      with:
        severity: CRITICAL,HIGH
        exit-code: 1     # pipeline fails on CRITICAL or HIGH CVE

Tagging rules:

latest tag is forbidden — never use latest in production. If the registry always serves the “latest” image and a broken version is pushed, every pod restart will pull the broken image. Semantic version pinning is mandatory.
Dual tagging (version + SHA): The version tag is used for deployments. The SHA tag is used for traceability — given any running pod’s image SHA, you can find the exact commit that produced it.
Cosign image signing (mentioned in session, referenced in scripts): Creates a cryptographic signature stored in the registry alongside the image. Admission webhooks can enforce that only signed images are deployed. Prevents supply-chain attacks (you cannot run an image that wasn’t built by your CI pipeline).

Docker build caching (type=gha): Uses GitHub Actions cache to store Docker layer cache. For a typical Node.js service, this reduces build time from 3–4 minutes to 30–60 seconds on repeated builds where dependencies haven’t changed.

10.5 Phase 3 — CD (Progressive Promotional Deployment)

From pre-prod.yml (the full production workflow):

Deploy-AP:
  needs: Build
  environment: Production-ap-south-1
  steps:
    - uses: google-github-actions/get-gke-credentials@v2
      with:
        cluster_name: ${{ fromJson(secrets.CLUSTER_MAP)['ap-south-1'] }}
        location:     ${{ fromJson(secrets.REGION_MAP)['ap-south-1'] }}

    - name: Deploy to AP
      run: |
        kubectl set image deployment/marketplace-seller \
          marketplace-seller=${{ env.REGISTRY }}/.../seller:${{ env.VERSION }}
        kubectl rollout status deployment/marketplace-seller
        # Waits until all pods have updated and are Ready

    - name: SLO Validation (AP)
      run: ./scripts/validate_slo.sh ap-south-1

Deploy-EU:
  needs: Deploy-AP      # starts ONLY if Deploy-AP succeeded
  # identical steps...

Deploy-US:
  needs: Deploy-EU      # starts ONLY if Deploy-EU succeeded
  steps:
    # ... deploy ...
    - name: SLO Validation + Synthetic Test (US)
      run: |
        if ! ./scripts/validate_slo.sh us-east-1; then
          ./scripts/rollback.sh us-east-1
          exit 1
        fi
        ./scripts/synthetic_test.sh us-east-1

Finalize:
  needs: Deploy-US
  steps:
    - uses: rickstaa/action-create-tag@v1
      with: { tag: "v${{ env.VERSION }}-prod" }
    - uses: rtCamp/action-slack-notify@v2
      env:
        SLACK_MESSAGE: "Production Release ${{ env.VERSION }} Successful"

10.6 Sequential Deployment Logic

SEQUENTIAL (correct for production):
  AP completes → SLO validation → if PASS: EU starts
                                  if FAIL: stop + rollback AP; EU and US never touched

PARALLEL (wrong for production):
  AP + EU + US all start simultaneously
  If a bug exists: all 3 regions are broken
  Blast radius = global

WHY SEQUENTIAL WINS: If a bug survives the CI phase and synthetic tests, the sequential
  deployment allows AP to absorb the impact while EU and US remain on the known-good version.
  The cost: deployment takes 3× as long. The benefit: 2 of 3 regions are always safe.

EXCEPTION — The Production.yml matrix approach:
  When you need to deploy to all 3 regions simultaneously (e.g., critical security patch,
  time-sensitive feature), use the matrix deploy with IgnoreRegion input to skip any
  region that's already been manually patched. Parallel is acceptable when you have
  extremely high confidence in the change (e.g., config-only change, not code change).

10.7 Validation Gates (SLO + Synthetic + Observability)

Three layers of validation after every deployment:

# validate_slo.sh ap-south-1
sleep 120    # wait 2 minutes for traffic to stabilise before measuring

# Layer 1: Prometheus error rate
ERROR_RATE=$(curl -sG \
  --data-urlencode "query=sum(rate(http_requests_total{service=\"marketplace-seller\",status=~\"5..\"}[5m])) \
    / sum(rate(http_requests_total{service=\"marketplace-seller\"}[5m]))" \
  $PROMETHEUS_URL/api/v1/query | jq -r '.data.result[0].value[1]')
[ "$ERROR_RATE" > "0.02" ] && exit 1    # fail if error rate > 2%

# Layer 2: P95 latency
P95=$(curl -sG \
  --data-urlencode "query=histogram_quantile(0.95, ...) * 1000" \
  $PROMETHEUS_URL/api/v1/query | jq -r '...')
[ "$P95" > "200" ] && exit 1    # fail if P95 > 200ms (fintech SLA)

# Layer 3: kubectl top (CPU/memory)
# Layer 4: CrashLoopBackOff check (restarts > 3 = fail)

# synthetic_test.sh ap-south-1
TOKEN=$(curl -s -X POST $BASE_URL/login -d '{"username":"synthetic_user","password":"test123"}' ...)
RESULT=$(curl -s -X POST $BASE_URL/payment -H "Authorization: Bearer $TOKEN" -d '{"amount":1}')
STATUS=$(echo $RESULT | jq -r '.status')
[ "$STATUS" != "SUCCESS" ] && exit 1

The philosophy: Don’t declare a deployment successful because pods are Running. Declare it successful when:

Pod is Running (infrastructure layer) ✓
P95 latency is within SLA (application performance) ✓
Error rate is within threshold (application correctness) ✓
A real payment transaction completes successfully (end-to-end validation) ✓

This is CI/CD connected to observability — the pipeline doesn’t trust Kubernetes health checks alone.

10.8 Deployment Strategies — Rolling / Canary / Blue-Green

Three strategies are included in the repo. Choice depends on the service’s criticality:

Strategy	How it works	When to use	Cost
Rolling Update	Replace pods one by one (maxUnavailable: 0, maxSurge: 100%)	Default for most services; safe for backward-compatible changes	No extra cost
Canary	Route 10% traffic to new version → validate → 25% → 50% → 100% (via Istio VirtualService weight)	Major changes; critical payment paths; new features with risk	No extra cost (same infra)
Blue-Green	Two full environments; switch traffic all-at-once via DNS/LB; old environment stays live for quick rollback	Most critical services where rollback must be instant; DB schema changes where both versions must run simultaneously	2× infrastructure cost

In TitanGrid, used as:

Rolling: default for most of 500 services
Canary: payment-initiate, fraud-detector (any service in the critical payment path)
Blue-Green: only for services with complex DB migration requirements or when business mandates instant rollback capability

Development.yml shows canary with Istio:

- name: Deploy Canary
  run: |
    kubectl set image deployment/marketplace-seller marketplace-seller=${{ env.IMAGE }}
    kubectl apply -f istio/canary-10.yaml    # Istio VirtualService: 10% new / 90% old
    sleep 180
- name: SLO Validation at 10%
  run: ./scripts/validate_slo.sh ap-south-1
- name: Promote to 100%
  run: kubectl apply -f istio/canary-100.yaml

10.9 DB Schema Migration for Canary Deployments

The question (from Suresh): When two versions of an application run simultaneously during canary (v1 and v2), and v2 needs a new DB column that v1 doesn’t know about — how do you handle this without breaking v1?

The answer — Expand → Migrate → Contract (Zero-Downtime Migration):

STEP 1: EXPAND (before deploying v2)
  Add the new column to the schema as nullable with no default:
    ALTER TABLE payments ADD COLUMN new_field VARCHAR(255) NULL;
  Deploy this schema change independently (not bundled with code change).
  Both v1 (running) and v2 (about to deploy) can now use this schema:
    - v1 ignores the new column (old code ignores unknown columns)
    - v2 writes to the new column

STEP 2: MIGRATE (during canary: v1 + v2 running simultaneously)
  v2 writes to both old_field and new_field
  v1 writes to old_field only (safe — new_field is nullable)
  Background job backfills new_field for any rows created by v1
  Traffic gradually shifts from v1 to v2 (10% → 25% → 50% → 100%)

STEP 3: CONTRACT (after 100% rollout + stability confirmed)
  v1 is fully removed
  Remove old_field from application code (v2 no longer reads it)
  ALTER TABLE payments DROP COLUMN old_field;
  This cleanup can happen days or weeks after the 100% rollout

Rule: Never deploy schema changes and code changes in the same release. Schema first (backward-compatible only), then code.

Rule: All schema changes must be backward-compatible with the previous version. If removing a NOT NULL constraint is needed, make it nullable first (in a previous release), then in a later release remove it entirely.

10.10 Rollback (Manual to Specific Version)

rollback.yml — matrix across all 3 regions, parallel:

on:
  workflow_dispatch:
    inputs:
      Version:       { required: true }     # version to roll back TO
      Reason:        { required: true }     # documented reason (audit trail)
      IgnoreRegion:  { type: choice, options: [None, ap-south-1, eu-central-1, us-east-1] }

jobs:
  rollback:
    strategy:
      matrix: { REGION: [ap-south-1, eu-central-1, us-east-1] }
    if: always()    # rollback all regions even if one fails

    steps:
      - name: Skip if region ignored
        if: github.event.inputs.IgnoreRegion == matrix.REGION
        run: exit 0

      - name: Generate Manifests with Old Image
        env:
          IMAGE: ${{ env.REGISTRY }}/.../seller:${{ env.VERSION }}    # ← OLD version tag
        run: envsubst < ./.k8s/prod/server.yml > server.yml

      - name: Apply Rollback Manifests
        run: |
          kubectl apply -f ./server.yml
          kubectl rollout status deploy/marketplace-seller

Key design decisions:

Parallel (not sequential) for rollback — you want all 3 regions back to the known-good version as fast as possible
if: always() — ensures the other regions roll back even if one region’s rollback fails
IgnoreRegion — if one region has already been manually fixed (e.g., via emergency hotfix), skip it in the rollback
Reason field — forces the engineer to document why they rolled back; stored in GitHub Actions run history and Slack notification

10.11 `rollback.sh` (Bash Script)

#!/bin/bash
# Simple rollback script (referenced in Development.yml)
VERSION=$1
REGION=$2

kubectl set image deployment/marketplace-seller \
  marketplace-seller=$REGISTRY/seller:$VERSION
kubectl rollout status deployment/marketplace-seller --timeout=120s

echo "Rollback to $VERSION complete in $REGION"

11. Architecture & Workflow Analysis

End-to-End Patching Flow

CVE discovered (Snyk/Inspector/SonarCloud)
  ↓
Security team classifies: Critical / Warning
  ↓
Change request ticket created (ServiceNow / Jira)
  ↓
Management approval
  ↓
DevOps engineer triggers `patching.yml`:
  cve_id = CVE-2026-XXXX
  patch_type = node
  ↓
GitHub Environment approval (production-patching)
  ↓
SEQUENTIAL:
  AP-South-1:
    patch_orchestrator.yml → rotate_nodegroup.yml (serial: 1 per node)
      → build_new_node_image.sh (new AMI)
      → update_nodegroup.sh (update launch template)
      → kubectl drain (evict pods)
      → delete_old_node.sh
      → validate_cluster.yml (API server + CrashLoops + latency + synthetic test)
    IF HEALTHY → EU-Central-1 starts
    IF UNHEALTHY → rollback_nodegroup.yml (restore_previous_image.sh) → STOP
  EU-Central-1: same
  US-East-1: same
  ↓
Notify Slack: "Patching CVE-2026-XXXX complete in all regions"
  ↓
Audit log stored in S3 bucket

CI/CD Progressive Deployment Flow

Developer merges PR (after quality.yml SonarQube gate)
  ↓
Trigger pre-prod.yml (workflow_dispatch: Version=1.2.3)
  ↓
Phase 1 — CI: npm ci + test + lint + CodeQL
  ↓
Phase 2 — Artifact Hardening:
  docker build → tag (1.2.3 + SHA) → push to Artifact Registry → Trivy scan
  [CRITICAL/HIGH CVE found? → pipeline fails here]
  ↓
Phase 3 — CD, Sequential:
  Deploy-AP:
    kubectl set image → rollout status → validate_slo.sh (Prometheus + kubectl)
    [SLO fail? → rollback AP → pipeline stops]
  Deploy-EU: (needs: Deploy-AP)
    [SLO fail? → rollback EU → pipeline stops; AP still on new version]
  Deploy-US: (needs: Deploy-EU)
    validate_slo.sh + synthetic_test.sh
    [SLO fail? → rollback US → pipeline stops]
  Finalize: (needs: Deploy-US)
    tag v1.2.3-prod → Slack: "Production Release 1.2.3 Successful"

12. Key Concepts Table

Concept	Explanation	Real implementation	Why it matters
Automation Maturity Model	4 stages: Script → Workflow → Policy-driven → Self-healing	Titan Grid operates at stage 3–4	Defines what to build next; avoids skipping foundational stages
SSA (Self-Service Application)	Internal React portal for developers to self-serve infra requests	”DevSpace” — form → Pub/Sub → Jenkins → Ansible → Terraform	Removes DevOps BAU ticket bottleneck
Configuration automation (separate from SSA)	Patching and security configs handled outside SSA	`patching.yml` requires DevOps engineer presence	Security-sensitive; audit trail; blast radius too high for self-service
Immutable patching	Never patch in-place; always build new AMI → rotate → delete old	`rotate_nodegroup.yml` with `serial: 1`	Prevents drift; clean rollback path; consistent state
`serial: 1` in Ansible	Process one node at a time	In `rotate_nodegroup.yml`	Prevents draining all nodes simultaneously
`if: failure()` in GitHub Actions	Step only runs if previous steps failed	Rollback step in `patching.yml`	Automatic recovery without human intervention
3 pipeline phases	CI (integrity) → Artifact Hardening → CD (deploy)	`pre-prod.yml` structure	Separates concerns: code quality, artifact quality, deployment quality
`latest` tag is forbidden	Always use semantic version + SHA tags	Dual tag in `Build` job	Prevents accidental rollout of unstable images
Image signing (Cosign)	Cryptographic signature on built images	Referenced in session; integrated in artifact hardening	Supply-chain security: only pipeline-built images can run
SLO-gated deployments	Deployment success = Prometheus SLO pass + synthetic test pass	`validate_slo.sh` + `synthetic_test.sh`	Catches production issues that pass Kubernetes health checks
Sequential deploys (production)	AP → EU → US with validation between each	`needs: Deploy-AP` chain	Limits blast radius; most releases only affect AP if broken
Parallel rollback	All 3 regions rolled back simultaneously	`strategy.matrix` in `rollback.yml` with `if: always()`	Recovery speed matters more than blast-radius control during rollback
`IgnoreRegion` input	Skip one region in rollback/deploy matrix	`if: github.event.inputs.IgnoreRegion == matrix.REGION`	Handles partial failures gracefully
DB migration: Expand-Migrate-Contract	Schema changes separate from code changes; backward-compatible only	Pattern applied to all DB-schema-changing deployments	Enables zero-downtime migration; canary-safe
PDB check before drain	Ensure PodDisruptionBudget allows eviction before draining	Missing from simplified scripts; noted as must-add	PDB blocking drain causes indefinite hang (the K8s war-room scenario)

13. Tools & Technologies

Tool	Used for	Session notes
GitHub Actions	Patching pipeline + CI/CD pipelines	`workflow_dispatch` only for patching; `concurrency: cancel-in-progress: false` for deploys
Ansible	Patching automation + access automation	`serial: 1` for node-by-node patching; idempotent playbooks
Pub/Sub (GCP)	SSA event bus (form submission → manager notification → Jenkins trigger)	Decouples UI from backend execution; retains events
Jenkins	SSA backend execution (holds infra credentials; runs Ansible)	Central credential store; audit log; reusable shared libraries
Terraform	State management for SSA-created resources	Instructor: actual SSA used Python + Terraform (not Ansible alone)
CodeQL	SAST scan in CI phase	`github/codeql-action/init@v3`; JavaScript language
Trivy	Container vulnerability scan in Artifact Hardening phase	`aquasecurity/trivy-action`; `severity: CRITICAL,HIGH`; `exit-code: 1`
Cosign	Image signing	Part of artifact hardening; referenced in session
SonarQube/SonarCloud	PR quality gate (runs on every PR)	`sonarsource/sonarqube-scan-action@v2`; `fetch-depth: 0`
Snyk	Dependency scan (SCA)	Integrated in CI phase alongside CodeQL
Prometheus	SLO validation post-deploy	Queried directly in `validate_slo.sh` via HTTP
Istio VirtualService	Canary traffic splitting	`canary-10.yaml` (10%) → validate → `canary-100.yaml`
FlightControl	Third-party infra automation (AWS-native, Slack/Teams integrated)	Alternative to building custom SSA
QBR.AI	End-to-end DevOps workflow platform	Alternative to custom automation for no-code teams

14. Interview Preparation

Q1. Why is configuration automation kept separate from a self-service application? Configuration automation (patching, OS hardening) touches running production infrastructure with the potential to take down an entire node or region if it goes wrong. The blast radius is too high for self-service execution. Additionally, security-related changes require a qualified engineer present during execution — not just at approval time. In regulated industries (fintech, healthcare), auditors expect documented human oversight of security configuration changes. SSA’s self-service model would make it harder to demonstrate this oversight.

Q2. Explain the difference between in-place patching and immutable (rotational) patching. Which should you use and why? In-place patching runs updates directly on a running server (apt-get upgrade, yum update). Problems: if step N fails out of M steps, the node is in an unknown intermediate state; over time nodes drift from the base configuration; debugging a failed patch is hard because you don’t know the exact starting state. Immutable patching builds a new server image (AMI) with the patch applied, then replaces old nodes with new nodes running the new image, then deletes old nodes. Any failure means you still have the known-good old node and can simply rollback to it. Production Kubernetes environments should always use immutable patching.

Q3. How do you handle DB schema migrations during a canary deployment where two versions of the application run simultaneously? Use the Expand → Migrate → Contract pattern. First, deploy a schema change separately (before the code change) that adds the new column as nullable — this is backward-compatible; v1 ignores it. Then deploy v2 (canary), which reads and writes both old and new columns. A backfill job populates the new column for existing rows. After 100% rollout and stability, deploy a cleanup that drops the old column. The key rules: schema changes always deploy separately from code changes; every schema change must be backward-compatible with the previous version.

Q4. The 3 phases of a production CI/CD pipeline — what are they and what does each gate check? Phase 1 (CI/Code Integrity): runs on every PR — linting, unit tests, SAST (CodeQL), dependency scanning (Snyk). Gate: any failure blocks PR merge. Phase 2 (Artifact Hardening): builds the Docker image, tags with semantic version + git SHA, pushes to registry, runs Trivy container scan (CRITICAL/HIGH = fail), signs with Cosign. Gate: any CRITICAL/HIGH CVE blocks the release. Phase 3 (CD/Progressive Deployment): deploys sequentially AP → EU → US; after each region, validates Prometheus SLO (error rate, P95 latency) + synthetic transaction test. Gate: SLO failure = automatic rollback of that region + stop; next region never gets the broken version.

Q5. Why is rollback done in parallel across all regions while forward deployment is sequential? Forward deployment is sequential to limit blast radius: if a broken change reaches AP and fails, EU and US remain on the known-good version. Rollback is parallel because, by definition, you already know the change is bad and want to return all regions to the known-good state as quickly as possible. Speed is the primary objective during rollback; blast radius is no longer a concern because the damage is already known. The if: always() flag on the rollback matrix ensures all 3 regions roll back even if one fails.

15. Exam & Certification Notes

GitHub Actions:

if: failure() — step condition: only run if previous steps in the same job failed
if: always() — step/job condition: always run regardless of previous status
workflow_dispatch — manual trigger with user-provided inputs
environment: — maps to GitHub Environment; can require approval reviewers
concurrency: cancel-in-progress: false — do not cancel a running production deploy

Security testing types (frequently examined as a set of 3):

SAST (Static Application Security Testing) — scans code without running it; catches code-level vulnerabilities; runs before build
DAST (Dynamic Application Security Testing) — scans running application; catches runtime vulnerabilities (injections, auth bypasses)
SCA (Software Composition Analysis) — scans dependencies for known CVEs; catches log4shell-class vulnerabilities
Container scanning — Trivy; scans the built Docker image; separate from SCA

Deployment strategies — exam-level distinctions:

Rolling update: default; replaces pods one-by-one; zero-downtime with maxUnavailable: 0; no extra infrastructure cost
Canary: routes a percentage of live traffic to new version; requires service mesh (Istio) or ingress-level routing; same infrastructure cost as rolling
Blue-Green: two full environments; traffic cuts over all-at-once; instant rollback; 2× infrastructure cost

DB migration pattern (zero-downtime canary):

Expand (add column, nullable) → Migrate (dual-write both versions) → Contract (remove old column)
Schema changes always deploy separately from code changes
All schema changes must be backward-compatible with the previous code version

16. Cheat Sheet

Automation maturity stages: Script → Workflow → Policy-driven → Self-healing

7 automation categories: Provisioning · Configuration · Access · Secrets · Observability · Incident · FinOps

SSA vs Config automation split:

SSA: Access, Secrets, Observability, Incident, FinOps, Provisioning (developer self-service, approval-gated)
Config automation: Patching (DevOps-controlled, engineer-present during execution)

Patching golden rules:

Immutable, not in-place
Sequential by region (AP → EU → US)
serial: 1 per node (never drain all simultaneously)
Validate 3 layers: infrastructure + Kubernetes + application
Rollback must be faster than forward patching
Stop and rollback on first failure; never promote a failed region

CI/CD 3 phases:

CI: lint + test + SAST (CodeQL) + SCA (Snyk)
Artifact Hardening: docker build + semantic+SHA tag + Trivy (CRITICAL/HIGH = fail) + Cosign sign
CD: deploy AP → SLO validate → deploy EU → SLO validate → deploy US → SLO validate + synthetic test → tag + Slack

Tagging rules: Never latest. Always {VERSION} + {github.sha} (dual tag).

SLO gates: error_rate < 2% + P95 < 200ms + CPU < 85% + mem < 85% + restarts < 3

Rollback: Parallel (all 3 regions), if: always(), IgnoreRegion input, re-render old manifests via envsubst

DB migration (canary-safe): Expand (add nullable column) → Migrate (dual-write) → Contract (drop old column after 100% stable)

Deployment strategies: Rolling (default) → Canary (critical path, Istio) → Blue-Green (most critical, 2× cost)

17. Gaps, Deferred Content & What’s Coming Next

Session explicitly deferred:

Helm (Phase 6) — chart standardisation, value inheritance, managing 500 charts
Observability (Phase 7) — Grafana dashboard JSON (300–400 dashboards), Loki pipeline, Tempo/OTel setup, Dynatrace AIOps integration
Terraform (Phase 4) — multi-project structure, shared VPC, state management, modules

Promised deliverables (instructor committed to uploading):

Live deployment demo recording (requested explicitly by attendees — instructor committed to recording and sharing)
SSA architecture diagram (EventBridge/Pub/Sub → Jenkins → Ansible → Terraform)
Simplied SSA scripts (React UI code not shared; backend script structure available in zip)
OS upgrade / application patching scripts (referenced; not shown in session)

Attendee feedback captured:

Too high-level: Multiple attendees noted the session explained what the scripts do but not why those design choices were made vs. alternatives. The patching scripts particularly lacked the “system design reasoning” behind the choices.
No live demo: The scripts were explained but not executed. Attendees requested seeing a real deployment that fails and rolls back automatically.
Canvas demo missing: The SSA was described as a React UI with forms, but no screenshot or recording was shared in-session (slides/UI snapshots promised separately).

What the previous Session 1 package already covers (do not re-read there): All CI/CD workflow code (Development.yml, Production.yml, rollback.yml, Staging.yml, etc.) is already documented in depth in the Session 1 package. This session added the reasoning (3 pipeline phases, deployment strategy selection, DB migration pattern) but the code was already in Session 1.