SRE Labs (Advanced Track) — Project 4: Titan Grid (500-Microservice Fintech Platform)

Structured educational resource covering sre labs (advanced track) — project 4: titan grid (500-microservice fintech platform).

senior 45 min read 16 sections

#kubernetes#cloud-k8s#aws#gcp

Complete Learning Package — Phases 1–3 + Real Code from Project Files

Context · System Architecture · Automation · CI/CD Pipelines · Kubernetes Manifests · Ansible Playbooks · Patching

Sources: 2026-02-21-19-06-51.md (session transcript) + real project files from Real_Infra_Projects.zip. The session described the architecture; the zip contains the actual implementation. This package integrates both: theory from the transcript, real code from the files, explained in context.

NDA note: The instructor anonymised the client as “Titan Grid.” Original service/company names appear in the code (marketplace-seller, driffle.com, cars24) — these are the real prior clients whose patterns were adapted. Credentials visible in values.yaml files belong to a previous client context and are not reproduced here.

2. Table of Contents

Executive Summary
Table of Contents
Project Context (Phase 1)
System Architecture (Phase 2)
CI/CD & Release Engineering — Real Code
- 5.1 Workflow Inventory
- 5.2 Development Pipeline (Canary + SLO Validation)
- 5.3 Production Pipeline — pre-prod.yml (Full CI/CD)
- 5.4 Production Pipeline — Production.yml (Matrix Rollout)
- 5.5 Staging Pipeline
- 5.6 Rollback Pipeline
- 5.7 Gateway-Only Pipelines
- 5.8 Reusable Build Workflow
- 5.9 PR Quality Gate (SonarQube)
- 5.10 SLO Validation Script
- 5.11 Synthetic Transaction Test
- 5.12 Key CI/CD Design Decisions
Kubernetes Manifests — Real Code
- 6.1 Production Server Deployment + HPA
- 6.2 Consumer Deployment
- 6.3 Gateway — Service + Kong HTTPRoute + Rate Limiting
- 6.4 Dev vs Prod Differences
- 6.5 Kong Per-Endpoint Rate Limiting (from gateway.yml)
Helm Chart Structure — Real Code
Automation (Phase 3) — Real Code
- 8.1 AWS Access Automation (Ansible)
- 8.2 GCP Access Automation
- 8.3 Jenkins Access Automation
- 8.4 Datadog Access Automation
- 8.5 IAM Policies (Least Privilege)
Patching Automation — Real Code
- 9.1 GitHub Actions Patching Pipeline
- 9.2 Ansible Node Rotation
- 9.3 Validation and Rollback
HealthCorp Non-Prod Scheduling Scripts
Architecture & Workflow Analysis
Key Concepts Table
Tools & Technologies
Interview Preparation
Exam & Certification Notes
Cheat Sheet
Gaps & What’s Still Coming

3. Project Context (Phase 1)

Industry: Fintech — real-time payments, risk/fraud, ledger, user management Regions: AP-South-1 (India), EU-Central-1 (Europe), US-East-1 (US) — all active-active Scale: ~100K daily transactions, 50K RPS, P95 < 200ms, 24×7, 1,000+ deploys/week Services: 500+ microservices across 150+ Business Units, each owning 4–5 services Compliance: PCI-DSS, GDPR, data residency, encryption at rest and in transit, full audit trail

Why 500 microservices: Not a technical decision — direct consequence of business expansion and the need for each BU to deploy independently. Each BU tags every resource: service_name, owner, bu, bms1/bms2, business_domain, env.

Team topology:

Team	Owns
Platform Engineering	GKE clusters, Terraform, networking, Helm base charts, observability platform
Domain Product Teams (per BU)	Service repos, Helm values, feature deployments within their namespace
SRE	Incident response, error budgets, SLA tracking, on-call escalation
Security	Audit, compliance, secrets, RBAC, DevSecOps pipeline gates

Domain decomposition: Payment · Risk/Fraud · Ledger · User · Notification · Analytics · 100+ more BUs.

4. System Architecture (Phase 2)

Traffic flow

Client (web / mobile / partner API / admin console)
  │
  ▼  Edge Layer
  GeoDNS   → latency + health + weight routing → nearest healthy region
  CDN      → cache static frontend assets
  DDoS     → block volumetric attacks
  Global LB → distribute to regional entry
  │
  ▼  Regional Entry (identical stack, all 3 regions)
  Regional LB (AWS ALB / GCP CLB)
  Kong API Gateway:
    ├── JWT authentication (validate Bearer token)
    ├── Per-endpoint rate limiting (Redis-backed KongPlugin)
    ├── IP allow/deny (block bots/scrapers)
    └── Inject X-Trace-ID header (unique per request)
  │
  ▼  Service Mesh (Istio)
  mTLS on all inter-service calls (encryption + mutual authentication)
  Retry policies + Circuit breakers
  Trace ID propagated via Istio headers to all downstream services
  │
  ▼  Domain Services (namespace-isolated on GKE)
  payment-initiate → fraud-detector → risk-scorer → ledger-service → notification-svc
  │
  ▼  Data Layer (per-region, self-contained)
  PostgreSQL multi-AZ (primary writes / read replica for reads)
  Redis (rate-limiting state, session cache)
  Kafka (async event streaming: payment-events, fraud-events, ledger-events, log-events)
  S3/GCS (audit logs, archives, analytics data)
  │
  ▼  Event Consumers (AP-South-1 only — home region)
  ledger-updater · risk-analytics · notification-worker · audit-writer

Multi-region write ownership (CAP: CP for writes, AP for reads)

Each account has a home region — the authoritative writer. Writes route cross-region to the home region if the user is accessing from elsewhere. Reads are served from local read replicas. This prevents split-brain and double-spend while keeping P95 latency within budget for reads (which far outnumber writes).

Kubernetes cluster structure

Node pools: General compute (On-Demand) · GPU/ML · Spot · Batch
Namespaces: one per domain (payments, risk, ledger, user, notification…)
Network Policies: enforce namespace-level communication restrictions
Resource Quotas: per-namespace compute budget (each BU has a limit)
RBAC: developers scoped to their own namespace only

5. CI/CD & Release Engineering — Real Code

5.1 Workflow Inventory

The payments service has 10 GitHub Actions workflows:

Workflow	Trigger	Purpose
`quality.yml`	PR open/sync/reopen	SonarQube SAST scan — blocks merge on findings
`Development.yml`	`workflow_dispatch`	Full canary deploy AP → EU → US with Istio + SLO validation
`Staging.yml`	`workflow_dispatch`	Build + single-region staging deploy
`pre-prod.yml`	`workflow_dispatch`	Full CI + Trivy + CodeQL + sequential prod deploy with SLO
`Production.yml`	`workflow_dispatch`	Matrix prod deploy; `IgnoreRegion` input to skip a degraded region
`rollback.yml`	`workflow_dispatch`	Matrix rollback to any version, any region
`dev_gw.yml`	`workflow_dispatch`	Dev gateway-only Kong config deploy
`stage_gw.yml`	`workflow_dispatch`	Stage gateway-only Kong config deploy
`prod_gw.yml`	`workflow_dispatch`	Prod gateway-only Kong config deploy
`reuse.yml`	`workflow_call`	Reusable Docker build (called from other service repos)

Authentication pattern used throughout — Workload Identity Federation (no long-lived keys):

- uses: google-github-actions/auth@v2
  with:
    workload_identity_provider: >-
      projects/${{ secrets.PROJECT_NUMBER }}/locations/global/
      workloadIdentityPools/github/providers/actions
    service_account: actions@${{ secrets.PROJECT_ID }}.iam.gserviceaccount.com

Cluster lookup pattern — region → cluster name via JSON secret:

- uses: google-github-actions/get-gke-credentials@v2
  with:
    cluster_name: ${{ fromJson(secrets.CLUSTER_MAP)['ap-south-1'] }}
    location:     ${{ fromJson(secrets.REGION_MAP)['ap-south-1'] }}
    project_id:   ${{ env.ENVIRONMENT_ID }}

CLUSTER_MAP and REGION_MAP are GitHub Secrets containing JSON maps. Adding a new region means updating the secret — no workflow changes needed.

5.2 Development Pipeline — Canary + SLO Validation

Development.yml is the most architecturally complete workflow. It implements progressive delivery: canary at 10% via Istio, validate with Prometheus + synthetic test, then promote to 100%. Runs sequentially — AP must pass before EU starts.

concurrency:
  group: Market-Seller
  cancel-in-progress: false   # never cancel a running deploy

jobs:
  CI:       # npm ci + test + lint + build
  Build:    # docker build + push to Artifact Registry (needs: CI)
  Deploy-AP:  # canary + validate + promote  (needs: Build)
  Deploy-EU:  # canary + validate + promote  (needs: Deploy-AP)
  Deploy-US:  # canary + validate + promote  (needs: Deploy-EU)
  Finalize:   # create git tag v{VERSION}-prod  (needs: Deploy-US)

Deploy-AP — the canary loop in detail:

Deploy-AP:
  runs-on: ubuntu-latest
  needs: Build
  environment: Production-ap-south-1
  steps:
    - uses: actions/checkout@v4
    - uses: google-github-actions/auth@v2 ...
    - uses: google-github-actions/get-gke-credentials@v2 ...

    - name: Deploy Canary
      run: |
        # Point new image at the deployment
        kubectl set image deployment/marketplace-seller \
          marketplace-seller=${{ env.IMAGE }}
        # Route 10% of traffic to new version via Istio VirtualService
        kubectl apply -f istio/canary-10.yaml
        # 3-minute observation window
        sleep 180

    - name: Synthetic Transaction Validation
      run: ./scripts/synthetic_test.sh ap-south-1

    - name: SLO + Burn Rate Validation
      run: |
        if ! ./scripts/validate_slo.sh ap-south-1; then
          echo "SLO failed. Rolling back..."
          ./scripts/rollback.sh ap-south-1
          exit 1
        fi

    - name: Promote to Full Traffic
      run: kubectl apply -f istio/canary-100.yaml   # 100% to new version

The canary mechanism: istio/canary-10.yaml is an Istio VirtualService with weighted routing (10% new / 90% old). After SLO validation passes, istio/canary-100.yaml shifts to 100%. This is L7 traffic splitting — both old and new pods run simultaneously; Istio distributes requests between them.

5.3 Production Pipeline (pre-prod.yml) — Full CI/CD

pre-prod.yml adds CodeQL SAST and dual image tagging (version + SHA):

jobs:
  CI:
    steps:
      - run: npm ci && npm test && npm run lint
      - uses: github/codeql-action/init@v3    # SAST: JavaScript
        with: { languages: javascript }
      - run: npm run build

  Build:
    needs: CI
    steps:
      - uses: docker/build-push-action@v5
        with:
          tags: |
            ${{ env.REGISTRY }}/.../seller:${{ env.VERSION }}
            ${{ env.REGISTRY }}/.../seller:${{ github.sha }}    # SHA tag for traceability
          cache-from: type=gha
          cache-to: type=gha,mode=max
      - uses: aquasecurity/trivy-action@master   # container vulnerability scan
        with:
          severity: CRITICAL,HIGH
          exit-code: 1    # fail pipeline if CRITICAL or HIGH CVEs found

  Deploy-AP:
    needs: Build
    environment: Production-ap-south-1
    steps:
      - run: |
          kubectl set image deployment/marketplace-seller \
            marketplace-seller=${{ env.IMAGE }}
          kubectl rollout status deployment/marketplace-seller
      - run: ./scripts/validate_slo.sh ap-south-1

  Deploy-EU:
    needs: Deploy-AP     # sequential — AP must pass first

  Deploy-US:
    needs: Deploy-EU     # sequential — EU must pass first
    steps:
      # also runs synthetic test in US
      - run: |
          if ! ./scripts/validate_slo.sh us-east-1; then
            ./scripts/rollback.sh us-east-1
            exit 1
          fi
      - run: ./scripts/synthetic_test.sh us-east-1

  Finalize:
    needs: Deploy-US
    steps:
      - uses: rickstaa/action-create-tag@v1
        with:
          tag: v${{ env.VERSION }}-prod
          force_push_tag: true

5.4 Production Pipeline (Production.yml) — Matrix Rollout

Production.yml uses a matrix strategy — all 3 regions run in parallel — with an IgnoreRegion input to skip a degraded region:

on:
  workflow_dispatch:
    inputs:
      Version:        { description: "Release Version", required: true, type: string }
      Fresh:          { description: "Fresh Deployment", type: choice, options: [Yes, No] }
      IgnoreRegion:
        description: "Which region to ignore?"
        type: choice
        options: [None, ap-south-1, eu-central-1, us-east-1]

jobs:
  Deploy:
    environment: Production(${{ matrix.REGION }})
    strategy:
      matrix:
        REGION: [ap-south-1, eu-central-1, us-east-1]
    env:
      CLUSTER_NAME:   ${{ fromJson(secrets.CLUSTER_MAP)[matrix.REGION] }}
      CLUSTER_REGION: ${{ fromJson(secrets.REGION_MAP)[matrix.REGION] }}

    steps:
      - name: Create Manifests
        env:
          IMAGE:            ${{ env.REGISTRY }}/.../${{ env.REPOSITORY }}:${{ env.VERSION }}
          HEALTHCHECK_PATH: /seller/health
          REGION:           ${{ matrix.REGION }}
        run: |
          # envsubst renders ${IMAGE}, ${REGION}, ${ENVIRONMENT_ID}, ${GITHUB_SHA}
          envsubst < ./.k8s/prod/server.yml   > server.yml
          envsubst < ./.k8s/prod/consumer.yml > consumer.yml
          envsubst < ./.k8s/prod/secrets.yml  > secrets.yml
          envsubst < ./.k8s/prod/gateway.yml  > gateway.yml

      - name: Deploy Consumers        # ONLY in AP-South-1 (home region)
        if: ${{ matrix.REGION == 'ap-south-1' }}
        run: |
          kubectl apply -f ./consumer.yml
          kubectl rollout status deploy/marketplace-seller-consumers

      - name: Deploy Server
        run: |
          kubectl apply -f ./server.yml
          kubectl rollout status deploy/marketplace-seller

      - name: Deploy Gateway Configuration
        run: kubectl apply -f ./gateway.yml

if: matrix.REGION == 'ap-south-1' is the CI/CD implementation of the home-region architecture. Kafka consumers only run in AP-South-1. EU and US get the server but no consumer — they produce events into Kafka; AP-South-1 consumes and processes them. This prevents duplicate event processing and maintains one authoritative ledger updater.

5.5 Staging Pipeline

Staging.yml — single region, no matrix, no SLO validation — used for integration testing before production:

jobs:
  Build:
    # Same build + push pattern as Production
    # Uses GCP_STAGING_ID (separate GCP project)

  Deploy:
    environment: Staging
    steps:
      - name: Create Manifests
        run: |
          envsubst < ./.k8s/stage/server.yml   > server.yml
          envsubst < ./.k8s/stage/consumer.yml > consumer.yml

      - name: Deploy Secrets
        run: |
          kubectl apply -f ./.k8s/stage/secrets.yml
          kubectl delete secrets/marketplace-seller    # force re-create
          sleep 5

      - name: Deploy Server
        run: |
          kubectl apply -f ./server.yml
          kubectl rollout status deploy/marketplace-seller

      - name: Deploy Consumers
        run: |
          kubectl apply -f ./consumer.yml
          kubectl rollout status deploy/marketplace-seller-consumers

      - name: Deploy Gateway Configuration
        run: kubectl apply -f ./.k8s/stage/gateway.yml

  slackNotification:
    needs: Deploy
    steps:
      - uses: rickstaa/action-create-tag@v1
        with: { tag: "v${{ env.VERSION }}-stage" }

Note: staging deploys consumers (both server and consumer processes) — unlike production where consumers are AP-only. Staging is a single cluster and doesn’t need the multi-region ownership constraint.

5.6 Rollback Pipeline

rollback.yml is particularly well-designed. Key architectural choices:

on:
  workflow_dispatch:
    inputs:
      Version:       { description: "Version to Rollback To (e.g. 1.2.3)", required: true }
      Reason:        { description: "Reason for rollback", required: true }
      IgnoreRegion:  { type: choice, options: [None, ap-south-1, eu-central-1, us-east-1] }

jobs:
  rollback:
    strategy:
      matrix:
        REGION: [ap-south-1, eu-central-1, us-east-1]
    if: always()    # run even if some matrix siblings fail

    steps:
      - name: Skip Ignored Region
        if: github.event.inputs.IgnoreRegion == matrix.REGION
        run: echo "Skipping ${{ matrix.REGION }}" && exit 0

      - name: Generate Manifests with Old Image
        env:
          IMAGE:   ${{ env.REGISTRY }}/.../seller:${{ env.VERSION }}    # ← OLD version
          REGION:  ${{ matrix.REGION }}
        run: |
          envsubst < ./.k8s/prod/server.yml   > server.yml
          envsubst < ./.k8s/prod/consumer.yml > consumer.yml
          envsubst < ./.k8s/prod/secrets.yml  > secrets.yml
          envsubst < ./.k8s/prod/gateway.yml  > gateway.yml

      - name: Apply Rollback Manifests
        run: |
          kubectl apply -f ./secrets.yml
          kubectl apply -f ./consumer.yml || echo "Consumer not applicable"
          kubectl apply -f ./server.yml
          kubectl apply -f ./gateway.yml
          kubectl rollout status deploy/marketplace-seller || true
          kubectl rollout status deploy/marketplace-seller-consumers || true

      - name: Slack Notification (Success)
        if: success()
        env:
          SLACK_MESSAGE: |
            Rolled back Seller Service to version `${{ env.VERSION }}`
            Region: `${{ matrix.REGION }}`
            Reason: `${{ github.event.inputs.Reason }}`

  tag-rollback:
    needs: rollback
    steps:
      - uses: rickstaa/action-create-tag@v1
        with: { tag: "v${{ env.VERSION }}-rollback" }

Why not kubectl rollout undo:

Can only roll back one revision, has no cross-region coordination.
Bypasses the manifest pipeline — no audit trail, no Slack notification.
The envsubst approach means rollback goes through exactly the same pipeline as forward deploy, just with an older $VERSION value. Every rollback is tagged, auditable, reason-documented.

if: always() on the matrix job: ensures all 3 regions are rolled back even if one fails — critical in a P0 where you need all regions reverted regardless of partial failures.

5.7 Gateway-Only Pipelines

dev_gw.yml, stage_gw.yml, prod_gw.yml — deploy only Kong config without touching the application Deployment. Allows updating rate limits, routing rules, and request transformations independently of app releases:

- name: Create Manifests
  env:
    HEADER_NAME:  'api-gateway-key'
    HEADER_VALUE: ${{ secrets.GATEWAY_HEADER_VALUE }}
    BASE_SLUG:    '/api/seller/legacy'
    BASE_URL:     'services.driffle.live'   # dev environment
    REDIS_SLUG:   'dev'
  run: envsubst < ./.k8s/api/gateway.yml > gateway.yml

- name: Deploy Gateway Configuration
  run: kubectl apply -f ./gateway.yml

gateway.yml is 896 lines of KongPlugin + HTTPRoute resources — one per API endpoint with different rate limits and transformations.

5.8 Reusable Build Workflow

reuse.yml — workflow_call that any service in the organisation can call to avoid duplicating build logic:

on:
  workflow_call:
    inputs:
      version:        { required: true, type: string }
      environment_id: { required: true, type: string }
      repository:     { required: true, type: string }
    secrets:
      artifact_registry: { required: true }
      project_id:        { required: true }
      project_number:    { required: true }
      github_token:      { required: true }

jobs:
  build-and-push:
    steps:
      - uses: actions/checkout@v4
      - uses: google-github-actions/auth@v2 ...
      - uses: docker/login-action@v3 ...
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v5
        with:
          tags: ${{ env.REGISTRY }}/${{ env.ENVIRONMENT_ID }}/${{ env.REPOSITORY }}:${{ env.VERSION }}
          cache-from: type=gha
          cache-to: type=gha,mode=max
          build-args: GITHUB_TOKEN=${{ secrets.github_token }}

Usage from another service repo:

jobs:
  build:
    uses: org/payments/.github/workflows/reuse.yml@main
    with:
      version: "1.2.3"
      environment_id: ${{ secrets.GCP_PROD_ID }}
      repository: risk-service
    secrets: inherit

At 500 services with identical build patterns, this prevents 500 copies of the same Docker build YAML drifting independently.

5.9 PR Quality Gate

quality.yml — blocks every PR from merging if SonarQube finds issues:

on:
  pull_request:
    types: [opened, synchronize, reopened]

jobs:
  Analysis:
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0    # full history for accurate coverage and blame
      - uses: sonarsource/sonarqube-scan-action@v2
        env:
          SONAR_TOKEN:    ${{ secrets.SONAR_TOKEN }}
          SONAR_HOST_URL: ${{ vars.SONAR_HOST_URL }}

fetch-depth: 0 ensures SonarQube gets the full git history — critical for accurate change-scope analysis and not just scanning the PR diff.

5.10 SLO Validation Script

validate_slo.sh — runs after every deployment, queries Prometheus live:

#!/bin/bash
REGION=$1
SERVICE="marketplace-seller"
PROMETHEUS_URL="https://prometheus.${REGION}.titangrid.internal"
ERROR_THRESHOLD=0.02         # 2% error rate maximum
P95_LATENCY_THRESHOLD=500    # 500ms P95 maximum
CPU_THRESHOLD=85             # 85% CPU maximum
MEM_THRESHOLD=85             # 85% memory maximum
WINDOW="5m"

sleep 120   # allow metrics to stabilise post-deploy

# 1. Error rate (5xx / total requests)
ERROR_RATE=$(curl -sG \
  --data-urlencode "query=sum(rate(http_requests_total{service=\"$SERVICE\",status=~\"5..\"}[$WINDOW])) \
    / sum(rate(http_requests_total{service=\"$SERVICE\"}[$WINDOW]))" \
  $PROMETHEUS_URL/api/v1/query | jq -r '.data.result[0].value[1]')

echo "Error rate: $ERROR_RATE"
if (( $(echo "$ERROR_RATE > $ERROR_THRESHOLD" | bc -l) )); then
  echo "Error rate exceeds threshold ($ERROR_THRESHOLD)"; exit 1
fi

# 2. P95 latency (histogram_quantile)
P95_LATENCY=$(curl -sG \
  --data-urlencode "query=histogram_quantile(0.95, \
    sum(rate(http_request_duration_seconds_bucket{service=\"$SERVICE\"}[$WINDOW])) by (le)) * 1000" \
  $PROMETHEUS_URL/api/v1/query | jq -r '.data.result[0].value[1]')

echo "P95 latency: ${P95_LATENCY}ms"
if (( $(echo "$P95_LATENCY > $P95_LATENCY_THRESHOLD" | bc -l) )); then
  echo "P95 latency exceeds ${P95_LATENCY_THRESHOLD}ms"; exit 1
fi

# 3. CPU and memory via kubectl top
CPU_USAGE=$(kubectl top pod -l app=$SERVICE --no-headers | awk '{sum+=$2} END {print sum}')
MEM_USAGE=$(kubectl top pod -l app=$SERVICE --no-headers | awk '{sum+=$3} END {print sum}')
[ "$CPU_USAGE" -gt "$CPU_THRESHOLD" ] && { echo "CPU too high"; exit 1; }
[ "$MEM_USAGE" -gt "$MEM_THRESHOLD" ] && { echo "Memory too high"; exit 1; }

# 4. CrashLoop check
CRASHLOOP=$(kubectl get pods -l app=$SERVICE -o json | \
  jq '.items[].status.containerStatuses[].restartCount' | awk '{sum+=$1} END {print sum}')
[ "$CRASHLOOP" -gt 3 ] && { echo "CrashLoop detected ($CRASHLOOP restarts)"; exit 1; }

echo "SLO validation passed for region: $REGION"; exit 0

What this gates: error_rate < 2% + P95 < 500ms + CPU < 85% + memory < 85% + restart count < 3. All five must pass. If any fails, the pipeline rolls back and stops propagation to the next region.

5.11 Synthetic Transaction Test

synthetic_test.sh — end-to-end payment flow test, runs before SLO validation:

#!/bin/bash
REGION=$1
BASE_URL="https://api.$REGION.titangrid.com"

# 1. Login and obtain JWT
TOKEN=$(curl -s -X POST $BASE_URL/login \
  -d '{"username":"synthetic_user","password":"test123"}' \
  -H "Content-Type: application/json" | jq -r '.token')

[ -z "$TOKEN" ] && { echo "Login failed"; exit 1; }

# 2. Execute a test payment (amount: 1 unit)
RESPONSE=$(curl -s -X POST $BASE_URL/payment \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"amount":1}')

STATUS=$(echo $RESPONSE | jq -r '.status')
[ "$STATUS" != "SUCCESS" ] && { echo "Payment failed: $STATUS"; exit 1; }

echo "Synthetic transaction passed"; exit 0

This validates the full user-facing path: Kong auth → payment service → response. It’s not a unit test — it proves the deployed version actually processes a payment end-to-end from the regional API endpoint.

5.12 Key CI/CD Design Decisions

Decision	Why
Sequential AP → EU → US	Bad release stops at AP; EU/US never see it. Parallel would hit all 3 simultaneously.
Canary via Istio VirtualService	True L7 traffic split; works with existing HPA; no duplicate Deployment needed
`envsubst` for manifest templating	Lightweight; no Helm release state to track; same template works for all 3 regions via `${REGION}`
Rollback re-renders old manifests	Same pipeline path as forward deploy; cross-region coordinated; auditable; tagged
Consumers only in `ap-south-1`	One authoritative Kafka consumer; no duplicate event processing across regions
Workload Identity Federation	No long-lived IAM keys to rotate, store, or leak
`cancel-in-progress: false`	Never interrupt a running deploy; prevents partial-region states
Docker GHA cache (`type=gha`)	Saves ~2–3 min per build by reusing unchanged layers
Trivy + CodeQL in pipeline	Container scan after build (before deploy), SAST in CI — both are hard blockers (exit-code: 1)
Slack on failure only for jobs; on success for final job	Signal-to-noise: alert on what needs action; celebrate completion

6. Kubernetes Manifests — Real Code

6.1 Production Server Deployment + HPA

.k8s/prod/server.yml (rendered via envsubst at deploy time):

apiVersion: v1
kind: ServiceAccount
metadata:
  name: marketplace-seller
  namespace: services
  annotations:
    iam.gke.io/gcp-service-account: marketplace-seller@${ENVIRONMENT_ID}.iam.gserviceaccount.com
    # Workload Identity: maps K8s ServiceAccount → GCP Service Account

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: marketplace-seller
  namespace: services
  labels:
    app.kubernetes.io/environment: production
    app.kubernetes.io/commit: ${GITHUB_SHA}
  annotations:
    kubernetes.io/change-cause: ${GITHUB_SHA}    # shown in rollout history
spec:
  selector:
    matchLabels:
      app.kubernetes.io/name: marketplace-seller
      app.kubernetes.io/environment: production
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 0     # zero-downtime: never kill old pod before new is Ready
      maxSurge: 100%        # can double pod count during rollout
  revisionHistoryLimit: 2
  replicas: 1               # base replica count; HPA manages actual count

  template:
    metadata:
      labels:
        app.kubernetes.io/name: marketplace-seller
        app.kubernetes.io/environment: production
    spec:
      serviceAccountName: marketplace-seller
      terminationGracePeriodSeconds: 15    # graceful drain window on SIGTERM

      containers:
        - name: server
          image: ${IMAGE}
          imagePullPolicy: Always
          command:
            - node
            - -r
            - "@aspecto/opentelemetry/auto-instrument"   # OTel traces — zero code changes
            - -r
            - "@driffle/log-tastic/console"              # structured JSON logging
            - /app/dist/server.js
          ports:
            - name: primary
              containerPort: 3000

          resources:
            requests: { memory: 512Mi, cpu: 256m }
            limits:   { memory: 1Gi,   cpu: 512m }

          envFrom:
            - secretRef: { name: marketplace-seller }   # all secrets injected as env vars

          env:
            - name:  OTEL_EXPORTER_OTLP_TRACES_ENDPOINT
              value: "https://otel.prod.driffle.com/v1/traces"
            - name:  NODE_ENV
              value: production
            - name:  FASTIFY_PORT
              value: "3000"
            - name:  FASTIFY_BODY_LIMIT
              value: "52428800"    # 50 MB (dev is 10 MB)
            - name:  ALLOWED_ORIGINS
              value: "https://driffle.com,https://web.prod.${REGION}.driffle.com"
              # ${REGION} substituted at deploy time — same manifest for all 3 regions

          livenessProbe:
            httpGet: { path: /seller/health, port: 3000 }
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3

          readinessProbe:
            httpGet: { path: /seller/health, port: 3000 }
            initialDelaySeconds: 5
            periodSeconds: 5
            failureThreshold: 3

---
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: marketplace-seller
  namespace: services
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: marketplace-seller
  minReplicas: 2
  maxReplicas: 4
  metrics:
    - type: Resource
      resource:
        name: cpu
        target: { type: Utilization, averageUtilization: 75 }
    - type: Resource
      resource:
        name: memory
        target: { type: Utilization, averageUtilization: 75 }

Key design choices in this manifest:

maxUnavailable: 0 + maxSurge: 100% = surge-first rolling update. New pods are created and must become Ready before any old pods are terminated. True zero-downtime.
terminationGracePeriodSeconds: 15 = Node.js gets 15 seconds after SIGTERM to drain in-flight requests before SIGKILL. Essential for zero-dropped-requests during deploys.
OTel auto-instrumentation injected via -r flag at process start — the application code has no tracing imports; the mesh layer handles it.
ALLOWED_ORIGINS contains ${REGION} — the same YAML file produces correct CORS for AP, EU, and US at deploy time.

6.2 Consumer Deployment

.k8s/prod/consumer.yml — same image, different entrypoint, separate Deployment:

spec:
  replicas: 3     # more than server; Kafka consumers benefit from parallelism

  template:
    spec:
      containers:
        - name: consumers
          image: ${IMAGE}
          command:
            - node
            - -r
            - "@driffle/log-tastic/console"
            - /app/dist/consumer.js       # different entrypoint: consumer process
          ports: []                        # no HTTP port — consumers pull from Kafka
          resources:
            requests: { memory: 256Mi, cpu: 256m }
            limits:   { memory: 768Mi }   # deliberately: no CPU limit on consumers

Why no CPU limit on consumers: CPU throttling would cause Kafka consumer lag — messages would pile up in topics faster than they’re processed. Consumer throughput should scale with available CPU, not be capped. Why no probes: consumers don’t expose HTTP endpoints. Kubernetes monitors process exit codes instead. Why 3 replicas: Kafka partitions can be consumed in parallel. 3 consumer replicas on a 3-partition topic means each replica handles exactly one partition.

6.3 Gateway — Service + Kong HTTPRoute + Rate Limiting

.k8s/prod/gateway.yml:

# 1. ClusterIP Service (internal cluster routing)
apiVersion: v1
kind: Service
metadata:
  name: marketplace-seller
  namespace: services
spec:
  type: ClusterIP
  selector:
    app.kubernetes.io/name: marketplace-seller
  ports:
    - port: 3000
      targetPort: 3000

---
# 2. GCP-native health check policy for the load balancer
apiVersion: networking.gke.io/v1
kind: HealthCheckPolicy
metadata:
  name: marketplace-seller
  namespace: services
spec:
  default:
    unhealthyThreshold: 2      # mark unhealthy after 2 consecutive failures
    healthyThreshold: 3        # recover after 3 consecutive successes
    timeoutSec: 4
    checkIntervalSec: 5
    config:
      type: HTTP
      httpHealthCheck:
        port: 3000
        requestPath: "/seller/health"
  targetRef: { kind: Service, name: marketplace-seller }

---
# 3. Internal HTTPRoute (region-scoped internal URL)
apiVersion: gateway.networking.k8s.io/v1beta1
kind: HTTPRoute
metadata:
  name: marketplace-seller-internal
  namespace: services
spec:
  hostnames:
    - seller.prod.${REGION}.driffle.com    # ${REGION} substituted at deploy time
  parentRefs:
    - kind: Gateway
      name: internal-lb
      namespace: infra
      sectionName: https

---
# 4. Kong HTTPRoute (external, public-facing)
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: marketplace-seller
  namespace: services
  annotations:
    konghq.com/plugins: global-cors, rl-seller-base   # CORS + base rate limiting
spec:
  parentRefs:
    - name: kong
  hostnames:
    - seller.driffle.com
  rules:
    - matches:
        - path: { type: PathPrefix, value: /seller }
      backendRefs:
        - name: marketplace-seller
          kind: Service
          port: 3000

---
# 5. Kong rate-limiting plugin (base: 10 req/sec per IP)
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: rl-seller-base
  namespace: services
  annotations:
    kubernetes.io/ingress.class: kong
plugin: rate-limiting
config:
  second: 10           # 10 req/sec maximum
  policy: local        # local counter (no Redis needed for this broad limit)
  limit_by: ip
  fault_tolerant: true # if rate limiter fails, allow traffic through
  error_code: 429
  error_message: "Too Many Requests"

6.4 Dev vs Prod Manifest Differences

Same structural YAML, different values:

Setting	Dev (`.k8s/dev/`)	Prod (`.k8s/prod/`)
`environment` label	`development`	`production`
OTel endpoint	`otel.dev.driffle.com`	`otel.prod.driffle.com`
`FASTIFY_BODY_LIMIT`	`10485760` (10 MB)	`52428800` (50 MB)
`ALLOWED_ORIGINS`	`web.dev.driffle.com, localhost:4001`	`driffle.com, web.prod.${REGION}.driffle.com`
`BASE_URL`	`seller.dev.driffle.com/seller`	`seller.driffle.com/seller`
HPA	Not present	min 2, max 4
`maxSurge`	100%	100%

The manifests are structurally identical by design — envsubst substitutes environment-specific values, and each environment folder (dev/, stage/, prod/) only differs in hardcoded strings.

6.5 Kong Per-Endpoint Rate Limiting

.k8s/api/gateway.yml (896 lines) — one KongPlugin per endpoint with different limits:

# Bulk operation — very strict: 12/min per authenticated user
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: rl-api-seller-offer-update-bulk
plugin: rate-limiting
config:
  minute: 12               # bulk updates are expensive; cap tightly
  limit_by: header
  header_name: Authorization    # per-user (not per-IP for authenticated endpoints)
  policy: redis
  redis:
    database: 6
    host: ${REDIS_SLUG}.redis.driffle.com    # distributed — shared across Kong pods

---
# Standard operation — lenient: 200/min per authenticated user
apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: rl-api-seller-offer-update
plugin: rate-limiting
config:
  minute: 200
  limit_by: header
  header_name: Authorization
  policy: redis
  redis:
    database: 6
    host: ${REDIS_SLUG}.redis.driffle.com

Request transformation — URL rewrite at gateway:

apiVersion: configuration.konghq.com/v1
kind: KongPlugin
metadata:
  name: api-seller-dashboard
plugin: request-transformer
config:
  http_method: GET
  replace:
    uri: /seller/dashboard/        # rewrite external path to internal path
  add:
    headers:
      - "${HEADER_NAME}:${HEADER_VALUE}"    # inject internal auth header

Design pattern: external API contracts (/api/seller/legacy/...) are decoupled from internal service paths (/seller/...). Internal services can be refactored without breaking external API consumers.

7. Helm Chart Structure — Real Code

The payments/ directory is a Helm chart for a Java service (LMS). This shows the dual-approach: raw envsubst manifests for the primary Node.js services, Helm for Java microservices with more complex configuration needs.

payments/ (Helm chart)
├── Chart.yaml              → apiVersion: v2, name: c2b-lms, version: 0.1.0
├── templates/
│   ├── _helpers.tpl        → Helm helper functions (name, fullname, labels)
│   ├── deployment.yaml     → Helm deployment template with Datadog integration
│   ├── service.yaml        → ClusterIP service
│   ├── config.yaml         → ConfigMap from .Values.configs
│   ├── ambassdor.yaml      → Ambassador/API gateway routing (env-aware)
│   └── ingress.yaml        → commented out (using ambassador)
├── qa/values.yaml          → QA environment Helm values
└── stage/values.yaml       → Staging environment Helm values

templates/deployment.yaml — Datadog integration pattern:

spec:
  template:
    metadata:
      annotations:
        rollme: {{ randAlphaNum 5 | quote }}    # force pod restart on every helm upgrade
        deployedAt: {{ now | date "Mon Jan 2 15:04:05 MST 2006" }}
        sha: {{ .Values.sha | quote }}
    spec:
      containers:
        - env:
            - name: DD_SERVICE
              value: {{ .Chart.Name }}              # Datadog service name = chart name
            - name: DD_AGENT_HOST
              valueFrom:
                fieldRef:
                  fieldPath: status.hostIP         # DaemonSet Datadog agent on node IP
            - name: DD_ENV
              value: {{ .Values.configs.ENV }}
            - name: DD_PROFILING_ENABLED
              value: {{ default "false" .Values.datadog.profiling | quote }}
            - name: DD_APM_ENABLED
              value: {{ default "false" .Values.datadog.profiling | quote }}

rollme: {{ randAlphaNum 5 | quote }} is a Helm trick — adds a random annotation to every deploy, which changes the pod template hash and forces Kubernetes to restart all pods even if the image tag hasn’t changed. Used when config changes need to propagate.

templates/ambassdor.yaml — environment-aware routing:

spec:
  {{ if eq .Values.configs.ENV "qa" }}
  host: {{ .Chart.Name }}.{{ .Values.domain.name }}
  {{ else }}
  host: {{ .Values.configs.ENV }}-{{ .Chart.Name }}.{{ .Values.domain.name }}
  {{ end }}

Helm values pattern (structure only — no actual credentials):

# qa/values.yaml structure:
replicaCount: 1
datadog:
  profiling: false
configs:
  ENV: qa
  ARANGODB_HOST: <host>      # ArangoDB (graph DB)
  GCP_PUBSUB_PROJECTID: ...  # Pub/Sub project
  LMS_CLIENT_ID: ...         # OAuth client
resources:
  requests: { cpu: 500m, memory: 512Mi }
  limits:   { cpu: 1000m, memory: 1Gi }

8. Automation (Phase 3) — Real Code

All access automation follows the same Ansible pattern: authenticate → check if entity exists → create if not → attach policy/role

This is the idempotency pattern — running the playbook twice produces the same result as running it once. Safe to re-run during onboarding retries.

8.1 AWS Access Automation

Infra Automation/AWS/eks_access.yml — representative of all 7 AWS resource playbooks:

- name: Manage AWS EKS IAM User Access
  hosts: localhost
  vars_files:
    - vars.yml    # aws_access_key, aws_secret_key, aws_session_token, user_email

  tasks:
    - name: Authenticate with AWS CLI (STS temporary credentials)
      shell: |
        export AWS_ACCESS_KEY_ID="{{ aws_access_key }}"
        export AWS_SECRET_ACCESS_KEY="{{ aws_secret_key }}"
        export AWS_SESSION_TOKEN="{{ aws_session_token }}"
        aws sts get-caller-identity
      register: aws_auth_output

    - name: Check if IAM user exists
      shell: aws iam get-user --user-name {{ user_email }}
      register: user_check
      failed_when: user_check.rc not in [0, 255]
      ignore_errors: yes

    - name: Create IAM user if not exists
      shell: aws iam create-user --user-name {{ user_email }}
      when: user_check.rc != 0    # idempotent: skipped if user already exists

    - name: Attach IAM policy to user for EKS access
      shell: |
        aws iam put-user-policy \
          --user-name {{ user_email }} \
          --policy-name EKSAccessPolicy \
          --policy-document file://{{ eks_policy_file }}

AWS resources covered (one playbook each): ec2 · eks · rds · s3_bucket · lambda · ecs · sqs

8.2 GCP Access Automation

Infra Automation/GCP/GCS/gke_access.yml:

- name: Grant access to GKE
  tasks:
    - name: Authenticate with GCP
      shell: |
        gcloud auth activate-service-account \
          --key-file={{ gcp_service_account_json }} \
          --project={{ gcp_project }}

    - name: Assign IAM role to user for GKE cluster
      shell: |
        gcloud projects add-iam-policy-binding {{ gcp_project }} \
          --member="user:{{ user_email }}" \
          --role="{{ gke_role }}" \
          --condition=None
          # gke_role examples: roles/container.viewer | roles/container.admin

GCP resources covered: gke · gcs · cloud_run · cloud_sql_read · compute_engine · pubsub · cloud_function

vars.yml pattern — single file, comment/uncomment the active resource block:

# Active config — GKE access:
gcp_service_account_json: "/path/to/automation.json"
gcp_project: "my-project-id"
user_email: "new.engineer@company.com"
gke_role: "roles/container.viewer"

# Commented out — GCS bucket:
# gcs_bucket_name: "my-bucket"
# gcs_role: "roles/storage.objectViewer"

8.3 Jenkins Access Automation

Infra Automation/CI-CD/Jenkins/jenkins_user_access.yml — uses Jenkins REST API:

- name: Manage Jenkins User Access
  tasks:
    - name: Check if user exists in Jenkins
      uri:
        url: "{{ jenkins_url }}/user/{{ username }}/api/json"
        method: GET
        user: "{{ jenkins_admin_user }}"
        password: "{{ jenkins_admin_token }}"
      register: user_check
      ignore_errors: yes

    - name: Create Jenkins user if not exists
      uri:
        url: "{{ jenkins_url }}/securityRealm/createAccount"
        method: POST
        body_format: form-urlencoded
        body:
          username: "{{ username }}"
          password1: "{{ password }}"
          fullname:  "{{ fullname }}"
          email:     "{{ email }}"
      when: user_check.status != 200

    - name: Assign job permissions to user
      uri:
        url: "{{ jenkins_url }}/job/{{ job_name }}/config.xml"
        method: POST
        body: |
          <hudson.security.AuthorizationMatrixProperty>
            <permission>hudson.model.Item.Read:{{ username }}</permission>
            <permission>hudson.model.Item.Build:{{ username }}</permission>
          </hudson.security.AuthorizationMatrixProperty>

8.4 Datadog Access Automation

Infra Automation/Observability/datadog_access.yml — uses Datadog REST API v2:

- name: Manage Datadog Access
  tasks:
    - name: List all users from Datadog API
      uri:
        url: "https://api.datadoghq.com/api/v2/users"
        method: GET
        headers:
          Authorization: "Bearer {{ datadog_api_key }}"
      register: user_check

    - name: Extract user ID by email
      set_fact:
        user_id: >-
          {{ user_check.json.data
             | selectattr('attributes.email', 'equalto', user_email)
             | map(attribute='id')
             | list | first }}

    - name: Fail if user not found in Datadog
      fail:
        msg: "User {{ user_email }} not found — must be invited to org first"
      when: user_id is not defined

    - name: Assign role to user
      uri:
        url: "https://api.datadoghq.com/api/v2/users/{{ user_id }}/roles"
        method: POST
        body_format: json
        body:
          data:
            - id: "{{ role_id }}"
              type: "roles"

8.5 IAM Policies — Least Privilege

// ec2_policy.json — scoped to a specific instance, not *
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Action": ["ec2:DescribeInstances", "ec2:StartInstances", "ec2:StopInstances"],
    "Resource": "arn:aws:ec2:ap-south-1:ACCOUNT_ID:instance/i-SPECIFIC_ID"
  }]
}

// eks_policy.json — read-only EKS access, wildcard on resource
{
  "Statement": [{
    "Effect": "Allow",
    "Action": [
      "eks:DescribeCluster", "eks:ListClusters",
      "eks:DescribeNodegroup", "eks:ListNodegroups",
      "eks:DescribeFargateProfile", "eks:ListFargateProfiles"
    ],
    "Resource": "*"   // read-only actions are safe with wildcard
  }]
}

The EC2 policy scopes to a specific instance ARN — an engineer onboarded for one instance cannot control any other. The EKS policy is read-only (describe/list only — no create/delete) so wildcard resource is acceptable.

9. Patching Automation — Real Code

9.1 GitHub Actions Patching Pipeline

Infra Patching/.github/workflows/patching.yml:

name: TitanGrid Production Patching

on:
  workflow_dispatch:
    inputs:
      cve_id:
        description: "CVE Reference (e.g. CVE-2026-12345)"
        required: true
      patch_type:
        description: "Type of patch (node | os | kubernetes)"
        required: true
        default: "node"

jobs:
  patch-ap:
    name: Patch AP-SOUTH
    runs-on: ubuntu-latest
    environment: production-patching    # requires manual approval gate in GitHub
    steps:
      - uses: actions/checkout@v4
      - run: sudo apt install -y ansible

      - name: Execute Patching — AP
        run: |
          ansible-playbook ansible360/patching/patch_orchestrator.yml \
            -e region=ap_south \
            -e patch_type=${{ github.event.inputs.patch_type }}

      - name: Validate Cluster Health — AP
        run: |
          ansible-playbook ansible360/patching/validate_cluster.yml \
            -e region=ap_south

      - name: Rollback AP if Failed
        if: failure()            # automatic rollback on any step failure
        run: |
          ansible-playbook ansible360/patching/rollback_nodegroup.yml \
            -e region=ap_south

  patch-eu:
    needs: patch-ap              # sequential: EU only starts if AP succeeded
    # ... identical steps with region=eu_central ...

  patch-us:
    needs: patch-eu              # sequential: US only starts if EU succeeded
    # ... identical steps with region=us_east ...

  notify-complete:
    needs: patch-us
    steps:
      - run: echo "TitanGrid patching for ${{ github.event.inputs.cve_id }} complete"

The environment: production-patching means this job requires a GitHub Environment protection rule — typically a manual approval from a senior engineer before patching begins.

9.2 Ansible Node Rotation

Infra Patching/patching/patch_orchestrator.yml:

- name: TitanGrid Multi-Region Patching
  hosts: localhost
  tasks:
    - name: Loop through regions sequentially
      include_tasks: rotate_nodegroup.yml
      loop: [ap_south, eu_central, us_east]
      loop_control:
        loop_var: region

Infra Patching/patching/rotate_nodegroup.yml:

- name: Rotate Nodegroup for {{ region }}
  serial: 1    # one node at a time — never drain all nodes simultaneously

  tasks:
    - name: Create new patched node image
      shell: ./build_new_node_image.sh {{ region }}
      # Bakes a new AMI with the CVE patch applied

    - name: Update nodegroup with new image
      shell: ./update_nodegroup.sh {{ region }}
      # Updates ASG launch template to use the new AMI

    - name: Drain old node
      shell: |
        kubectl drain {{ inventory_hostname }} \
          --ignore-daemonsets \
          --delete-emptydir-data
      # Evicts all pods off the old node; they reschedule onto other nodes

    - name: Validate pods rescheduled
      shell: kubectl get pods -A
      # Verify no pods stuck in Pending

    - name: Delete old node
      shell: ./delete_old_node.sh {{ inventory_hostname }}
      # Terminates the old EC2 instance

9.3 Validation and Rollback

Infra Patching/patching/validate_cluster.yml:

tasks:
  - name: Check API server health
    shell: kubectl get nodes
    # If kubectl can't reach the API server, the cluster is in bad shape

  - name: Check for CrashLoopBackOff
    shell: kubectl get pods -A | grep CrashLoopBackOff

  - name: Check latency spike via metrics
    shell: ./check_latency.sh
    register: latency_result

  - name: Fail if latency too high
    fail:
      msg: "Health check failed — latency is {{ latency_result.stdout }}ms (threshold: 200ms)"
    when: latency_result.stdout > 200
    # If this task fails, the GitHub Actions `if: failure()` step triggers rollback

Infra Patching/patching/rollback_nodegroup.yml:

tasks:
  - name: Restore previous node image
    shell: ./restore_previous_image.sh {{ region }}
    # Re-points ASG launch template to the pre-patch AMI

  - name: Validate restoration
    shell: kubectl get nodes

End-to-end patching flow:

CVE-2026-XXXX detected
  → Operator triggers patching.yml with cve_id=CVE-2026-XXXX patch_type=node
  → Manual approval required (GitHub Environment)
  → AP-South-1:
      build_new_node_image.sh  (bake patched AMI)
      update_nodegroup.sh      (update ASG launch template)
      kubectl drain node-1     (move pods to other nodes)
      delete_old_node.sh       (terminate old EC2)
      validate_cluster.yml     (kubectl + latency check)
      IF FAIL → rollback_nodegroup.yml (restore old AMI)
  → EU-Central-1 (only if AP passed):
      same steps
  → US-East-1 (only if EU passed):
      same steps

10. HealthCorp Non-Prod Scheduling Scripts

From AWS Cost Engineering/Env-StartStop/ — the real scripts used for the HealthCorp healthcare client:

Lambda version (Python):

# stop_lambda.py
import boto3

def lambda_handler(event, context):
    ec2 = boto3.client("ec2")
    response = ec2.describe_instances(Filters=[
        {"Name": "tag:Environment", "Values": ["nonprod"]},
        {"Name": "instance-state-name", "Values": ["running"]}
    ])
    instances = [i["InstanceId"]
                 for r in response["Reservations"]
                 for i in r["Instances"]]
    if not instances:
        return "No NON-PROD instances to stop"
    ec2.stop_instances(InstanceIds=instances)
    return f"Stopped: {instances}"

# start_lambda.py — identical, filter: "stopped" → ec2.start_instances()

Bash version (Jenkins-hosted, HealthCorp-specific):

#!/bin/bash
REGION="us-east-1"
PROFILE="healthcorp"

# Primary tag key
INSTANCE_IDS=$(aws ec2 describe-instances \
  --filters "Name=tag:Environment,Values=nonprod" \
            "Name=instance-state-name,Values=running" \
  --query "Reservations[*].Instances[*].InstanceId" \
  --output text --region $REGION --profile $PROFILE)

# Fallback tag key (inconsistent tagging in HealthCorp's estate)
if [ -z "$INSTANCE_IDS" ]; then
  INSTANCE_IDS=$(aws ec2 describe-instances \
    --filters "Name=tag:env,Values=nonprod" \
              "Name=instance-state-name,Values=running" \
    --query "Reservations[*].Instances[*].InstanceId" \
    --output text --region $REGION --profile $PROFILE)
fi

[ -z "$INSTANCE_IDS" ] && { echo "No running NON-PROD instances."; exit 0; }

aws ec2 stop-instances --instance-ids $INSTANCE_IDS \
  --region $REGION --profile $PROFILE

The dual tag-key fallback (tag:Environment then tag:env) is a real-world lesson — HealthCorp’s estate had inconsistent tagging from different teams who used different capitalisation. The script handles both without failing.

11. Architecture & Workflow Analysis

Full Traffic Flow with Real Config

Client Request
  │
  ▼ Kong Gateway (from gateway.yml)
    KongPlugin: rl-seller-base → 10 req/sec/IP (local)
    KongPlugin: rl-api-*       → 12–200 req/min/user (Redis)
    KongPlugin: request-transformer → rewrite /api/seller/legacy/* to /seller/*
    HTTPRoute: /seller → ClusterIP Service marketplace-seller:3000
    X-Trace-ID header injected
  │
  ▼ Istio Service Mesh (mTLS + trace propagation)
  ▼ Deployment marketplace-seller
    Node.js + OTel auto-instrument (-r flag) → traces to otel.prod.REGION.driffle.com
    @driffle/log-tastic → structured JSON logs
    Reads: K8s Secret marketplace-seller (mounted as env vars via secretRef)
    HPA: min 2, max 4 pods (75% CPU + memory threshold)
  │
  ▼ Downstream (mTLS)
  ▼ Kafka (payment-events) → consumers in AP-South-1 only

CI/CD Sequential Deploy

Code → PR → quality.yml (SonarQube gate) → merge

Release (pre-prod.yml):
  CI: npm test + lint + CodeQL SAST
  Build: docker build + push + Trivy scan (CRITICAL/HIGH → fail)
  Deploy-AP: kubectl set image → rollout status → validate_slo.sh
  Deploy-EU: (needs: Deploy-AP) same
  Deploy-US: (needs: Deploy-EU) + synthetic_test.sh
  Finalize: tag v{VERSION}-prod

Rollback (rollback.yml):
  matrix [AP, EU, US] parallel
  envsubst old image into manifests → kubectl apply
  tag v{VERSION}-rollback → Slack notification with Reason

12. Key Concepts Table

Concept	Real implementation	Why
Sequential multi-region deploy	`needs: Deploy-AP` chain	Bad release stops at first region
Canary via Istio	`kubectl apply -f istio/canary-10.yaml` → validate → `canary-100.yaml`	True L7 traffic split; no duplicate Deployments
`envsubst` templating	`envsubst < .k8s/prod/server.yml > server.yml`	Lightweight; same template for all regions via `${REGION}`
Rollback by re-render	Same `envsubst` with old `$VERSION`	Same pipeline path; cross-region coordinated; auditable
Consumers only in AP	`if: matrix.REGION == 'ap-south-1'`	One authoritative Kafka consumer; no duplicate processing
Workload Identity Federation	`google-github-actions/auth` with OIDC	No long-lived IAM keys
`maxUnavailable: 0` + `maxSurge: 100%`	All Deployment specs	Surge-first; zero-downtime rolling update
`terminationGracePeriodSeconds: 15`	All Deployment specs	In-flight requests drain on SIGTERM before SIGKILL
OTel auto-instrumentation	`node -r @aspecto/opentelemetry/auto-instrument`	Zero application code changes needed for distributed tracing
Per-endpoint Kong rate limits	12/min bulk, 200/min standard, 10/sec base	Protects expensive operations without over-restricting normal use
Redis-backed rate limiting	`policy: redis` in KongPlugin	Distributed — all Kong pods share state; consistent across replicas
Ansible idempotency	check-if-exists → create-if-not → attach	Running twice safe; no duplicate users or policies
Sequential patching	`needs:` chain AP → EU → US	Bad patch stops at AP; EU/US never get patched
Automatic patch rollback	`if: failure()` in patching job	No manual intervention needed; immediate recovery
`rollme: {{ randAlphaNum 5 }}`	Helm deployment annotation	Forces pod restart on every upgrade even if image unchanged

13. Tools & Technologies

Tool	Real usage
GitHub Actions	All 10 workflows; matrix strategy; workflow_call; concurrency groups
Workload Identity Federation	`google-github-actions/auth@v2` — keyless GCP auth
GKE	`get-gke-credentials@v2` + `kubectl` — all deploys
Istio VirtualService	`canary-10.yaml` / `canary-100.yaml` — canary traffic splitting
Kong	`KongPlugin` + `HTTPRoute` — gateway config; rate limiting; request transformation
Prometheus	Queried directly in `validate_slo.sh` for error rate + P95 latency
Docker Buildx	`build-push-action` with GHA cache (`type=gha`)
Trivy	`aquasecurity/trivy-action` — CRITICAL/HIGH exit-code: 1
CodeQL	`github/codeql-action` — JavaScript SAST
SonarQube	`sonarsource/sonarqube-scan-action` — PR quality gate
OpenTelemetry	`@aspecto/opentelemetry/auto-instrument` — auto-instrumented at Node.js startup
Slack	`rtCamp/action-slack-notify` — failure alerts + completion notifications
Ansible	10+ playbooks — AWS/GCP/Jenkins/Datadog access provisioning + node patching
Redis	Kong distributed rate limiting state across Kong pod replicas
HPA (autoscaling/v2)	Dual-metric (CPU + memory), min 2 / max 4
envsubst	Manifest templating via shell variable substitution
rickstaa/action-create-tag	Automated Git tagging post-deploy and post-rollback

14. Interview Preparation

Q1. Walk through how TitanGrid deploys a new version to production. CI runs tests, lint, and CodeQL SAST. Build produces a Docker image, tags it with both the version and the git SHA, and runs Trivy for vulnerability scanning (exit-code 1 on CRITICAL/HIGH). Deploy is sequential: AP-South-1 first — kubectl set image, rollout status, SLO validation via Prometheus queries (error rate < 2%, P95 < 500ms), and a synthetic transaction test. If AP passes, EU deploys (same steps). EU passing unlocks US. If any SLO check fails, an automatic rollback is triggered for that region and the pipeline stops — EU and US are never touched. On full success, a git tag v{VERSION}-prod is created.

Q2. How does the canary deployment work and why is it done via Istio rather than a separate Deployment? When a new image is deployed, istio/canary-10.yaml is applied — this is an Istio VirtualService that routes 10% of traffic to the new pod version and 90% to the old. After a 3-minute observation window and SLO validation, istio/canary-100.yaml shifts 100% to the new version. Using Istio means both versions share the same Kubernetes Deployment resource — the HPA continues to work normally, there are no extra Deployments to clean up, and traffic splitting is at L7 (application-layer) not L4. A separate canary Deployment would require duplicating pod specs, managing two HPAs, and explicitly cleaning up after promotion.

Q3. How does rollback work, and why not kubectl rollout undo? rollback.yml re-renders the target version’s Kubernetes manifests using envsubst with the old image tag, then applies them via kubectl apply — exactly the same pipeline as a forward deployment. It runs as a matrix across all 3 regions in parallel (if: always() so all regions roll back even if one fails). kubectl rollout undo can only roll back one revision, has no cross-region coordination, bypasses the manifest pipeline, produces no Slack notification with a reason, and creates no git tag. The re-render approach means every rollback is auditable, tagged as v{VERSION}-rollback, Slack-notified with an explicit Reason field.

Q4. How does Kong rate limiting work and how is it different per endpoint? Each endpoint has its own KongPlugin Kubernetes resource defining its rate limit independently. Bulk operations (e.g., bulk price updates) are capped at 12 requests/minute per authenticated user — they’re expensive server-side. Standard operations get 200 requests/minute. The base catch-all rule gives 10 requests/second per IP. All per-user limits use limit_by: header, header_name: Authorization — counting per authenticated session, not per IP. For distributed Kong deployments (multiple Kong pods), policy: redis shares the rate limit counter via a Redis cluster, so a user can’t bypass limits by hitting a different Kong pod.

Q5. How does the access automation work for onboarding a new engineer? An Ansible playbook is triggered with the target user_email and resource type. It authenticates using AWS STS temporary credentials (not permanent keys), checks whether the IAM user already exists (aws iam get-user), creates them if not, then attaches the pre-defined least-privilege policy for that resource. The same idempotent pattern covers EC2, EKS, RDS, S3, Lambda, ECS, SQS on AWS, and GKE, GCS, Cloud Run, Cloud SQL, Compute Engine, Pub/Sub, Cloud Functions on GCP. Running the playbook twice produces the same result as running it once — safe for retries.

15. Exam & Certification Notes

GitHub Actions patterns tested:

strategy.matrix — parallel jobs per matrix value
needs: — sequential dependency; job waits for all needs to succeed
if: always() — run even if dependency failed (rollback/notify jobs)
workflow_call vs workflow_dispatch — reusable vs manual trigger
concurrency: cancel-in-progress: false — never cancel a running deploy
environment: — links to GitHub Environment protection rules (approval gates)

Kubernetes patterns tested:

maxUnavailable: 0 + maxSurge: 100% — surge-first rolling update
revisionHistoryLimit — controls old ReplicaSet retention
autoscaling/v2 HPA with multiple metrics
envFrom: secretRef — all keys from a Secret as env vars
terminationGracePeriodSeconds — drain window after SIGTERM
if: matrix.REGION == 'ap-south-1' pattern for conditional job steps

DevSecOps pipeline gates (all three):

SAST (CodeQL/SonarQube) — static, runs before build
Container scanning (Trivy) — after build, before deploy
DAST (ZAP) — against running staging environment

Common exam misconceptions:

“Rollback with kubectl rollout undo is best practice” → acceptable but not production-grade for multi-region coordinated rollbacks
”HPA needs only one metric” → autoscaling/v2 supports multiple; common pattern is CPU + memory
”STS credentials are permanent” → temporary by definition; expire in hours; best practice for Ansible automation

16. Cheat Sheet

Deploy pipeline (sequential):

CI (test+lint+CodeQL) → Build (docker+Trivy) → Deploy-AP (set image→SLO) → Deploy-EU → Deploy-US → tag

SLO gates: error_rate < 2% (Prometheus) + P95 < 500ms (Prometheus) + CPU < 85% + mem < 85% + restarts < 3

Rollback (parallel matrix): envsubst old $VERSION → kubectl apply × 3 regions → tag v{VERSION}-rollback

Kong rate limits: bulk = 12/min/user (Redis) · standard = 200/min/user (Redis) · base = 10/sec/IP (local)

Patching (sequential): CVE → AP (AMI→drain→validate) → EU → US; if: failure() → rollback each region

Access automation: auth → check exists → create if not → attach policy (same pattern: AWS/GCP/Jenkins/Datadog)

K8s deploy config: maxUnavailable:0, maxSurge:100%, terminationGracePeriodSeconds:15, minReplicas:2, maxReplicas:4

Consumers: AP-South-1 only · no HPA · no CPU limit · no HTTP probes · 3 replicas · consumer.js entrypoint

Helm trick: rollme: {{ randAlphaNum 5 | quote }} in annotations forces pod restart on every upgrade

17. Gaps & What’s Still Coming

Files referenced in code but not in the zip:

istio/canary-10.yaml + istio/canary-100.yaml — referenced in Development.yml but not shipped. These are Istio VirtualService manifests with weight: 10 / weight: 100 splits.
scripts/rollback.sh — referenced in Development.yml canary failure handling. Likely a kubectl set image back to the previous tag.
ansible360/patching/inventory.yml — present but empty (placeholder); actual inventory not shipped.
./build_new_node_image.sh, ./update_nodegroup.sh, ./delete_old_node.sh