SRE Labs (Advanced Track) — Project Call 2: AWS Cost Optimization (HealthCorp) — Continued

Structured educational resource covering sre labs (advanced track) — project call 2: aws cost optimization (healthcorp) — continued.

senior 45 min read 62 sections

#kubernetes#cloud-k8s#aws#gcp

Complete Learning Package (EC2 Migration · EKS Migration · Spot · Savings Plans · EKS Security Scanning)

Source: 2026-01-31-19-05-34.md — SRT transcript of the second Sunday project call. This session continues directly from Project Call 1 (see the companion document). The same HealthCorp healthcare client ($98K/month AWS bill). A guest presenter (Ravi) joins for the EKS security scanning section.

Companion reading: This document assumes familiarity with Call 1 content: the HealthCorp client background, the layered optimization framework, tagging strategy, native AWS tools (Cost Explorer, Compute Optimizer, Trusted Advisor), standalone EC2 Intel→AMD migration, and architecture families (Intel/AMD/ARM). Read Call 1’s learning package first.

2. Table of Contents

Executive Summary
Table of Contents
Detailed Structured Notes
- 3.1 Session Recap — What Was Covered in Call 1
- 3.2 ASG Instance Migration — Intel → AMD (Live Demo)
- 3.3 EKS Node Group Migration — Intel → AMD (Live Demo)
- 3.4 AMD → ARM Migration — Concepts and Assessment Workflow
- 3.5 Assignment — AMD → ARM Hands-On
- 3.6 Non-Prod Environment Shutdown
- 3.7 Right-Sizing EC2 Instances — the PRC Framework
- 3.8 On-Demand → Spot Migration
- 3.9 Spot in Production — Architecture Patterns
- 3.10 Savings Plans — Overview and Buying Strategy
- 3.11 EBS Volume Optimization (Recap)
- 3.12 EKS Security Scanning — Tool Overview
- 3.13 Tool 1 — Kube-hunter (Penetration Testing)
- 3.14 Tool 2 — Kubescape (Misconfiguration and Compliance)
- 3.15 Tool 3 — Kube-bench (CIS Benchmarking)
- 3.16 Running All Three as CronJobs
- 3.17 Q&A — Key Points Captured
Key Concepts Table
Architecture & Workflow Analysis
Commands, Scripts & Configs
Tools & Technologies
Real-World Production Usage
Interview Preparation (Beginner / Intermediate / Advanced)
Exam & Certification Notes
Cheat Sheet
Gaps, Assumptions & Incomplete Areas
Gap-Fill — What the Session Left Unfinished, Completed Here

3. Detailed Structured Notes

3.1 Session Recap — What Was Covered in Call 1

Quick orientation for reference:

HealthCorp client: ~$98K/month AWS bill, single AWS account, ~170 EC2 instances (mostly Intel), EKS, RDS, S3, CloudWatch, NAT Gateway, etc.
Optimization framework: 5 layers — Compute → Network → Storage/Data → Monitoring/Logging → Other tools.
EC2 architecture families: Intel (no suffix, most expensive) → AMD (a suffix, ~15–20% cheaper) → ARM/Graviton (g suffix, ~30–40% cheaper).
Three EC2 instance categories for migration: Standalone, ASG-managed, EKS node group.
Call 1 completed: standalone Intel→AMD migration (AMI backup → stop → change instance type → start → monitor → verify in Cost Explorer → delete AMI).

3.2 ASG Instance Migration — Intel → AMD (Live Demo)

Context

Auto Scaling Groups (ASGs) do not use individual instance configurations — they delegate all instance spec to a Launch Template (LT). To change the instance type for an ASG, you create a new LT version and trigger an Instance Refresh.

Step-by-Step (Live Demo: `sre-labs-asg-demo`, T3.medium → T3a.medium)

Step 1 — Identify the current Launch Template

Navigate to: EC2 → Auto Scaling Groups → select your ASG → Details tab.
Find the attached Launch Template name and current version (e.g., Intel-LT, version 1, instance type t3.medium).

Step 2 — Create a new LT version

EC2 → Launch Templates → select Intel-LT → Actions → Modify template (Create new version).
Description: "Intel to AMD LT version".
Source template: version 1 (base).
Change only the instance type: t3.medium → t3a.medium.
Leave all other settings identical (VPC, security groups, IAM, EBS, etc.).
Click Create template version.
Result: new version created (e.g., version 3), with description and instance type visible for verification.

Step 3 — Update the ASG to use the new LT version

EC2 → Auto Scaling Groups → select ASG → Edit → Launch Template section.
Select the new version (version 3, visible via description and instance type).
Click Update. The ASG configuration is updated but existing instances are still on Intel — no instances have been replaced yet.

Step 4 — Trigger Instance Refresh

Inside the ASG → Instance Refresh tab → Start instance refresh.
Key settings:
- Replace behavior: “Launch before terminate” (for zero-downtime production) OR “Terminate and launch” (simultaneous — used here to avoid paying for both old and new instances during migration; keeps cost down during cost engineering engagement).
- Min healthy percentage: ≥ 80% (below 80% is not production-grade).
- Instance warmup time: at least 5 minutes — ensures health checks pass before the old instance is terminated.
Click Start instance refresh.
What happens: ASG launches new AMD instances → health checks pass → old Intel instances are terminated. Duration: ~5 minutes.

Step 5 — Verify

ASG → Instances tab: verify all instances now show T3a.medium.
Cost Explorer → EC2 → filter by instance ID → daily view: cost drop visible within 24 hours.
Example: $20/day (Intel) → ~$16–17/day (AMD) = ~$3–4/day saving = ~$100–130/month per instance.

Rollback

Old LT version (version 1, Intel) is never deleted — it acts as the rollback plan.
To roll back: update ASG to use LT version 1 → start another instance refresh.

Observability window

Monitor for minimum 72 hours after migration.
If any issues: trigger rollback via LT version 1 instance refresh.

3.3 EKS Node Group Migration — Intel → AMD (Live Demo)

Why this category is the most complex (Intel→AMD)

EKS node groups are tightly integrated with the Kubernetes scheduler. You cannot simply change the instance type in-place. You must: create a parallel AMD node group, transfer workloads from Intel to AMD nodes using drain/cordon, and only then decommission the old node group. This requires 2–3 minutes of downtime because the scheduler must re-place pods.

Prerequisites before starting

Understand application nature: Is the workload stateless or stateful?
- Stateless: safe to migrate. Pods can be terminated and re-created anywhere.
- Stateful: significantly harder. Persistent volume claims (PVCs) may need to be re-attached. Approach with caution; consider not migrating stateful workloads unless necessary.
Document the node group configuration: Inspect labels and taints on the Intel node group. The new AMD node group must have identical labels and taints so the scheduler routes pods to it correctly.
Check subnets: Note which subnets are attached to the Intel node group. The AMD node group must use the same subnets.
Check the IAM node role: Each EKS cluster has a specific node role (IAM). Use the same role for the new node group.

Step-by-Step (Live Demo: `sre-labs-eks`, node group `ng-intel`, T3.medium → T3a.medium)

Phase 1 — Create new AMD Launch Template version (in the ASG attached to the EKS node group)

Find the ASG managing the EKS node group (visible in the node group console).
Launch Templates → select the LT → create new version.
- Description: "EKS node group AMD LT version".
- Change instance type to t3a.medium.
- Critical: In Advanced Details → IAM instance profile → select “Do not include IAM instance profile”.
  - Why: When you create a node group with eksctl or the AWS CLI, the command automatically creates and attaches the instance profile from the node role. If you also specify one in the LT, AWS throws an error: you cannot have duplicate instance profile assignments.
- Update user data only if your bootstrap script has architecture-specific content (rare).
- Create the version (e.g., version 4 with t3a.medium and the correct description).

Phase 2 — Get the subnets attached to the Intel node group

aws eks describe-nodegroup \
  --cluster-name sre-labs-eks \
  --nodegroup-name ng-intel \
  --region ap-south-1 \
  --query 'nodegroup.subnets' \
  --output text
# Output: subnet-xxxxx subnet-yyyyy subnet-zzzzz

Phase 3 — Get the ASG scaling configuration

In the ASG console: note desired = 2, min = 2, max = 4. Use the same values for the new node group.

Phase 4 — Create the new AMD node group (CLI)

aws eks create-nodegroup \
  --cluster-name sre-labs-eks \
  --nodegroup-name ng-amd \
  --scaling-config minSize=2,maxSize=4,desiredSize=2 \
  --subnets subnet-xxxxx subnet-yyyyy subnet-zzzzz \
  --launch-template name=<LT-name-ending-in-86>,version=4 \
  --node-role arn:aws:iam::<account-id>:role/<eks-node-role> \
  --region ap-south-1

The new ng-amd node group appears in the EKS console in “Creating” state.
Do not update the ASG manually — the eksctl/CLI command handles the node group + ASG relationship.

Phase 5 — Copy labels and taints to the new node group

Once ng-amd is in “Active” state:

Check the Intel node group’s labels and taints (from console or CLI).
Apply the same labels and taints to the new AMD node group (via console UI or CLI).

Why this matters: Kubernetes scheduling is based on node labels and taints. Pods with nodeSelector or nodeAffinity targeting the Intel node group’s labels will only be rescheduled onto the AMD node group if it has matching labels. If labels differ, pods will remain in Pending state.

Phase 6 — Drain the Intel nodes (move all pods off them)

# Get the internal IPs of the Intel nodes:
kubectl get nodes -o wide | grep ng-intel

# Drain each Intel node:
kubectl drain <node-name-or-ip> \
  --ignore-daemonsets \
  --delete-emptydir-data

--ignore-daemonsets: DaemonSet pods (monitoring agents, CNI plugins) are intentionally not drained — they will be terminated and re-created when the node is deleted.
--delete-emptydir-data: removes pods using emptyDir volumes (ephemeral storage). Only use this if workloads are truly stateless.
Effect: all non-DaemonSet pods are evicted from the Intel node. The scheduler places them on the AMD node group (which has matching labels/taints and is Ready).

Phase 7 — Cordon the Intel nodes (prevent new scheduling)

kubectl cordon <node-name-or-ip>

Marks the Intel nodes as SchedulingDisabled — the scheduler will never place new pods here.
Combined with draining, the Intel nodes are now fully idle.

Phase 8 — Verify workloads on AMD nodes

kubectl get pods -A -o wide | grep ng-amd
# All pods should now show AMD node IPs as their NODE

Phase 9 — Observe for 72 hours then delete the Intel node group

Monitor application behaviour, error rates, CloudWatch metrics for 72 hours.
If stable: delete the Intel node group from the EKS console or via CLI.

Rollback (within the 72-hour window)

Drain all pods from AMD node group nodes.
Cordon AMD nodes.
Un-cordon the Intel nodes: kubectl uncordon <intel-node-name> — marks them fit for scheduling again.
Scheduler automatically places evicted pods back onto Intel nodes.
Downtime: ~2–3 minutes.

3.4 AMD → ARM Migration — Concepts and Assessment Workflow

Why AMD→ARM is architecturally different

Intel and AMD are both x86_64 (same ISA, different manufacturers). Application binaries compiled for Intel run on AMD without modification.
ARM/Graviton is a different instruction set architecture (AArch64). Application code compiled for x86_64 will not run on ARM without recompilation or a compatibility shim.
Any third-party library with native compiled binaries (.so, .dll, JNI, Python C-extensions, etc.) must also have an ARM-compatible version.

Migration complexity matrix (revisited)

Category	Intel → AMD	AMD → ARM
Standalone	Simplest (change instance type)	Hardest (must provision ARM instance from scratch; migrate data/app manually)
ASG	Medium (new LT version + instance refresh)	Medium (new LT with ARM AMI + instance refresh)
EKS node group	Complex (parallel node group + drain/cordon)	Simplest (update node group AMI; EKS handles rolling replacement)

The inversion: ARM migration is easiest for EKS because Kubernetes already knows how to rolling-update node groups (same mechanism used for cluster version upgrades). For standalone instances, there is no equivalent — you must launch a brand-new instance, migrate the application data and configuration manually, test, then decommission the old one.

Blast radius consideration

Intel → AMD: Small blast radius. Low risk, same ISA.
Intel → ARM directly: Large blast radius. High risk. Only acceptable if developers have pre-validated full compatibility.
Recommended path: Intel → AMD first (validate) → AMD → ARM second (validate again). Two controlled steps reduce total risk.

AMD → ARM Assessment Workflow

Step 1 — Developer sign-off (always first)

Call a meeting with the application team. Ask explicitly:

Does this application use any architecture-specific code paths (x86 SIMD, SSE, AVX instructions)?
Does it use native binaries (JNI, Python C-extensions, Rust FFI, etc.)?
Has it ever been tested on ARM?

If the developer confirms no architectural dependencies → proceed to Step 2.

Step 2 — AWS Porting Advisor for Graviton

Installation (two options):

# Option 1: Python package
pip install aws-graviton-porting-advisor
porting-advisor --output text /path/to/application/repo

# Option 2: Docker image
docker run --rm \
  -v /path/to/application:/app \
  public.ecr.aws/graviton-porting-advisor/porting-advisor:latest \
  --output html /app
# Generates HTML report with clickable findings

What it scans: dependency manifests (package.json, requirements.txt, pom.xml, go.mod, Gemfile) and C/C++ source files for preprocessor directives targeting x86 architectures.

Output example (from the session):

Line 473: preprocessor error — x86-specific macro detected
Third-party package: <library-name> — no ARM wheel available

Limitations stated by instructor: The tool identifies dependencies but cannot verify runtime behaviour. It is a starting point, not a guarantee. The developer must still confirm the finding is real and fix it.

Step 3 — Fix identified issues

Replace non-ARM-compatible libraries with ARM-compatible versions.
For Python: install ARM wheels (pip install --platform linux_aarch64 ...).
For Java: recompile native JNI dependencies for AArch64.
For Node.js: most pure-JS packages work; native addons (node-gyp) need recompilation.

Step 4 — Test on an ARM instance in non-prod

Launch a Graviton instance (e.g., t4g.medium) in a non-prod environment.
Deploy and exercise the application. Run your full test suite.
Only after clean test results should you proceed to production migration.

3.5 Assignment — AMD → ARM Hands-On

Task assigned to participants:

Starting from the AMD ASG instance just migrated in the live demo:

Create the infrastructure (ASG or standalone EC2 on AMD).
Deploy a basic application (Nginx or any simple app).
Migrate the instance/ASG from AMD to ARM.
Record the migration and share with the group.

Each participant should independently perform this across all three categories:

Standalone AMD → ARM
ASG AMD → ARM
EKS node group AMD → ARM

3.6 Non-Prod Environment Shutdown

The Problem

Non-production workloads (dev, staging, QA) run 24×7 even though the engineering team only uses them during working hours. This is wasted spend.

HealthCorp result: ~$900/month saved from non-prod scheduling alone (their dev team worked round the clock, so scope was limited compared to standard orgs).

The Solution

Tag-based automated shutdown via Python scripts hosted on:

Option A: AWS Lambda + EventBridge — fully managed; small additional Lambda/EventBridge cost.
Option B: Jenkins (or any CI/CD) with a cron scheduler — no additional cloud cost; preferred when you already have Jenkins.

Scheduling

Stop: 10:00 PM IST (22:00 IST)
Start: 8:00 AM IST (08:00 IST)
Running window: 14 hours/day → saves ~42% of compute cost on non-prod vs 24×7.

Tag Strategy

All non-prod instances must carry the tag ENV=non-prod (or ENV=dev, ENV=staging). Scripts filter by this tag. Production instances are never touched.

Scripts (uploaded to Google Drive)

Python — Start Lambda (`start_lambda.py`)

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')
    # Find all stopped instances tagged ENV=non-prod
    response = ec2.describe_instances(
        Filters=[
            {'Name': 'tag:ENV',               'Values': ['non-prod']},
            {'Name': 'instance-state-name',   'Values': ['stopped']}
        ]
    )
    instance_ids = [
        i['InstanceId']
        for r in response['Reservations']
        for i in r['Instances']
    ]
    if instance_ids:
        ec2.start_instances(InstanceIds=instance_ids)
        print(f"Started: {instance_ids}")
    return {'started': instance_ids}

Python — Stop Lambda (`stop_lambda.py`)

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2')
    # Find all running instances tagged ENV=non-prod
    response = ec2.describe_instances(
        Filters=[
            {'Name': 'tag:ENV',               'Values': ['non-prod']},
            {'Name': 'instance-state-name',   'Values': ['running']}
        ]
    )
    instance_ids = [
        i['InstanceId']
        for r in response['Reservations']
        for i in r['Instances']
    ]
    if instance_ids:
        ec2.stop_instances(InstanceIds=instance_ids)
        print(f"Stopped: {instance_ids}")
    return {'stopped': instance_ids}

Bash equivalent (for Jenkins hosting)

#!/bin/bash
# Usage: ./nonprod_stop.sh  or  ./nonprod_start.sh
ACTION=$1   # "stop" or "start"
TAG_KEY="ENV"
TAG_VALUE="non-prod"

INSTANCE_IDS=$(aws ec2 describe-instances \
  --filters "Name=tag:${TAG_KEY},Values=${TAG_VALUE}" \
            "Name=instance-state-name,Values=$([ "$ACTION" = "stop" ] && echo running || echo stopped)" \
  --query 'Reservations[*].Instances[*].InstanceId' \
  --output text)

if [ -n "$INSTANCE_IDS" ]; then
  aws ec2 ${ACTION}-instances --instance-ids $INSTANCE_IDS
  echo "${ACTION^}ped: $INSTANCE_IDS"
fi

Note: Lambda does not natively support Bash scripts — use Python for Lambda. Both Python and Bash can run on Jenkins.

EventBridge Rules (when using Lambda)

# Stop rule (10 PM IST = 4:30 PM UTC):
cron(30 16 * * ? *)

# Start rule (8 AM IST = 2:30 AM UTC):
cron(30 2 * * ? *)

3.7 Right-Sizing EC2 Instances — the PRC Framework

The Three Pillars: PRC

Any right-sizing decision must balance all three — not just cost:

Pillar	Meaning	Risk of ignoring
P — Performance	Application must meet latency/throughput SLOs under expected load	Outage if under-provisioned
R — Reliability	Infrastructure must handle traffic spikes and failure modes	Production incidents from capacity starvation
C — Cost	Minimize spend without compromising P and R	Over-spend; budget overrun

Rule: Never optimize Cost at the expense of P or R. The goal is to find the minimum instance size that still satisfies P and R with a safety buffer.

How to Right-Size

Step 1 — Collect metrics over a baseline period

Minimum baseline: 90 days. Longer is better for workloads with seasonal variation (e.g., e-commerce, healthcare with reporting cycles).

Metrics to gather per instance:

CPU utilization (average, P95, P99)
Memory utilization (requires CloudWatch agent or SSM agent for RAM metrics)
Disk IOPS (if storage-intensive)
Network throughput (in/out)

Step 2 — Use Compute Optimizer

AWS Console → Compute Optimizer (must be enabled; takes up to 24 hours after enablement to show data).
Shows: current utilization graph + recommended instance type + projected savings.
Example from session (GCP Active Assist equivalent): downsize from 4 vCPU/32 GB → 2 vCPU/16 GB; saves $37/month on that instance.
GCP equivalent: Active Assist (same concept, different UI).

Step 3 — Apply the 20–33% buffer rule

Never size to the average. Always size to the P99 peak + 20–33% headroom.
Example: P99 CPU = 60% → size for 80–90% capacity ceiling → choose the instance size where 60% CPU ≈ your max expected load × 0.75.
This buffer absorbs unexpected traffic spikes without causing degradation.

Step 4 — Validate before and after

Before: document current performance metrics.
After: run for 72 hours minimum; watch for latency increase, error rate increase, or OOM events.
If metrics degrade → increase instance size (not necessarily back to original — may just need one size up).

When NOT to Right-Size

Situation	Reason
Uncertain or rapidly growing traffic	Headroom assumption becomes invalid too fast
Active cloud migration in progress	Different cloud providers have different vCPU/RAM performance ratios; right-sizing during migration compounds risk (real example: L&T migration from GCP to AWS)
After a major feature release	New feature may change traffic patterns significantly
Within 30 days of initial provisioning	Not enough baseline data

Tools Available

AWS Compute Optimizer — primary; ML-based recommendations.
AWS Trusted Advisor — lists low-utilization instances; less granular than Compute Optimizer.
Third-party: Spot.io, StormForge, CastAI — automated continuous right-sizing with more granular controls.

3.8 On-Demand → Spot Migration

What Spot Instances Are

Spare AWS capacity sold at up to 70–90% discount vs On-Demand, but AWS can reclaim them with 2-minute notice when capacity is needed elsewhere.

Demo — Launching an instance as Spot

Take an AMI backup of the current On-Demand instance (same process as standalone migration).
EC2 → AMIs → select the backup → Launch Instance from AMI.
In Advanced Details → Purchasing option → change from None to Spot.
All other settings remain identical.
Launch.
The instance’s Lifecycle column in the EC2 console changes from normal to spot.

Cost verification: AWS Cost Calculator — create identical specs for On-Demand vs Spot → observe cost difference.

3.9 Spot in Production — Architecture Patterns

This was the most debated topic of the Q&A. Key points consolidated:

The Myth

”Spot is only for dev/test.” This is outdated. Pinterest, Slack, and Netflix run significant production workloads on Spot.

The Reality

Spot is safe in production if your architecture handles interruption gracefully. Bad architecture + Spot = catastrophic failure. Good architecture + Spot = major savings with acceptable reliability.

When Spot is Appropriate

Use case	Notes
Dev / QA / staging	No question — use Spot without any special architecture
Stateless microservices	Safe with ≥2 replicas, HPA, and PDB configured
Event consumers / message queue workers	Interruption just means a message goes back to the queue
Batch processing / ETL jobs	Interruption is handled by job retry mechanisms
CI/CD build agents (GitHub Actions, Jenkins)	Build retried on interruption; no state needed
Data processing / Spark / EMR	Native Spot support built in

When Spot Is Risky / Inappropriate

Situation	Reason
Stateful workloads (databases, message brokers, ZooKeeper, Kafka with local storage)	State is lost on interruption; requires complex recovery
Single-replica critical services	Interruption = 100% downtime
Workloads without retry mechanisms	Interruption means lost work

Production Spot Architecture Pattern: 70% Spot / 30% On-Demand Mix

EKS Cluster
├── Node Group A: 70% Spot (multiple instance types for capacity diversity)
│   └── t3a.medium, t3.medium, t2.medium (fallback chain)
│       If t3a.medium unavailable → try t3.medium → try t2.medium
│
└── Node Group B: 30% On-Demand (stable baseline capacity)
    └── t3a.medium

Why multiple instance types in Spot node pool: Spot availability varies by instance type and AZ. If your Spot node group only uses t3a.medium and AWS reclaims all of that capacity type in your AZ, you have no Spot nodes. Diversifying across 3–5 compatible instance types dramatically reduces the probability of simultaneous reclamation.

Required Kubernetes Safeguards for Production Spot

Safeguard	What it does	Why it’s needed
HPA (Horizontal Pod Autoscaler)	Scales pod count based on CPU/memory	Maintains capacity when Spot node is interrupted
PDB (PodDisruptionBudget)	Limits how many pods of a workload can be disrupted simultaneously	Prevents drain from removing all replicas at once
≥ 2 replicas per workload	Ensures at least one pod survives a node interruption	Single replica + Spot = guaranteed downtime on interruption
Graceful shutdown handling	Application catches SIGTERM and completes in-flight requests	Avoids dropped requests during the 2-minute Spot reclamation window
Instance type diversification	Multiple instance types in the Spot node pool	Avoids simultaneous reclamation of all Spot capacity
Multi-AZ deployment	Pods spread across AZs	AZ-level Spot capacity issues don’t take down all pods

Node-Level vs Application-Level Division

Attendee confusion addressed in Q&A: The 70/30 split is on nodes, not on applications. You don’t assign specific microservices to Spot or On-Demand. The scheduler places pods on any Ready node. The split ensures the cluster has mixed capacity types, and if Spot nodes are interrupted, the pods reschedule onto On-Demand nodes automatically.

3.10 Savings Plans — Overview and Buying Strategy

This was covered at a high level (detailed mathematics are in a separate 30-page document + recording):

Two Types (Recap from Call 1, with additions)

Type	Scope	Flexibility	Typical discount
Compute Savings Plan	EC2 (any family, region, OS) + Lambda + Fargate/ECS	Highest	~17–66% vs On-Demand
EC2 Instance Savings Plan	Specific EC2 family in specific region	Lower	~20–72% vs On-Demand

Rule: Compute Savings Plans cover Lambda and ECS Fargate; EC2-only plans do not. If your stack includes Lambda or Fargate, start with Compute Savings Plans.

Neither type applies to Spot. Savings Plans only reduce On-Demand billing.

Buying Strategy Principle

Spot handles your volatile, interruptible workloads cheaply.
Savings Plans handle your stable, always-on On-Demand workloads cheaply.
They are complementary, not alternatives.

Correct approach: commit only the $/hour that will definitely be consumed On-Demand 24×7. Leave room for growth. Do not buy to cover Spot instances.

Common Mistake

Most people choose the wrong plan type (EC2-only when they have Lambda/Fargate, or over-commit the dollar amount). This results in paying more than necessary. Use the AWS Savings Plans Calculator and the detailed documentation before purchasing.

3.11 EBS Volume Optimization (Quick Recap)

Mentioned as “pretty straightforward” — not a new demo in this session:

Delete idle/unattached volumes: aws ec2 describe-volumes --filters Name=status,Values=available → delete listed volumes.
GP2 → GP3 migration: cheaper per GB and better baseline IOPS. Done in-place in the console; no downtime.
Volume shrinking: Not directly supported. Procedure: detach volume → snapshot → create smaller volume from snapshot → attach. Rarely done without explicit business sign-off.

3.12 EKS Security Scanning — Tool Overview

Presenter: Guest presenter (Ravi), a senior DevOps/SRE practitioner.

Why scan before optimizing EKS costs? Before right-sizing EKS node groups or adding cost-optimization tooling (Karpenter, CastAI), ensure the cluster’s security posture is known. A misconfigured cluster has broader attack surface; fixing misconfigurations before scaling-related changes avoids compounding risk.

Industry context: In finance and healthcare domains, a clean security report is a hard prerequisite for deploying to production at many organizations.

Three Tools Covered

Tool	Primary purpose	Output
Kube-hunter	Penetration testing — simulates an attacker with pod access	Vulnerabilities an attacker can exploit
Kubescape	Comprehensive misconfiguration + compliance scanning	Detailed findings per pod/namespace + remediation links
Kube-bench	CIS benchmark auditing	Pass/Warn/Fail per CIS control

Deployment pattern: All three run as Kubernetes CronJobs. Logs are captured after execution. YAML files and log outputs uploaded to Google Drive for reference.

3.13 Tool 1 — Kube-hunter (Penetration Testing)

Purpose

Simulates an attacker who has gained access to a pod inside the cluster. Identifies what that attacker can do — what services, APIs, and credentials are accessible.

How It Works

Runs as a CronJob (e.g., daily at 2:00 AM).
The pod acts as the “attacker.”
It attempts to: spoof identities, access sensitive interfaces, access the API server, read credentials, and other known attack vectors.
Reports which vulnerabilities it successfully exploited.

Running a Manual Scan

# Trigger manual job from the CronJob definition:
kubectl create job cube-hunter-manual \
  --from=cronjob/cube-hunter

# Watch the pod:
kubectl get pods -w | grep cube-hunter

# View results when pod completes (~1–2 minutes):
kubectl logs <cube-hunter-pod-name>

Sample Output (from session)

Node detected: <internal-IP>
Services detected: <list>

Vulnerabilities found:
ID          | Location     | Category                      | Description
KHV00002    | Pod          | Spoofing                      | [description]
KHV00xxx    | Pod          | Accessing sensitive interfaces | [description]
KHV00xxx    | Pod          | Accessing API server          | [description]
KHV00xxx    | Pod          | Credentials exposed           | [description]

Remediation Process

Each vulnerability has an ID (e.g., KHV00002).
Search the Aqua Security site (Kube-hunter’s vendor) for the ID to get specific remediation steps.
Fix → re-run Kube-hunter → verify the vulnerability no longer appears.

3.14 Tool 2 — Kubescape (Misconfiguration and Compliance)

Purpose

The most comprehensive of the three tools. Scans the cluster (nodes, namespaces, pods, deployments, configurations) against security frameworks (NSA, MITRE ATT&CK, CIS, etc.) and produces detailed findings with remediation links.

How It Works

Runs as a CronJob.
Scans each pod across all namespaces.
For each pod: runs a set of control checks; reports pass/fail with evidence and remediation guidance.

Running a Manual Scan

kubectl create job kubescape-manual \
  --from=cronjob/kubescape

kubectl logs <kubescape-pod-name>

Sample Output (from session — pod named “web”)

Pod: web (default namespace)
Controls checked: 14
Controls failed:  8

Failed controls:
- CPU limits not set           → Risk: resource starvation
- Memory limits not set        → Risk: OOM kills affecting other pods
- Container hardening missing  → Risk: syscall attack surface
- Ingress/egress not blocked   → Risk: unrestricted network exposure
- Running as root              → Risk: container escape to host

Each finding links to a specific control ID on the Kubescape documentation site (armosec.io), which explains:

What the misconfiguration is
Why it is a security risk
Exact remediation steps

Common Findings and Fixes

Finding	Risk	Fix
No CPU limits	One pod can starve all others	Add `resources.limits.cpu` to pod spec
No memory limits	OOM kill cascades	Add `resources.limits.memory` to pod spec
Running as root	Container escape to host	Add `securityContext.runAsNonRoot: true` and `runAsUser: <non-zero>`
No network policy (ingress/egress)	Unrestricted pod communication	Add `NetworkPolicy` resources
Privilege escalation allowed	Container can gain more permissions	Add `allowPrivilegeEscalation: false`
Writable root filesystem	Malware persistence	Add `readOnlyRootFilesystem: true`

3.15 Tool 3 — Kube-bench (CIS Benchmarking)

Purpose

Auditing tool. Checks the cluster configuration against the CIS (Center for Internet Security) Kubernetes Benchmark — the industry-standard hardening guide for Kubernetes. Produces a simple Pass/Warn/Fail result per check.

How It Works

Runs as a CronJob.
Checks system-level configuration: Kubeconfig file permissions, kubelet settings, API server flags, etcd configuration, etc.
Output is structured: each check gets a status with a remediation suggestion for failures/warnings.

Running a Manual Scan

kubectl create job kube-bench-manual \
  --from=cronjob/kube-bench

kubectl logs <kube-bench-pod-name>

Sample Output (from session — training cluster)

Checks run:       13+
Checks passed:    13
Checks warned:    3
Checks failed:    0

PASS:
[1.1.1] Ensure kubeconfig file permissions are 644 or more restrictive
[1.1.2] Ensure kubelet is configured correctly
...

WARN:
[1.2.x] Hostname override argument is not set
  Remediation: Set --hostname-override flag to <value>

Interpretation from session: 13 passes, 3 warnings, 0 failures → “okay, but not best.” Warnings should be addressed over time; zero failures is a good baseline.

When auditors use this: Many organizations (especially fintech, healthcare) require a Kube-bench report as part of their quarterly or annual infrastructure audit. “Clean” typically means 0 failures and < 5 warnings.

3.16 Running All Three as CronJobs

Recommended Schedule

Tool	Schedule	Rationale
Kube-hunter	Daily at 2:00 AM	New attack vectors emerge; daily scan keeps posture current
Kubescape	Daily or weekly (off-hours)	More comprehensive; slightly longer runtime
Kube-bench	Weekly (Saturday 3:00 AM)	Configuration rarely changes; weekly is sufficient for most teams

CronJob pattern (shared across all three)

# Example: kube-hunter CronJob definition (abbreviated)
apiVersion: batch/v1
kind: CronJob
metadata:
  name: cube-hunter
  namespace: security
spec:
  schedule: "0 2 * * *"   # Daily at 2:00 AM UTC
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: kube-hunter
            image: aquasec/kube-hunter:latest
            args: ["--pod"]   # Run in "from a pod" mode
          restartPolicy: Never

Log Persistence

After the pod completes, logs are ephemeral unless you redirect them:

Option A: Fluentd/Fluent Bit → S3 → review next day.
Option B: kubectl logs piped to a file and stored in S3 by a post-job hook.
Option C: Integrate Kubescape with its SaaS dashboard (Armo Security) for historical reporting.

Adding to CI/CD

All three tools can be added to your infrastructure CI/CD pipeline (not application CI/CD):

Run scans after cluster provisioning (Terraform apply → scan → report).
Block deployment if Kube-hunter or Kube-bench report critical failures.
Send Kubescape report as a PR comment for review.

3.17 Q&A — Key Points Captured

Q: Does Intel→AMD migration cause application downtime in ASG? A: No, if you use “Launch before terminate” in the Instance Refresh settings. New AMD instances are launched and become healthy before old Intel instances are terminated. Only the “Terminate and launch” option (used in the demo for cost reasons) causes simultaneous termination + launch.

Q: Should all production changes go through Terraform, not the console? A: Yes. Console/CLI changes in production cause IaC drift (Terraform state no longer matches actual state), make auditing impossible, and are error-prone at scale. The session showed console for conceptual understanding; the Terraform equivalent was promised as a separate recorded demo and will be uploaded to Drive.

Q: AMD→ARM — where is the real learning curve? A: In the compatibility assessment: using AWS Porting Advisor to identify incompatible libraries, understanding what each finding means, and fixing the code/dependencies to make the application ARM-compatible. A dedicated demo on this was promised for the Tuesday doubt class.

Q: Can Spot instances be used in production? A: Yes, companies like Netflix, Pinterest, and Slack use Spot in production. The key requirements: stateless workloads, multiple replicas, HPA configured, PDB configured, mixed instance types in the Spot node pool, graceful shutdown in the application, and a 70/30 Spot/On-Demand ASG mix as a safety net.

Q: When should I use Compute Savings Plan vs EC2 Savings Plan? A: Compute if you have Lambda, Fargate, or multiple EC2 families/regions. EC2 Instance if you’re stable on one family in one region. Compute is always the safe default.

Q: What triggers a cost optimization engagement? A: Sudden spike in cloud bill; consistently high bill vs. low utilization (CPU <20%, memory <40%); post-migration from on-prem or monolithic to microservices (usually over-provisioned); post-lift-and-shift to cloud (instances sized like on-prem VMs, not cloud-native).

Q: Are Kube-hunter/Kubescape/Kube-bench safe to run in production? A: Yes. They are read-only scanning tools, not active exploiters (Kube-hunter simulates from inside a pod, not from outside). Runtime is under 5 minutes. If there are concerns about resource impact, schedule during low-traffic windows (e.g., Saturday night). Outputs can be redirected to S3.

Q: What’s the difference between 70/30 Spot/On-Demand on nodes vs on applications? A: The 70/30 split is a node-level division. You don’t assign specific microservices to Spot vs On-Demand. The Kubernetes scheduler places pods on any available Ready node. The mix ensures the cluster always has baseline On-Demand capacity so that if Spot nodes are reclaimed, pods reschedule onto On-Demand nodes automatically.

4. Key Concepts Table

Concept	Explanation	Example	Why It Matters
Launch Template (LT)	ASG configuration blueprint (instance type, AMI, SG, VPC, IAM)	`Intel-LT` v1: `t3.medium` → v3: `t3a.medium`	Changing LT version is the only change needed for ASG instance type migration
Instance Refresh	ASG mechanism that rolling-replaces all instances with new LT config	Start → launch AMD → health check → terminate Intel	Zero-downtime (launch-before-terminate) or simultaneous (terminate-and-launch)
Min healthy percentage	% of instances that must be healthy during instance refresh	80% (production minimum)	Below 80% risks insufficient capacity during replacement
Instance warmup time	Grace period before new instance counts toward healthy %	5 minutes	Ensures app fully starts and health checks pass before old instance is killed
EKS node group labels	Key-value metadata on nodes for scheduler targeting	`role=worker, env=prod`	Must be copied to new AMD node group or pods won’t schedule there
EKS node group taints	Key-value constraints that repel pods without a matching toleration	`dedicated=gpu:NoSchedule`	Must be matched on new node group for GPU workloads or specialty pods
kubectl drain	Evicts all non-DaemonSet pods from a node; marks node Unschedulable	`kubectl drain ng-intel-node --ignore-daemonsets`	Safely migrates workloads to new nodes before decommissioning
kubectl cordon	Marks a node as `SchedulingDisabled`; no new pods will be placed here	`kubectl cordon ng-intel-node`	Prevents scheduler from placing new pods on old Intel nodes post-drain
kubectl uncordon	Marks a node as schedulable again	`kubectl uncordon ng-intel-node` (rollback)	Restores Intel nodes to schedulable state for rollback
IAM instance profile	AWS identity attached to EC2 instances; grants access to AWS services	`eks-node-role-instance-profile`	Must NOT be specified in LT when creating EKS node groups via CLI (command creates it)
x86_64 (Intel/AMD ISA)	Instruction set architecture shared by Intel and AMD	`t3.medium` (Intel) ↔ `t3a.medium` (AMD)	Intel binaries run on AMD without change
AArch64 / ARM64 (Graviton ISA)	Different instruction set from x86; binaries are not interchangeable	`m6g.large` (Graviton)	x86 binaries will not run on ARM; must recompile or use multi-arch containers
AWS Porting Advisor for Graviton	Scans app dependencies for x86-specific incompatibilities before ARM migration	Detects x86 preprocessor macros, non-ARM native libraries	Prevents production breakage from undetected architecture dependencies
PRC framework	Performance + Reliability + Cost — the three pillars to balance in right-sizing	Never downsize below P99 peak + 20% buffer	Prevents optimization from causing production outages
20–33% buffer rule	When right-sizing, add 20–33% headroom above P99 metric	P99 CPU = 60% → size for 80% utilization ceiling	Absorbs traffic spikes; prevents OOM and CPU starvation
Compute Optimizer	AWS ML-based right-sizing recommendations tool	”Downsize M5.xlarge to M5.large — save $45/month”	Removes guesswork; based on actual CloudWatch utilization data
Spot lifecycle	EC2 instance attribute indicating it runs on Spot capacity	`lifecycle = spot` (vs `normal` for On-Demand)	Spot can be reclaimed with 2-minute notice
70/30 Spot/On-Demand ASG	Mixed instance policy: 70% Spot nodes, 30% On-Demand nodes	EKS cluster with two node pools	On-Demand provides stable fallback when Spot is interrupted
PDB (PodDisruptionBudget)	K8s policy limiting simultaneous pod disruptions	`minAvailable: 2` for a 3-replica deployment	Prevents drain/Spot interruption from removing all replicas simultaneously
Kube-hunter	K8s penetration testing tool; simulates attacker from inside a pod	Detects API server exposure, credential leaks	Shows what a compromised pod can access; essential for security posture
Kubescape	Comprehensive K8s misconfiguration + compliance scanner	Detects no CPU limits, root containers, open ingress/egress	Most comprehensive single tool; use if you can only use one
Kube-bench	CIS Kubernetes Benchmark auditing tool	Pass/Warn/Fail per CIS control	Required by auditors in fintech/healthcare; baseline hardening check
CIS Benchmark	Center for Internet Security standard for secure K8s configuration	13 checks, 0 failures, 3 warnings (training cluster)	Industry-standard audit baseline; required for compliance in many sectors

5. Architecture & Workflow Analysis

5.1 ASG Migration Flow (Intel → AMD)

CURRENT STATE:
  ASG (sre-labs-asg-demo)
    └── Launch Template: Intel-LT v1 (t3.medium)
          └── Instances: [i-xxxx (t3.medium, running)]

MIGRATION STEPS:
  1. Create LT v3 (same as v1 but instance_type = t3a.medium)
  2. Update ASG → use LT v3
  3. Instance Refresh (launch-before-terminate, min_healthy=80%, warmup=5min)
       ├── ASG launches new i-yyyy (t3a.medium) → waits health check pass
       └── ASG terminates old i-xxxx (t3.medium)

RESULT:
  ASG (sre-labs-asg-demo)
    └── Launch Template: Intel-LT v3 (t3a.medium)
          └── Instances: [i-yyyy (t3a.medium, running)]

ROLLBACK: Update ASG → use LT v1 → start another Instance Refresh

5.2 EKS Node Group Migration Flow (Intel → AMD)

CURRENT STATE:
  EKS Cluster: sre-labs-eks
  ├── Node Group: ng-intel (t3.medium, desired=2, labels={role:worker}, taints={})
  │     ├── node-1 (Intel): pods [A, B, C, D]
  │     └── node-2 (Intel): pods [E, F, G, H]
  └── Applications: Nginx (4 replicas) on ng-intel

MIGRATION STEPS:
  1. Inspect ng-intel: subnets, node role, labels, taints, ASG config
  2. Create LT v4: same config + t3a.medium + NO IAM instance profile
  3. Create ng-amd: same subnets, same node role, same scaling, LT v4
  4. Copy labels + taints from ng-intel to ng-amd
  5. Drain ng-intel nodes:
       kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
       kubectl drain node-2 --ignore-daemonsets --delete-emptydir-data
       → Pods A-H evicted → scheduler places them on ng-amd nodes
  6. Cordon ng-intel nodes:
       kubectl cordon node-1 && kubectl cordon node-2
       → SchedulingDisabled; no new pods will land here
  7. Verify: all pods on ng-amd nodes
  8. Observe 72 hours → if stable: delete ng-intel node group

RESULT:
  EKS Cluster: sre-labs-eks
  └── Node Group: ng-amd (t3a.medium, desired=2)
        ├── node-3 (AMD): pods [A, B, C, D, E, F]
        └── node-4 (AMD): pods [G, H]

ROLLBACK (within 72h):
  kubectl drain ng-amd-node-3 && kubectl drain ng-amd-node-4
  kubectl uncordon ng-intel-node-1 && kubectl uncordon ng-intel-node-2
  → Pods reschedule back to Intel nodes (~2-3 min downtime)

5.3 Production Spot Architecture (70/30 Mix)

EKS Cluster
├── Node Pool: spot-pool (70% of desired capacity)
│   ├── Instance types: [t3a.medium, t3.medium, t2.medium]  ← diversified for availability
│   ├── Lifecycle: spot
│   └── Fallback chain: t3a.medium → t3.medium → t2.medium
│
├── Node Pool: ondemand-pool (30% of desired capacity)
│   ├── Instance type: t3a.medium
│   └── Lifecycle: on-demand
│
├── HPA: scales pods up/down based on CPU/memory
├── PDB: minAvailable=N-1 for each critical workload
└── Graceful shutdown: app handles SIGTERM → drain in-flight requests

EVENT: AWS reclaims t3a.medium Spot in ap-south-1a
  → Spot pool tries t3.medium → found → new nodes start
  → Pods on evicted nodes reschedule to remaining spot + on-demand nodes
  → HPA scales up if remaining capacity is under-provisioned
  → Users see: ~2-second latency spike, zero dropped requests (if graceful shutdown set up)

5.4 EKS Security Scanning Architecture

EKS Cluster
└── Namespace: security
    ├── CronJob: kube-hunter   (daily, 2:00 AM)
    ├── CronJob: kubescape     (daily/weekly)
    └── CronJob: kube-bench    (weekly)

Execution flow:
  CronJob triggers → Job created → Pod spins up → Tool runs scan → Pod exits
  → kubectl logs <pod> → results captured
  → [Optional] Logs forwarded to S3 / CloudWatch / Kubescape SaaS dashboard

Teams process:
  Weekly review of logs → prioritize findings → fix in dev → verify scan pass → apply to prod

6. Commands, Scripts & Configs

ASG — Get the current Launch Template

aws autoscaling describe-auto-scaling-groups \
  --auto-scaling-group-names sre-labs-asg-demo \
  --query 'AutoScalingGroups[*].LaunchTemplate' \
  --output table

ASG — Trigger Instance Refresh

aws autoscaling start-instance-refresh \
  --auto-scaling-group-name sre-labs-asg-demo \
  --preferences '{
    "MinHealthyPercentage": 80,
    "InstanceWarmup": 300,
    "CheckpointDelay": 3600
  }'

ASG — Check Instance Refresh status

aws autoscaling describe-instance-refreshes \
  --auto-scaling-group-name sre-labs-asg-demo \
  --query 'InstanceRefreshes[0].{Status:Status,PercentageComplete:PercentageComplete}'

EKS — Describe node group subnets

aws eks describe-nodegroup \
  --cluster-name sre-labs-eks \
  --nodegroup-name ng-intel \
  --region ap-south-1 \
  --query 'nodegroup.{Subnets:subnets,NodeRole:nodeRole,ScalingConfig:scalingConfig}' \
  --output json

EKS — Create AMD node group from new LT version

aws eks create-nodegroup \
  --cluster-name sre-labs-eks \
  --nodegroup-name ng-amd \
  --scaling-config minSize=2,maxSize=4,desiredSize=2 \
  --subnets subnet-c2xxxx subnet-yyyyy subnet-5ezzzz \
  --launch-template name=<LT-name>,version=4 \
  --node-role arn:aws:iam::<account>:role/<eks-node-role> \
  --region ap-south-1

EKS — Drain and cordon Intel nodes

# Get Intel node names
kubectl get nodes -l eks.amazonaws.com/nodegroup=ng-intel -o name

# Drain each node (repeat for all Intel nodes):
kubectl drain node/<node-name> \
  --ignore-daemonsets \
  --delete-emptydir-data \
  --force

# Cordon (mark scheduling-disabled):
kubectl cordon node/<node-name>

# Rollback — uncordon:
kubectl uncordon node/<node-name>

EKS — Copy labels to new node group

# Get labels from Intel node group
kubectl get nodegroup ng-intel -o jsonpath='{.labels}' --context <cluster-context>
# Or from a specific node:
kubectl describe node <intel-node-name> | grep Labels -A 20

# Apply to AMD node group (via AWS CLI):
aws eks update-nodegroup-config \
  --cluster-name sre-labs-eks \
  --nodegroup-name ng-amd \
  --labels addOrUpdateLabels={role=worker,env=prod}

EKS — Verify pod placement on AMD nodes

kubectl get pods -A -o wide | grep <amd-node-ip>

Non-prod shutdown — Lambda start/stop scripts

(See Section 3.6 for full Python and Bash scripts)

EventBridge rules for non-prod shutdown

# Stop rule (10 PM IST = 16:30 UTC):
aws events put-rule \
  --name "nonprod-stop-rule" \
  --schedule-expression "cron(30 16 * * ? *)" \
  --state ENABLED

# Start rule (8 AM IST = 02:30 UTC):
aws events put-rule \
  --name "nonprod-start-rule" \
  --schedule-expression "cron(30 2 * * ? *)" \
  --state ENABLED

Kubescape — Manual scan (without CronJob)

# Install kubescape CLI:
curl -s https://raw.githubusercontent.com/armosec/kubescape/master/install.sh | /bin/bash

# Run scan against current cluster:
kubescape scan framework nsa --exclude-namespaces kube-system,kube-public

# Scan with output to JSON:
kubescape scan framework nsa --format json --output results.json

Kube-bench — Manual run (outside CronJob)

# On a worker node (requires host access):
docker run --rm \
  -v /etc:/etc:ro \
  -v /var:/var:ro \
  -v /usr/bin/containerd:/usr/bin/containerd:ro \
  aquasec/kube-bench:latest \
  --benchmark eks-1.3.0   # Use the benchmark matching your EKS version

Pod SecurityContext — Fix “running as root” finding

# In your Deployment/Pod spec:
spec:
  securityContext:
    runAsNonRoot: true
    runAsUser: 1000
    fsGroup: 2000
  containers:
  - name: app
    securityContext:
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop: ["ALL"]

Resource limits — Fix “no CPU/memory limits” finding

containers:
- name: app
  resources:
    requests:
      cpu: "100m"
      memory: "128Mi"
    limits:
      cpu: "500m"
      memory: "512Mi"

PodDisruptionBudget example

apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: app-pdb
spec:
  minAvailable: 2          # At least 2 pods must always be available
  selector:
    matchLabels:
      app: my-application

Mixed Spot/On-Demand ASG instance policy

{
  "MixedInstancesPolicy": {
    "InstancesDistribution": {
      "OnDemandBaseCapacity": 0,
      "OnDemandPercentageAboveBaseCapacity": 30,
      "SpotAllocationStrategy": "capacity-optimized"
    },
    "LaunchTemplate": {
      "LaunchTemplateSpecification": {
        "LaunchTemplateName": "app-lt",
        "Version": "$Latest"
      },
      "Overrides": [
        {"InstanceType": "t3a.medium"},
        {"InstanceType": "t3.medium"},
        {"InstanceType": "t2.medium"}
      ]
    }
  }
}

7. Tools & Technologies

Tool	Category	Purpose	Key session notes
AWS EC2 Launch Templates	Native AWS	Define instance configs; version-controlled; used by ASGs	Core mechanism for ASG migration — never modify instance type directly
AWS Instance Refresh	Native AWS	Rolling ASG instance replacement	Two modes: launch-before-terminate (zero downtime) vs terminate-and-launch (brief overlap)
kubectl drain	Kubernetes CLI	Evict pods from a node safely	Use `--ignore-daemonsets --delete-emptydir-data` for clean migration
kubectl cordon / uncordon	Kubernetes CLI	Mark node as scheduling-disabled / re-enable it	Essential for controlled rollback
AWS Porting Advisor for Graviton	Native AWS	Scan app dependencies for ARM incompatibilities	Prerequisite for Intel/AMD→ARM migration; generates HTML report
AWS Compute Optimizer	Native AWS	Right-size EC2, Lambda, EKS recommendations	Takes ~24h to initialize; ML-based; shows CPU/memory trends
GCP Active Assist	Native GCP	GCP equivalent of Compute Optimizer	Used in Polomial shadowing sessions
AWS Savings Plans Calculator	Native AWS	Size Savings Plan commitments correctly	Covered in detail in 30-page doc + recording (separate from this session)
Kube-hunter	Open-source (Aqua)	K8s penetration testing from inside a pod	Daily CronJob; findings indexed by `KHV` ID at aquasec.io
Kubescape	Open-source (Armo)	K8s misconfiguration + compliance scanning	Most comprehensive tool; remediation links per finding; optional SaaS dashboard
Kube-bench	Open-source (Aqua)	CIS K8s Benchmark auditing	Required for auditors; pass/warn/fail per CIS control
HPA (Horizontal Pod Autoscaler)	Kubernetes native	Scale pod count based on metrics	Required safeguard for Spot workloads
PDB (PodDisruptionBudget)	Kubernetes native	Limit simultaneous pod disruptions	Required to prevent drain from killing all replicas at once
Lambda + EventBridge	Native AWS	Serverless non-prod shutdown scheduler	Adds small Lambda/EventBridge cost; zero-maintenance
Jenkins (CI/CD cron)	Self-hosted	Alternative scheduler for non-prod shutdown	No additional cloud cost; requires existing Jenkins
Karpenter	Open-source (AWS)	EKS node autoscaler (cost-optimized)	Covered next session; code + demo in Drive already
CastAI	Third-party SaaS	Full EKS cost optimization	Covered next session; integrates with Karpenter
Kubecost	Open-source / SaaS	K8s cost visibility per namespace/workload	Covered next session

8. Real-World Production Usage

ASG migrations at scale:

For large fleets (100+ ASGs), use Terraform to update all Launch Templates simultaneously.
Never use console for bulk migrations — IaC keeps state coherent and enables safe rollback.
The session showed console/CLI for pedagogical clarity; the Terraform approach is identical in concept but parameterized.

EKS node group migrations:

At enterprises with multi-cluster setups, this procedure is scripted and triggered via CI/CD.
Consider using Cluster Autoscaler or Karpenter to handle node group scaling during migrations instead of manually setting desired capacity.
Multi-AZ node groups: drain/cordon must be done per-node, not per-node-group, to maintain AZ spread.

Right-sizing lessons from real engagements:

Real example from session: During L&T (Larsen & Toubro) migration from GCP to AWS, simultaneous right-sizing caused incidents. Different cloud providers have different hypervisor performance characteristics; an instance that appeared over-provisioned on GCP was actually correctly sized for AWS. Lesson: never right-size during migration.
Startups often aggressively right-size and then face outage during traffic events. Conservative buffers (20–33%) prevent this.

Spot in production at scale:

Netflix, Slack, Pinterest: production Spot users.
Key design principle: applications must be disposable — no in-memory state, no local filesystem dependency for persistent data, external session stores (Redis), externalized config (Secrets Manager).
The 2-minute reclamation notice is sufficient for graceful shutdown if SIGTERM handling is implemented.

Security scanning:

Finance/healthcare: clean Kube-bench report is often a hard deploy gate.
Running as root: extremely common unintentional misconfiguration; easy fix (runAsNonRoot: true); caught by Kubescape every time.
CPU/memory limits: absence is a very frequent finding even in mature organizations — developers skip limits during early development, forget to add them before production.

9. Interview Preparation

Beginner

Q1. What is an EC2 Launch Template and why does it matter for ASG migrations? A: A Launch Template is a reusable EC2 configuration blueprint (instance type, AMI, security groups, IAM role, etc.) used by Auto Scaling Groups to launch new instances. Migrating an ASG to a new instance type requires creating a new LT version with the desired type, then triggering an Instance Refresh — the ASG uses the new version for all new instances it launches during the refresh.

Q2. What is the difference between kubectl drain and kubectl cordon? A: cordon marks a node as SchedulingDisabled — no new pods will be scheduled on it, but existing pods keep running. drain does both: it evicts all running pods (except DaemonSets) and marks the node as unschedulable. For EKS node group migration, you drain first (move pods off), then cordon (confirm no new pods will land). To rollback: uncordon re-enables scheduling on the old nodes.

Q3. What are the three EKS security scanning tools covered and what does each focus on? A: Kube-hunter (penetration testing — simulates an attacker inside a pod), Kubescape (misconfiguration and compliance scanning — most comprehensive, with per-finding remediation links), and Kube-bench (CIS Benchmark auditing — pass/fail per CIS control, used by auditors).

Intermediate

Q4. Walk through an ASG Intel→AMD migration with zero downtime. A: Create a new Launch Template version with t3a.medium instead of t3.medium. Update the ASG to reference the new LT version. Trigger Instance Refresh with MinHealthyPercentage=80 and InstanceWarmup=300 seconds, using the “Launch before terminate” replacement behavior. The ASG launches new AMD instances, waits for health checks to pass, then terminates old Intel instances. Rollback: update ASG to previous LT version and start another Instance Refresh.

Q5. Why does EKS node group migration require ~2–3 minutes of downtime, while ASG migration has zero downtime? A: ASG migrations use Instance Refresh (launch-before-terminate): new instances are fully healthy and serving traffic before old instances are removed. In EKS, you must drain pods off Intel nodes (evicting them) so they can reschedule onto AMD nodes. During the re-scheduling window — from drain until pods are Running on the new nodes — the pod count is reduced. With HPA and PDB configured, this window is ~2–3 minutes. Without them, downtime could be longer.

Q6. Explain the PRC framework for right-sizing decisions. A: Performance, Reliability, and Cost. All three must be balanced. Right-sizing optimizes Cost but must not degrade Performance (latency, throughput) or Reliability (ability to handle traffic spikes). The 20–33% buffer rule ensures the instance has headroom above the P99 peak for traffic spikes. Never size to the average — size to P99 + buffer.

Q7. What are the mandatory Kubernetes safeguards before running Spot instances in production? A: HPA (scales pod count when Spot is interrupted and remaining capacity is under-pressure), PDB (prevents drain from removing all replicas simultaneously), ≥2 replicas per workload (ensures one survives any single node interruption), instance-type diversification (3–5 compatible types in the Spot pool to avoid simultaneous reclamation), graceful shutdown (SIGTERM handling to complete in-flight requests within the 2-minute reclamation window), and a 70/30 Spot/On-Demand ASG mix (On-Demand as stable fallback).

Advanced

Q8. Why can you NOT specify an IAM instance profile in the Launch Template when creating EKS node groups via CLI? A: When you use aws eks create-nodegroup with a --node-role, the EKS service automatically creates and attaches an instance profile derived from that node role to the EC2 instances in the node group. If the Launch Template also specifies an instance profile, AWS returns an error because only one instance profile can be attached to an EC2 instance. The fix: in the Launch Template’s Advanced Details, explicitly set “Do not include IAM instance profile.”

Q9. A Kubescape scan finds 8 of 14 control checks failing for a pod. The most critical finding is “running as root.” What’s the complete fix, and why does it matter? A: Add securityContext: { runAsNonRoot: true, runAsUser: <non-zero-UID> } to the pod spec, and allowPrivilegeEscalation: false to the container spec. Running as root inside a container means that if an attacker achieves container escape (exploiting a kernel vulnerability), they have root on the host node — potentially compromising all other pods on that node. Non-root container compromise still risks lateral movement but is contained at the application level, not the host level.

Q10. A team wants to move their stateful MySQL pod to a Spot instance to save costs. What would you tell them? A: Strongly advise against it. Stateful workloads like databases store critical data locally. When AWS reclaims a Spot instance with 2-minute notice, MySQL’s data directory is on the terminated instance. Even with EBS volumes (which persist after instance termination), the 2-minute window may not be enough for a clean MySQL shutdown, risking database corruption. The correct pattern: keep MySQL on On-Demand. If cost is a concern, purchase an RDS Reserved Instance (which is effectively a committed savings plan for managed databases). Only stateless workloads belong on Spot.

10. Exam & Certification Notes

AWS DevOps Professional / Solutions Architect relevant:

Launch Template versioning and Instance Refresh — know the MinHealthyPercentage and InstanceWarmup parameters.
Spot vs Reserved vs On-Demand vs Savings Plans: a frequent exam comparison question.
PDB: a common exam topic in the context of availability guarantees during cluster operations.
HPA: know the difference from VPA (Vertical Pod Autoscaler) — HPA adds pods, VPA changes CPU/memory requests.
IAM instance profile: one profile per instance; only specifiable in LT OR via EKS node role, not both.

Kubernetes CKA/CKAD/CKS relevant:

kubectl drain, cordon, uncordon — these are explicitly tested in CKA.
Security context (runAsNonRoot, allowPrivilegeEscalation, readOnlyRootFilesystem) — tested in CKS.
PDB — tested in CKAD and CKA.
Network policies (mentioned as a Kubescape finding) — key CKS/CKAD topic.

Potential trick questions:

“Instance Refresh with MinHealthyPercentage=100% guarantees no downtime” → False — at 100% you cannot replace any instances because that would drop below 100%. A practical minimum-downtime setting is 80–90%.
”Spot instances can be purchased under Savings Plans” → False — Savings Plans apply to On-Demand only.
”AMD instances require recompilation of x86 applications” → False — both Intel and AMD are x86_64; binaries are interchangeable.
”kubectl cordon removes existing pods from a node” → False — cordon only prevents new pods from scheduling. drain removes existing pods.
”A PDB with minAvailable: 0 prevents all disruptions” → False — minAvailable: 0 means zero pods need to be available, so all can be disrupted. Use maxUnavailable: 0 to prevent all disruptions.

11. Cheat Sheet

ASG Intel→AMD migration (zero downtime): LT v_new (t3a.medium) → ASG update → Instance Refresh (launch-before-terminate, 80% healthy, 5min warmup) → verify Cost Explorer → delete old LT version after 72h

EKS node group Intel→AMD migration (~2-3 min downtime):

1. Inspect ng-intel: labels, taints, subnets, node role, ASG scaling config
2. Create LT v_new: t3a.medium + NO IAM instance profile
3. Create ng-amd: same subnets + node role + scaling config + LT v_new
4. Copy labels + taints to ng-amd
5. kubectl drain <intel-nodes> --ignore-daemonsets --delete-emptydir-data
6. kubectl cordon <intel-nodes>
7. Verify pods on ng-amd → observe 72h → delete ng-intel

AMD→ARM migration pre-check: Developer sign-off → AWS Porting Advisor scan → fix incompatible libs → test on ARM non-prod → then migrate

Migration complexity matrix:

	Intel→AMD	AMD→ARM
Standalone	Easiest	Hardest
ASG	Medium	Medium
EKS node group	Hardest	Easiest

Non-prod shutdown: Tag ENV=non-prod → Python/Bash script → Lambda + EventBridge OR Jenkins cron → stop 10PM IST, start 8AM IST

Right-sizing (PRC): Baseline 90d → Compute Optimizer → size to P99 + 20-33% buffer → never size during migration

Spot production pattern: 70% Spot / 30% On-Demand → multiple instance types → HPA + PDB + ≥2 replicas + graceful SIGTERM → On-Demand as fallback

EKS security scanning (all as CronJobs):

Kube-hunter → daily → penetration test; findings at KHV IDs
Kubescape → daily/weekly → misconfig + compliance; most comprehensive; add one tool = this one
Kube-bench → weekly → CIS audit; pass/warn/fail; required for auditors

Common Kubescape fixes:

No CPU/memory limits → add resources.limits
Running as root → runAsNonRoot: true, runAsUser: 1000
Privilege escalation → allowPrivilegeEscalation: false
Open ingress/egress → add NetworkPolicy

12. Gaps, Assumptions & Incomplete Areas

Explicitly deferred to next sessions:

EKS cost optimization (Karpenter, CastAI, Kubecost) — promised as the full focus of Project Call 3.
ARM compatibility demo (Porting Advisor end-to-end with a real application, fixing incompatibilities) — promised for Tuesday doubt class and to be recorded.
Database (RDS/ElastiCache/DocumentDB) cost optimization — not yet covered.
Networking cost optimization (NAT Gateway, VPC Endpoints) — not yet covered.
Monitoring/logging cost optimization (CloudWatch) — deferred.
Security audit project (SecureAsset fintech client) — next project in the series.

Items promised for upload but not in transcript:

Kube-hunter YAML, Kubescape YAML, Kube-bench YAML (uploaded to Google Drive by guest presenter Ravi).
Kube-hunter log output, Kubescape log output, Kube-bench log output (uploaded to Google Drive).
Terraform code for instance migrations (Terraform demo was recorded separately; video to be uploaded).
AWS SDK inventory extraction script and Excel sheets (uploaded to Google Drive after Call 1).
Non-prod shutdown scripts (start_lambda.py, stop_lambda.py, Bash equivalent) — uploaded to Drive.
Savings Plans detailed documentation (~30 pages) + recording — available separately.
Karpenter demo code and CastAI demo code — uploaded to Drive already (for self-study before Call 3).
Week 2 assignments — to be uploaded Monday.

Transcription artifacts:

“Cube bench / cube hunter / cubecape” = Kube-bench / Kube-hunter / Kubescape
”Ease cluster / ease” = EKS (Elastic Kubernetes Service)
“AG / ASG” = Auto Scaling Group
”Terapform” = Terraform
”Genkins” = Jenkins
”PTB / PDB” = Pod Disruption Budget
”Carpenter” = Karpenter
”Casti” = CastAI
”Cube cost” = Kubecost
”Active assist” (GCP) = Recommender API / Active Assist (GCP equivalent of AWS Compute Optimizer)
“Aguas / aquas” = Aqua Security (vendor behind Kube-hunter and Kube-bench)
“Armosec” = ARMO Security (vendor behind Kubescape)

Assumptions made:

The “IAM instance profile” issue in EKS node group LT is correctly attributed to the aws eks create-nodegroup command auto-creating the instance profile. This matches documented EKS behavior.
The guest presenter’s name “Ravi” is inferred from “Ravi I think uh you are not sharing that screen."
"LNT” (mentioned in right-sizing context) = Larsen & Toubro (Indian conglomerate), a former client.

13. Gap-Fill — What the Session Left Unfinished, Completed Here

Filled from Kubernetes and AWS engineering knowledge. Clearly labeled as gap-fill.

GAP 1 — Terraform Equivalents for the Console Operations Shown

The session showed console/CLI operations; the instructor confirmed Terraform is the production standard. Here are the Terraform equivalents for the key operations.

ASG Instance Type Migration (Intel→AMD via Terraform):

# Step 1: Update the launch template with new instance type
resource "aws_launch_template" "app" {
  name        = "Intel-LT"
  description = "Intel to AMD LT version"

  instance_type = "t3a.medium"   # Changed from t3.medium

  # All other config unchanged
  vpc_security_group_ids = [aws_security_group.app.id]

  tag_specifications {
    resource_type = "instance"
    tags = {
      Name  = "sre-labs-app"
      ENV   = "prod"
      Owner = "devops"
    }
  }
}

# Step 2: Update ASG to use new LT version + enable instance refresh
resource "aws_autoscaling_group" "app" {
  name                = "sre-labs-asg-demo"
  min_size            = 1
  max_size            = 4
  desired_capacity    = 1

  launch_template {
    id      = aws_launch_template.app.id
    version = "$Latest"           # Always use latest version
  }

  # Rolling instance replacement on LT change:
  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 80
      instance_warmup        = 300
    }
  }

  # Tags for non-prod scheduling:
  tag {
    key                 = "ENV"
    value               = "prod"
    propagate_at_launch = true
  }
}

EKS Node Group Migration (AMD) via Terraform:

resource "aws_eks_node_group" "amd" {
  cluster_name    = "sre-labs-eks"
  node_group_name = "ng-amd"
  node_role_arn   = aws_iam_role.eks_node.arn

  subnet_ids = [
    aws_subnet.private_a.id,
    aws_subnet.private_b.id,
    aws_subnet.private_c.id,
  ]

  launch_template {
    name    = aws_launch_template.eks_amd.name
    version = aws_launch_template.eks_amd.latest_version
  }

  scaling_config {
    desired_size = 2
    max_size     = 4
    min_size     = 2
  }

  labels = {
    role = "worker"
    env  = "prod"
  }

  # Do NOT set taint or launch_template IAM instance profile in Terraform for EKS node groups
  # Node role handles instance profile automatically via aws_iam_instance_profile
}

GAP 2 — What “Launch Before Terminate” vs “Terminate and Launch” Actually Means

The session mentioned both options without fully explaining the cost/availability tradeoff.

Launch Before Terminate (zero-downtime):

Time 0:  2 Intel instances running (capacity = 2)
Time 1:  ASG launches 2 AMD instances (capacity = 4) → wait health checks pass
Time 2:  ASG terminates 2 Intel instances (capacity = 2 AMD)
Cost:    You pay for 4 instances briefly during the overlap (~5 min * 4 instances)

Use when: production workloads where downtime is unacceptable.

Terminate and Launch (brief downtime, lower cost during migration):

Time 0:  2 Intel instances running (capacity = 2)
Time 1:  ASG terminates 1 Intel instance (capacity = 1)
Time 2:  ASG launches 1 AMD instance and waits health check
Time 3:  Once AMD is healthy, terminates next Intel → launches next AMD
Cost:    No overlap — no double-billing
Downtime: Brief degraded capacity (80% min healthy = 1 of 2 instances)

Use when: dev/test environments, or when saving the overlap cost matters and brief capacity reduction is acceptable.

In HealthCorp’s case: The instructor used terminate-and-launch during the demo (“controls the cost”) because it was a demo environment and they wanted to avoid the overlap billing.

GAP 3 — Karpenter vs CastAI vs Kubecost — Preview (for Session 3)

Since these are promised for the next session and have Drive resources, here is the essential context:

Karpenter (AWS open-source):

Replaces Cluster Autoscaler for EKS.
Provisions the cheapest node that fits the pod’s resource requests directly (not from a predefined node group).
Can mix On-Demand and Spot at the pod level (via karpenter.sh/capacity-type node selector).
Supports instance type diversification natively.
Decommissions nodes when empty (bin-packing), reducing idle node cost.

CastAI:

SaaS product. Works across EKS, GKE, AKS.
Combines: right-sizing, Spot orchestration, autoscaling recommendations, multi-cloud visibility.
More opinionated and automated than Karpenter alone.
Can work alongside Karpenter (Karpenter for provisioning, CastAI for optimization recommendations).

Kubecost:

Cost visibility tool. Shows cost per namespace, workload, label, team.
Does NOT optimize — it reports.
Essential for chargeback/showback and understanding where EKS spend comes from.
Use alongside Karpenter + CastAI: Kubecost shows the cost impact of Karpenter/CastAI changes.

GAP 4 — Full AMD → ARM Standalone Migration Procedure (Not Covered in Session)

The session said standalone AMD→ARM is “the hardest” but didn’t demonstrate it. Here is the procedure:

WHY IT'S HARD: You cannot change instance type from AMD (x86_64) to ARM (AArch64) in-place.
AWS does not support cross-ISA instance type changes. You must:
  1. Launch a NEW ARM instance from an ARM-compatible AMI.
  2. Install and configure the application on the new instance.
  3. Migrate any persistent data (EBS snapshot → new ARM EBS).
  4. Update DNS / load balancer / service discovery to point to new instance.
  5. Test for 72 hours.
  6. Terminate old AMD instance.

Step-by-step:

# Step 1: Find ARM-compatible AMI (Ubuntu 22.04 ARM64)
aws ec2 describe-images \
  --owners amazon \
  --filters \
    "Name=name,Values=ubuntu/images/hvm-ssd/ubuntu-jammy-22.04-arm64-*" \
    "Name=state,Values=available" \
  --query 'sort_by(Images, &CreationDate)[-1].ImageId' \
  --output text
# Returns: ami-xxxxxxxxxx (ARM64 AMI)

# Step 2: Launch new ARM instance
aws ec2 run-instances \
  --image-id <arm64-ami-id> \
  --instance-type t4g.medium \    # t4g = Graviton 2; cheapest Graviton
  --key-name <your-key> \
  --security-group-ids <sg-id> \
  --subnet-id <subnet-id> \
  --iam-instance-profile Name=<instance-profile> \
  --tag-specifications 'ResourceType=instance,Tags=[{Key=Name,Value=app-arm},{Key=ENV,Value=prod}]'

# Step 3: Take EBS snapshot from old AMD instance for data migration
INSTANCE_ID="i-old-amd-instance"
VOLUME_ID=$(aws ec2 describe-instances --instance-ids $INSTANCE_ID \
  --query 'Reservations[0].Instances[0].BlockDeviceMappings[0].Ebs.VolumeId' \
  --output text)
SNAP_ID=$(aws ec2 create-snapshot --volume-id $VOLUME_ID \
  --description "AMD to ARM migration backup" \
  --query 'SnapshotId' --output text)

# Wait for snapshot completion:
aws ec2 wait snapshot-completed --snapshot-ids $SNAP_ID

# Step 4: Create volume from snapshot in same AZ as new ARM instance
aws ec2 create-volume \
  --snapshot-id $SNAP_ID \
  --availability-zone ap-south-1a \
  --volume-type gp3 \
  --tag-specifications 'ResourceType=volume,Tags=[{Key=Name,Value=app-arm-data}]'

# Step 5: Attach data volume to new ARM instance; mount and verify data

# Step 6: Deploy application binaries compiled for ARM64
# (Application must already be ARM-compatible per Porting Advisor assessment)
# For containerized apps: pull multi-arch images or ARM-specific image tags

# Step 7: Test application on ARM instance
# Step 8: Update DNS/ALB target group to point to ARM instance
# Step 9: Monitor 72 hours → terminate AMD instance → delete snapshot

GAP 5 — Network Policy (Fix for Kubescape “Ingress/Egress not blocked” Finding)

Kubescape frequently flags pods with no NetworkPolicy — meaning any pod can talk to any other pod in the cluster. Here is the fix:

# Default deny-all ingress for a namespace:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
  namespace: production
spec:
  podSelector: {}       # applies to all pods in namespace
  policyTypes:
  - Ingress

---
# Allow ingress only from specific pods:
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

---
# Default deny-all egress (very strict; add exceptions as needed):
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-egress
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Egress

Common gotcha: DNS (port 53 UDP/TCP to kube-dns) must be explicitly allowed even after a deny-all egress policy, or all service DNS resolution in the namespace will break.

# Allow DNS egress:
- egress:
  - to:
    - namespaceSelector:
        matchLabels:
          kubernetes.io/metadata.name: kube-system
    ports:
    - protocol: UDP
      port: 53
    - protocol: TCP
      port: 53

GAP 6 — Cost Comparison: Lambda+EventBridge vs Jenkins for Non-Prod Scheduling

The session presented both options without quantifying the tradeoff. Here it is:

Lambda + EventBridge:

Lambda cost: ~0.0000002 USD per invocation × 2 invocations/day × 30 days = ~$0.000012/month ≈ $0.
EventBridge cost: $1/million events. 60 events/month = effectively $0.
Total overhead cost: negligible (~$0/month).
Benefit: zero server to manage; runs even if Jenkins is down.

Jenkins cron:

Requires Jenkins always running (EC2 or EKS pod). If Jenkins is on a non-prod instance and shuts down with the others, it cannot start them back up.
Solution: host Jenkins on a prod-tagged instance that is always running, or on a small cheap dedicated instance (e.g., t3a.nano at ~$3/month).
Benefit: no additional AWS-managed service cost for simple scheduling; Bash or Python both work.

Verdict: Lambda + EventBridge is simpler and cheaper unless you already use Jenkins for CI/CD (in which case the Jenkins approach leverages an already-paid-for resource). The instructor’s suggestion to use Jenkins was motivated by organizations that are already paying for Jenkins and don’t want to add Lambda invocation to the bill — even though the Lambda cost is negligible.

Complete Learning Package (EC2 Migration · EKS Migration · Spot · Savings Plans · EKS Security Scanning)

2. Table of Contents

3. Detailed Structured Notes

3.1 Session Recap — What Was Covered in Call 1

3.2 ASG Instance Migration — Intel → AMD (Live Demo)

Context

Step-by-Step (Live Demo: sre-labs-asg-demo, T3.medium → T3a.medium)

Step 1 — Identify the current Launch Template

Step 2 — Create a new LT version

Step 3 — Update the ASG to use the new LT version

Step 4 — Trigger Instance Refresh

Step 5 — Verify

Rollback

Observability window

3.3 EKS Node Group Migration — Intel → AMD (Live Demo)

Why this category is the most complex (Intel→AMD)

Prerequisites before starting

Step-by-Step (Live Demo: sre-labs-eks, node group ng-intel, T3.medium → T3a.medium)

Phase 1 — Create new AMD Launch Template version (in the ASG attached to the EKS node group)

Phase 2 — Get the subnets attached to the Intel node group

Phase 3 — Get the ASG scaling configuration

Phase 4 — Create the new AMD node group (CLI)

Phase 5 — Copy labels and taints to the new node group

Phase 6 — Drain the Intel nodes (move all pods off them)

Phase 7 — Cordon the Intel nodes (prevent new scheduling)

Phase 8 — Verify workloads on AMD nodes

Phase 9 — Observe for 72 hours then delete the Intel node group

Rollback (within the 72-hour window)

3.4 AMD → ARM Migration — Concepts and Assessment Workflow

Why AMD→ARM is architecturally different

Migration complexity matrix (revisited)

Blast radius consideration

AMD → ARM Assessment Workflow

Step 1 — Developer sign-off (always first)

Step 2 — AWS Porting Advisor for Graviton

Step 3 — Fix identified issues

Step 4 — Test on an ARM instance in non-prod

3.5 Assignment — AMD → ARM Hands-On

3.6 Non-Prod Environment Shutdown

The Problem

The Solution

Scheduling

Tag Strategy

Scripts (uploaded to Google Drive)

Python — Start Lambda (start_lambda.py)

Python — Stop Lambda (stop_lambda.py)

Bash equivalent (for Jenkins hosting)

EventBridge Rules (when using Lambda)

3.7 Right-Sizing EC2 Instances — the PRC Framework

The Three Pillars: PRC

How to Right-Size

Step 1 — Collect metrics over a baseline period

Step 2 — Use Compute Optimizer

Step 3 — Apply the 20–33% buffer rule

Step 4 — Validate before and after

When NOT to Right-Size

Tools Available

3.8 On-Demand → Spot Migration

What Spot Instances Are

Demo — Launching an instance as Spot

3.9 Spot in Production — Architecture Patterns

The Myth

The Reality

When Spot is Appropriate

When Spot Is Risky / Inappropriate

Production Spot Architecture Pattern: 70% Spot / 30% On-Demand Mix

Required Kubernetes Safeguards for Production Spot

Node-Level vs Application-Level Division

3.10 Savings Plans — Overview and Buying Strategy

Two Types (Recap from Call 1, with additions)

Buying Strategy Principle

Common Mistake

3.11 EBS Volume Optimization (Quick Recap)

3.12 EKS Security Scanning — Tool Overview

Three Tools Covered

3.13 Tool 1 — Kube-hunter (Penetration Testing)

Purpose

How It Works

Running a Manual Scan

Sample Output (from session)

Step-by-Step (Live Demo: `sre-labs-asg-demo`, T3.medium → T3a.medium)

Step-by-Step (Live Demo: `sre-labs-eks`, node group `ng-intel`, T3.medium → T3a.medium)

Python — Start Lambda (`start_lambda.py`)

Python — Stop Lambda (`stop_lambda.py`)