SRE Labs (Advanced Track) — Project Call 1: AWS Cost Optimization (Client: HealthCorp)

Structured educational resource covering sre labs (advanced track) — project call 1: aws cost optimization (client: healthcorp).

senior 45 min read 12 sections

#kubernetes#cloud-k8s#aws#gcp

Complete Learning Package (AWS FinOps · EC2 · EKS · Storage · Networking · FinOps Tooling)

Source: 2026-01-24-19-03-57.md — SRT transcript of the first Sunday project call of a 6-week DevOps training program called SRE Labs (Advanced Track). The session introduces the program’s structure, the client (HealthCorp — a Texas-based healthcare organization), the AWS cost inventory, a layered optimization framework, and a live demo of migrating EC2 instances from Intel to AMD architecture.

Framing note (read before trusting anything): This is a first-pass introductory call, not a technical deep-dive. Many specifics (scripts, Terraform files, architectural diagrams, savings-plan mathematics) were promised for later uploads and are not in this transcript. Where the session was interrupted by Q&A and left topics incomplete, I note it explicitly. Gap-fills from AWS engineering knowledge are clearly labelled in Section 14.

2. Table of Contents

Executive Summary
Table of Contents
Detailed Structured Notes
- 3.1 Program Structure — SRE Labs (Advanced Track) (6 weeks, 4 pillars)
- 3.2 The Four Pillars of Real DevOps
- 3.3 Project Series Overview (all 6 projects)
- 3.4 Client Profile — HealthCorp (Healthcare)
- 3.5 AWS Infrastructure Inventory — What HealthCorp Was Running
- 3.6 Cost Baseline and Starting Point
- 3.7 The Optimization Framework — Layered Approach
- 3.8 Foundational Steps Before Any Optimization
- 3.9 Tagging Strategy
- 3.10 Native AWS Cost Tools to Enable
- 3.11 EC2 Cost Optimization — Full Coverage
- 3.12 CPU Architecture Families — Intel vs AMD vs ARM
- 3.13 Architecture Assessment Tool (AWS Porting Advisor for Graviton)
- 3.14 EC2 Migration Categories (Standalone / ASG / EKS Node Group)
- 3.15 Live Demo — Migrating a Standalone EC2 from Intel to AMD
- 3.16 Post-Migration Validation and Verification
- 3.17 EKS Cost Optimization Tools
- 3.18 Storage (EBS, S3) Cost Optimization
- 3.19 Savings Plans — Concepts and Mathematics
- 3.20 Networking Cost Optimization
- 3.21 Monitoring and Logging Cost Optimization
- 3.22 Security Tooling Cost Optimization
- 3.23 Other Services (RDS, ElastiCache, DocumentDB, ECR → Harbor)
- 3.24 Live Client Engagement — Polomial (GCP)
- 3.25 Playground vs Production — What’s Replicated and What Isn’t
Key Concepts Table
Architecture & Workflow Analysis
Commands, Scripts & Configs
Tools & Technologies
Real-World Production Usage
Interview Preparation (Beginner / Intermediate / Advanced)
Exam & Certification Notes
Cheat Sheet
Gaps, Assumptions & Incomplete Areas
Gap-Fill — What the Session Left Unfinished, Completed Here

3. Detailed Structured Notes

3.1 Program Structure — SRE Labs (Advanced Track)

SRE Labs (Advanced Track) is a 6-week Sunday project-call program pairing theory with real client implementations. Each week maps to a project; each project maps to one of four DevOps pillars (cost, security, availability, scalability). The sessions include:

Live walk-through of a real client’s infrastructure and decisions.
A minimal playground replication participants can experiment on.
Production outages / failure case studies encountered in the project.
War-room-style debugging (separate Saturday sessions).
Weekly assignments (uploaded Mondays).

Two separate call types:

Saturday (“war room” / production-outage drills): Replicate the original infrastructure that the organization follows — production-grade setup.
Sunday (project calls): Walk through past real-client configurations, issues, and solutions.

Key distinction the instructor drew: Shadowing (live client call, e.g., Polomial on weekdays) = implementation live on client infra. Project calls = past client configurations/decisions + minimal replication in the playground. These are parallel tracks, not the same thing.

3.2 The Four Pillars of Real DevOps

Pillar	Core question	Maps to project
Cost	Are we spending cloud money efficiently?	HealthCorp AWS cost optimization
Security	Is the infrastructure protected?	SecureAsset fintech security audit
Availability	Can it stay up under failure/load?	500-microservice MNC (high availability)
Scalability	Can it grow with traffic?	Automation + AI project, MNC scale-out

3.3 Project Series Overview

#	Client / Project	Focus	Scheduled
1	HealthCorp (healthcare, Texas)	AWS cost optimization — all layers	Session 1 (this call)
2	SecureAsset (fintech, Grow ecosystem, India)	Security audit on real infrastructure (GCP + AWS/EKS)	Week 2
3	Polomial (blockchain, India)	Live client engagement — GCP cost opt → security hardening → GCP-to-AWS migration (50% workload)	Weekly weekdays
4	MNC (unnamed, India)	500+ microservices; CI/CD at scale; Terraform at enterprise scale; full observability (system + user behavior + latency)	~8 Feb (moved up)
5	CleverOps (internal)	Automation of DevOps repetitive tasks + self-healing infra + AI/MLOps integration	~22 Feb
6	Case study (own cloud platform)	MLOps (Kubeflow, MLflow); cloud-platform architecture; AI infrastructure	End of series

3.4 Client Profile — HealthCorp (Healthcare)

Type: Healthcare organization
Location: Texas, USA
Workload: Full AWS stack — compute, serverless, DB, storage, networking, security, monitoring
Environment model (problem): Single AWS account for prod, dev, and staging — no isolation.
Engagement: Cost optimization only. Instructor’s team was onboarded specifically for cost — not responsible for application design or architectural decisions. Original DevOps team was still in place.
Timeline: Engagement happened ~1 year prior to this session; access revoked since. Documentation, scripts, and inventory preserved locally.
Outcome: ~80% reduction in monthly AWS bill claimed.

3.5 AWS Infrastructure Inventory — What HealthCorp Was Running

The full inventory across all AWS services used:

Compute:

EC2 (~170 instances, majority Intel-based, mix of memory-optimized/compute-optimized/general)
EKS (Elastic Kubernetes Service)
Lambda (serverless)
ECS (Elastic Container Service)
Lightsail (commitment-based compute, similar to EC2)

Storage:

EBS (volumes, snapshots, GP2/GP3)
S3 (multiple tiers/lifecycle)
ECR (Elastic Container Registry — later migrated to Harbor)
Snapshots (idle volumes, unattached)

Networking:

NAT Gateway
VPC Peering
ALB (Application Load Balancer)
External/Elastic IPs
Global Accelerator

Database / Middleware:

RDS (PostgreSQL, SQL Server, MariaDB)
DynamoDB
DocumentDB
RedShift
ElastiCache (Redis/Memcached)
SQS, SNS, SCS

Monitoring / Logging:

CloudWatch (log groups, metrics)

Security:

AWS Inspector
GuardDuty
WAF (Web Application Firewall)
IAM

Other:

Savings Plans
Cost Explorer, Compute Optimizer, Trusted Advisor
Cost Anomaly Detection

3.6 Cost Baseline and Starting Point

Environment	Monthly Cost
Production	~$45,000
Non-production (dev + staging)	~$53,000
Total (all environments)	~$98,000

Key insight: The non-production bill was nearly larger than production — a strong signal that dev/staging instances were running 24×7 with no scheduling, and were likely heavily overprovisioned since dev workloads are bursty and low-traffic.

Savings achieved (claimed): ~80% of total monthly bill.

3.7 The Optimization Framework — Layered Approach

The instructor explicitly recommends approaching AWS cost optimization layer by layer, in priority order based on cost impact:

Layer 1: COMPUTE
   ├── Server Computing: EC2, EKS, Lightsail
   └── Serverless: Lambda, ECS (Fargate)

Layer 2: NETWORK
   ├── NAT Gateway
   ├── VPC Peering / Data Transfer
   ├── ALB / ELB
   └── External/Elastic IPs

Layer 3: STORAGE / DATA
   ├── EBS (volumes, snapshots, GP2→GP3)
   └── S3 (lifecycle, tiering, Glacier)

Layer 4: MONITORING / LOGGING
   └── CloudWatch (log groups, retention, storage)

Layer 5: OTHER TOOLS
   ├── RDS / ElastiCache / DocumentDB
   ├── GuardDuty, Inspector, WAF
   └── ECR / artifact stores

Ordering rationale: Pick the layer consuming the most cost first, OR start from compute (usually the biggest driver) and work downward. The layer model ensures nothing is missed — the same mental model as the OSI-layer troubleshooting approach but applied to cost engineering.

3.8 Foundational Steps Before Any Optimization

Step 1 — Export a full inventory (by account, by environment, by resource type). HealthCorp used a Python/AWS SDK script to extract every resource’s metadata, utilization metrics, and current cost into a consolidated Excel workbook with one sheet per resource category.

Step 2 — Enable native AWS tools. (See Section 3.10.)

Step 3 — Implement tagging. (See Section 3.9.)

Step 4 — Categorize resources into:

Overprovisioned (underutilized) → downsize or switch architecture
Properly utilized → leave or optimize architecture only
Underprovisioned (overloaded) → upsize (rare during cost optimization engagements)

Step 5 — Establish a baseline period. Typically 90 days minimum; 365 days+ for stable production workloads with seasonal variation. Use CloudWatch metrics (CPU, RAM, storage I/O) over this window to identify trends and categorize instances. An attendee correctly noted that 90 days can miss seasonal or new-feature traffic spikes — baselining is a continuous and evolving process, not a one-time exercise.

Step 6 — Define and document the optimization plan per layer, per service, with expected savings in percentage and dollars.

3.9 Tagging Strategy

”Tagging is the most important thing if you are talking about cost engineering in any cloud provider.”

Mandatory tags for every resource (regardless of how it was created — console, Terraform, CLI):

Tag Key	Example Value	Purpose
`Name`	`healthcorp-prod-eks-node-1`	Human-readable identification
`Owner`	`devops-team`	Accountability for the resource
`ENV`	`prod` / `dev` / `staging`	Environment isolation; enables cost split and shutdown scripts
`LOB`	`LOB-1` / `LOB-2`	Line of Business — assigns cost to the team that owns it

Why LOB matters: In organizations with multiple business units, each team is accountable for their cloud spend. Without LOB tags, you cannot produce a team-level cost report or audit. When teams are audited they are asked why their spend deviated from budget — without tags, this is impossible to answer.

Automation benefit: Shutdown scripts for non-prod environments operate by tag — ENV=dev → stop all tagged instances at 10 PM IST, restart at 6 AM IST.

3.10 Native AWS Cost Tools to Enable

Enable these before any optimization work begins:

Tool	What it does	Priority
Cost Explorer	Visualizes cost by service, resource, tag, time range; filter by instance ID, account, region	First
Compute Optimizer	Recommends right-size for EC2, Lambda, EKS, Auto Scaling Groups; based on CloudWatch utilization data	Second
Trusted Advisor	Five categories: Cost Optimization (idle/underused resources), Security (MFA, open ports), Performance (autoscaling, health checks), Service Limits (quota monitoring), Operational Excellence (AWS best practices)	Third
Cost Anomaly Detection	Alerts on unexpected cost spikes per service/account; configurable thresholds	Fourth

Cost Explorer — practical usage for post-migration validation:

Filter by resource → EC2 → filter by instance ID.
Filter by instance type (paste the new AMD/ARM type).
Group by day to compare before/after migration costs.
Export or screenshot for client reporting.

Trusted Advisor — key outputs:

Idle/underutilized EC2 instances → terminate or stop.
Unattached EBS volumes → delete.
Underutilized RDS → downsize or delete.
IAM users without MFA → security gap.
Open security groups (0.0.0.0/0 on 22/3389) → security gap.
High-utilization resources approaching quota → scale warning.

3.11 EC2 Cost Optimization — Full Coverage

The instructor listed every lever that exists. Grouped for clarity:

Architecture-level:

Migrate Intel → AMD (A-suffix instances): ~15–20% cost saving, minimal risk.
Migrate Intel/AMD → ARM/Graviton (G-suffix): ~30–40% cost saving, requires compatibility check.
Migration approach: assess → plan → categorize → migrate by category → monitor → verify.

Commitment-based:

Savings Plans (Compute or EC2): commit to a $/hr spend for 1 or 3 years; applies across instance families/regions.
Reserved Instances: commit to a specific instance type; larger discount but less flexible than Savings Plans.
3-year + All Upfront = deepest discount; 1-year + No Upfront = most flexible.
Do not buy blindly. There is a calculator and dedicated mathematics for this (covered in a dedicated section of a future call).

Sizing:

Right-sizing: analyze 90–365 days of CloudWatch CPU/RAM/disk/network data; downsize overprovisioned; upsize underprovisioned (rare in cost context).
Tools: Compute Optimizer, Trusted Advisor, third-party tools (Spot.io, CastAI, StormForge).

Scheduling:

Non-prod shutdown: stop dev/staging instances off-hours (e.g., 10 PM–6 AM IST) via Lambda + EventBridge or a Jenkins-hosted script (avoids Lambda cost for the scheduler itself).
Tag-based targeting: ENV=dev → stop; ENV=prod → never touch.
Production instances: do not schedule stop/start unless explicitly cleared by the business.

Instance type switching:

Spot Instances for non-critical workloads: significant savings (up to 70–90%), but no availability guarantee — not suitable for production.
Lightsail: commitment-based compute, predictable billing; useful for stable, small workloads.

Availability Zone (AZ) optimization:

If instances are spread across multiple AZs unnecessarily, consolidate to reduce cross-AZ data transfer costs.
Note: Never compromise high-availability topology for cost — this requires explicit business sign-off.

Autoscaling:

Properly sized Auto Scaling Groups with min/max/desired ratios reduce idle capacity.
Scale-in during low-traffic periods; scale-out on demand.

Platform (OS) optimization:

Windows licenses are included in EC2 pricing and are significantly more expensive than Linux.
If the application can run on Linux, switch: immediate per-hour savings.

Stopped/idle instances:

Stopped instances still incur EBS charges. Identify and terminate truly unused instances.

Public IPs:

Elastic IPs not associated with a running instance are billed. Release unassigned EIPs.

3.12 CPU Architecture Families — Intel vs AMD vs ARM

Architecture	AWS naming	Relative cost	Notes
Intel (x86_64)	No suffix (T3, M5, C5…)	Highest	Default when creating instances; most widely used historically
AMD (x86_64)	`a` suffix (T3a, M5a, C5a…)	~15–20% cheaper than Intel equivalent	Same x86_64 instruction set; compatible with Intel binaries for 99% of workloads
ARM / Graviton	`g` suffix (M6g, C7g, T4g…)	~30–40% cheaper than Intel equivalent	Different instruction set; requires ARM-compatible binaries / dependencies

Identification shortcut:

Instance name ending in a (e.g., c5a.4xlarge) → AMD
Instance name ending in g (e.g., m6g.large) → Graviton/ARM
No letter suffix (e.g., t3.medium) → Intel

The HealthCorp example:

c5.4xlarge (Intel) → c5a.4xlarge (AMD): cost per hour $0.80 → $0.62 = $131/month saved per instance.
Across 120 instances: ~$935/month saved (architecture migration only).
ARM migration across eligible instances: additional ~$1,500/month.
Total from architecture change alone on 170 instances: ~$2,400+/month.

Migration safety rule:

Intel → AMD: Very safe. Both x86_64. Almost always a drop-in replacement. Verify with a 48–72 hr monitoring window.
Intel/AMD → ARM: Requires explicit compatibility assessment (see Section 3.13). Dependencies must be checked. Can migrate directly Intel → ARM, but Intel → AMD → ARM is safer (smaller blast radius per step).

3.13 Architecture Assessment Tool — AWS Porting Advisor for Graviton

Purpose: Scan application code dependencies for ARM incompatibilities before migrating to Graviton.

How it works:

Install the tool locally (as a Python package or run as a Docker image).
Point it at the project directory.
It scans third-party library/dependency files (e.g., package.json for Node, requirements.txt for Python, pom.xml for Java) — not the application source code itself.
It reports which libraries have x86-specific native binaries or known ARM incompatibilities, with line numbers.
Output: a report identifying which packages are incompatible → developer changes those packages/versions → re-scan until clean → proceed with migration.

Workflow:

Developer: confirm application logic has no arch-specific code
DevOps: run AWS Porting Advisor → fix dependency issues
DevOps: test application on ARM instance in non-prod
DevOps: migrate prod instance after 7-day monitoring

Best practice: Developers own the assessment (they know the code). DevOps validates at the infrastructure level. Both perspectives are needed before migrating production.

3.14 EC2 Migration Categories

When migrating EC2 instances between architectures (Intel→AMD or Intel→ARM), instances fall into three operational categories, each with a different migration strategy:

Category	Description	Migration complexity
Standalone instances	Individual EC2 not in any ASG or EKS node group	Simplest (Intel→AMD). Hardest (AMD→ARM — requires re-image).
ASG-managed instances	EC2 in an Auto Scaling Group	Intermediate (Intel→AMD). Simpler (AMD→ARM — update Launch Template; ASG replaces nodes).
EKS node group instances	EC2 forming Kubernetes worker nodes	Most complex (Intel→AMD — drain, cordon, replace). Simplest (AMD→ARM — update managed node group AMI).

Important: The complexity is inverted between Intel→AMD and AMD→ARM. Standalone is easiest for Intel→AMD but hardest for AMD→ARM (because you must re-image the whole node). EKS is hardest for Intel→AMD (node drain/cordon required) but easiest for AMD→ARM (Kubernetes manages the replacement).

3.15 Live Demo — Migrating a Standalone EC2 Instance from Intel to AMD

Exact steps demonstrated in the AWS console:

Phase 1: Identify the equivalent instance type

Current:    t3.medium   (Intel, x86_64)
Equivalent: t3a.medium  (AMD, x86_64)

Rule: same family, same size, add a suffix → equivalent AMD instance.

Phase 2: Take an AMI backup

In EC2 console → select the instance → Actions → Image and Templates → Create Image.
Name: IntelInstance-backup-<YYYY-MM-DD-HH:MM> (date+time in name for auditability).
Description: "Backup taken while migrating Intel instance to AMD on <date>".
Attach the EBS volume to the image.
Add tags: Name, Owner, ENV, LOB.
Click Create Image.
Wait until the AMI status is Available before proceeding. Do not skip this step in production.

Phase 3: Stop the instance

Actions → Instance State → Stop.
Wait until status = stopped.

Phase 4: Change instance type

Actions → Instance Settings → Change Instance Type.
Enter t3a.medium → confirm.
The console shows the new per-hour price for verification.

Phase 5: Start the instance

Actions → Instance State → Start.
Instance comes up on AMD architecture.

Estimated downtime: ~1–2 minutes (stop + change + start). Plan for a maintenance window.

Phase 6: Monitor for 48–72 hours

Watch application performance, error rates, CloudWatch metrics.
If issues arise: stop instance → change back to t3.medium → start (rollback using the same change-instance-type flow, or launch from AMI backup).

Phase 7: Delete AMI backup after ~1–2 weeks

AMI storage costs money. Once confirmed stable, delete the backup.
Do not forget: the goal is cost reduction, not adding AMI snapshot storage costs.

Phase 8: Verify in Cost Explorer

Navigate to Cost Explorer → EC2 → filter by instance ID → group by day.
Compare yesterday’s cost (Intel) vs. today’s cost (AMD).
Screenshot the cost difference for client reporting.

3.16 Post-Migration Validation and Verification

Action	Timing	Purpose
Monitor application performance	First 48–72 hours	Catch arch-specific runtime issues
Check error rates / logs	First 48–72 hours	Catch dependency or binary compatibility issues
Verify cost in Cost Explorer	After ~24 hours	Confirm actual billing reflects new instance type
Performance test (load test)	Before and after migration	Quantitative evidence of no degradation
Delete AMI backup	After 1–2 weeks stable	Remove snapshot storage cost
Update Terraform / IaC	Immediately	Keep infra-as-code reflecting actual state
Document and report savings	Weekly cadence	Client accountability; monthly reporting

Production best practice (from attendee): All instance changes in production should go through Terraform / Pulumi / IaC, not through console click-ops, for auditability, state file integrity, and rollback capability. In HealthCorp’s case, the original team used click-ops exclusively — the instructor acknowledged this is suboptimal and would be fixing it going forward.

3.17 EKS Cost Optimization Tools

The session named these tools but deferred the deep-dive to a future call:

Tool	Type	What it does
Karpenter	Open-source (AWS)	Node autoscaler; provisions the right instance type (cheapest that fits pod requirements) rather than generic ASG scaling
CastAI	Third-party SaaS	Full cluster cost optimization: right-sizing, spot orchestration, autoscaling, multi-cloud; also works on non-EKS clusters
Kubecost	Open-source / SaaS	Kubernetes cost visibility and monitoring per namespace/deployment/pod
StormForge (implied name)	Third-party	Similar to CastAI but with different feature set
EKS node group architecture migration	Native AWS	Managed node group AMI update to ARM/Graviton; simplest path for AMD→ARM

Comparison mentioned: Karpenter alone vs. CastAI alone vs. Karpenter + CastAI together — the session plans a live analysis of all three combinations.

3.18 Storage Cost Optimization

EBS:

Identify and delete idle/unattached volumes. An unattached EBS volume continues billing. HealthCorp saving opportunity: ~$232/month from EBS cleanup alone.
Check assigned vs. used capacity. A 100 GB volume using 10 GB: either shrink or move data and re-attach a smaller volume.
Migrate GP2 → GP3: GP3 is cheaper per GB and offers better IOPS performance baseline. Straightforward for most workloads.
Volume shrinking procedure: Detach → snapshot → create new smaller volume from snapshot → attach. Cannot shrink in place; requires the snapshot round-trip.

S3:

Implement S3 Lifecycle Policies: automatically transition objects from Standard → Standard-IA (infrequent access) → Glacier (archive) based on age.
Rule of thumb: objects not accessed in 30 days → Standard-IA; not accessed in 90 days → Glacier or Glacier Deep Archive.
Significant savings at scale (logging buckets, backup buckets).

3.19 Savings Plans — Concepts and Mathematics

The session flagged this as important and complex, with a ~30–40 page dedicated document and its own calculator. Key concepts introduced:

Two types:

Compute Savings Plans: Commit to a $/hr spend on compute; applies to EC2, Lambda, Fargate across all instance families, regions, and OS. Most flexible.
EC2 Instance Savings Plans: Commit to a specific instance family in a specific region. Less flexible but deeper discount.

Commitment options (affecting discount depth):

Term	Payment	Discount
1 year	No upfront	~20–30%
1 year	Partial upfront	~25–35%
1 year	All upfront	~30–40%
3 years	All upfront	Deepest discount (~40–60%)

Critical rule: Savings Plans apply only to On-Demand instances, not Spot. If you buy a plan expecting to cover Spot usage, it doesn’t work.

The mathematics the session deferred: How to analyze your baseline On-Demand spend, project future utilization, and determine the correct $/hr commitment to buy to maximize coverage without over-committing (unused Savings Plan commitment is wasted money). The AWS Savings Plans calculator + Cost Explorer “Recommendations” tab helps, but understanding the logic manually is essential.

What happens if you overbuy a Savings Plan? You’re locked in for 1–3 years. The plan cannot be resold directly (AWS Marketplace for Reserved Instances exists, but Savings Plans have no secondary market). You can reduce the effective loss by ensuring maximum On-Demand coverage, but you cannot cancel.

3.20 Networking Cost Optimization

Named for coverage but deferred to a later session:

NAT Gateway: Often the biggest surprise billing item. Every GB of data processed by a NAT Gateway is billed. Use VPC endpoints (S3, DynamoDB) to route traffic internally without NAT.
VPC Peering vs. Transit Gateway: Peering is cheaper for simple topologies; Transit Gateway adds cost but simplifies multi-VPC/multi-account.
Cross-AZ traffic: Data transferred between AZs is billed. Minimize unnecessary cross-AZ calls within an application.
ALB vs. NLB: ALBs are billed per LCU (Load Balancer Capacity Unit) — consolidate where possible.
Elastic IPs: Any EIP not associated with a running instance is billed. Audit and release.
Global Accelerator: Expensive per data transfer; validate whether latency improvement justifies cost vs. CloudFront.

3.21 Monitoring and Logging Cost Optimization

The Polomial live-client engagement had a ~$15,000/month CloudWatch logging bill alone. This was the focus of a separate live session. Key levers:

Log group retention policies: CloudWatch logs stored indefinitely by default. Set retention to 7/14/30/90 days based on compliance requirements.
Log export to S3 → Glacier: Export older logs to S3 and then transition to Glacier for cold storage at a fraction of CloudWatch pricing.
Metric filter reduction: Eliminate unused custom metrics and alarms.
Log level discipline: Applications logging at DEBUG level in production generate 10–100× the log volume of INFO/WARN. Set production log levels appropriately.
Third-party alternatives: Self-hosted Loki + Grafana as a cheaper alternative to CloudWatch for application logs.

3.22 Security Tooling Cost Optimization

Mentioned but not detailed in this session:

GuardDuty pricing: Based on GB of analyzed data (VPC Flow Logs, DNS logs, CloudTrail). Large environments pay significant amounts.
Inspector: Per-instance assessment pricing; schedule for non-prod vs. continuous for prod.
WAF: Per rule, per million requests.
Third-party security tools (unnamed) will be covered in the security audit project.
ECR → Harbor migration: ECR has per-GB storage and data-transfer costs. Self-hosted Harbor on EKS eliminates per-image-pull costs within the cluster. HealthCorp executed this migration as part of their cost optimization.

3.23 Other Services

Service	Optimization lever mentioned
RDS	Right-size instances; choose correct engine (PostgreSQL < SQL Server for licensing cost); use Aurora Serverless v2 for variable workloads; optimize storage auto-scaling
ElastiCache	Right-size node type; use reserved nodes for stable workloads
DocumentDB	Covered in inventory; optimization deferred
RedShift	Covered in inventory; optimization deferred
ECR	Migrated to Harbor (self-hosted on EKS) — eliminates ECR pull costs
Lambda	Rightsize memory (Lambda pricing scales with GB-seconds); move cron/scheduler Lambdas to Jenkins-hosted scripts to save Lambda invocation cost
SQS/SNS	Usually low-cost; audit message retention and throughput

3.24 Live Client Engagement — Polomial (GCP)

A separate, parallel track from the Sunday project calls. Polomial is a blockchain-based organization with infrastructure on GCP.

Three phases:

Phase 1 (current): GCP cost optimization on pre-prod environment.
Phase 2: Security hardening on GCP.
Phase 3: Migrate 50% of workloads from GCP to AWS (for high availability / multi-cloud).

Format: 1–2 calls per week on weekdays, live in the client’s GCP console. Participants can contribute to the actual client engagement. Hiring opportunity mentioned.

Yesterday’s activity: Exported inventory from the Polomial GCP console using scripts, producing the same type of inventory workbook as HealthCorp’s.

3.25 Playground vs Production — What’s Replicated

The session produced an extended Q&A on this. Clarified expectations:

Aspect	Production (real client)	Playground (training)
Infrastructure scale	500 microservices, R6/X1 instances	3–4 representative microservices
Instance types	Large (r6.16xlarge, x1, etc.)	Small (T3/T3a medium) — architecture mirrors prod; size doesn’t
Network topology	Full (ALB, ingress, DNS, Route53)	Mirrors topology but on smaller scale
Real traffic	Yes (1M+ TPS)	Simulated via load tools
Application code	Real (secrets removed)	Full code on GitHub
Architecture shape	Full (500 services, all add-ons)	3–4 services but same upstream/downstream flow
What you analyze	Real 500-service environment (read-only view)	Playground environment (hands-on)

Key nuance: For the MNC project, participants will see and analyze the real 500-microservice production environment, but implement and experiment only in the playground (3–4 services replicating the same topology).

4. Key Concepts Table

Concept	Explanation	Example	Why It Matters
FinOps / Cost Engineering	Discipline of managing cloud spend as a business metric, not an afterthought	HealthCorp $98K/month → $20K target	Cloud costs compound; unmanaged infra can 5–10× over time
Single account anti-pattern	All environments (prod/dev/stage) in one AWS account	HealthCorp’s original setup	No cost isolation, no blast radius containment, no billing attribution
Resource tagging	Metadata attached to every cloud resource for attribution, automation, and billing	`ENV=dev`, `LOB=LOB-1`	Foundation of all FinOps: without tags, cost attribution is impossible
Compute Optimizer	AWS tool that analyzes utilization and recommends right-sized instances	”Downsize m5.xlarge to m5.large — 80% savings, < 40% CPU used”	Removes guesswork from right-sizing
Trusted Advisor	AWS tool across 5 pillars: cost, security, performance, limits, ops excellence	Lists idle EIPs, unattached EBS, MFA gaps	Free health check across cost + security in one place
Cost Anomaly Detection	Alerts on unexpected cost spikes per service	”EC2 cost spiked 300% vs yesterday”	Catch runaway costs before end-of-month bill shock
Intel → AMD migration	Switching EC2 instance from Intel (no suffix) to AMD (`a` suffix)	`t3.medium` → `t3a.medium`	~15–20% savings; minimal risk; 99% of workloads are compatible
Intel → ARM/Graviton migration	Switching to ARM-based Graviton instances (`g` suffix)	`m5.large` → `m6g.large`	~30–40% savings; requires compatibility assessment
AWS Porting Advisor for Graviton	Tool that scans application dependencies for ARM incompatibilities	Scans `package.json`, `pom.xml`, `requirements.txt`	Prevents production breakage from incompatible native binaries
Standalone EC2 instance	Instance not managed by ASG or EKS node group	A single Nginx reverse-proxy EC2	Simplest to migrate Intel→AMD; harder AMD→ARM
ASG-managed instance	EC2 in an Auto Scaling Group	Web tier with min 2, max 10	Intermediate complexity; update Launch Template
EKS node group instance	EC2 as Kubernetes worker node	`ng-general-1` node group	Complex Intel→AMD (drain/cordon); simple AMD→ARM (AMI update)
AMI backup	Amazon Machine Image snapshot before migration	Pre-migration backup	Rollback option if migration causes issues
Right-sizing	Adjusting instance type to match actual utilization	Downsize r5.4xlarge (8% CPU used) to r5.large	Typically the fastest-payback optimization
Savings Plans	Commit to $/hr compute spend for 1–3 years for discount	$1,000/hr commitment for 3yr all-upfront	Deep discounts (40–60%); must be sized carefully
GP2 → GP3 migration	Upgrade EBS volume type	Old volumes on GP2; GP3 is cheaper and faster	Reduces EBS cost and improves baseline IOPS
S3 lifecycle policy	Auto-transition S3 objects to cheaper tiers over time	Standard → S3-IA (30d) → Glacier (90d)	Log/backup buckets often grow unbounded
Non-prod scheduling	Stop dev/staging instances outside business hours	10 PM–6 AM shutdown	~67% compute saving on non-prod (8 hours billed vs 24)
Spot instances	Spare AWS capacity at up to 90% discount; interruptible	Batch jobs, CI/CD workers	Not for production; perfect for stateless non-critical workloads
Harbor vs ECR	Self-hosted registry vs AWS managed	HealthCorp migrated ECR to Harbor	Eliminates per-image-pull ECR costs inside the cluster
Karpenter	AWS open-source node autoscaler for EKS	Provisions cheapest node that fits pod	Replaces managed ASG node groups; big cost wins on EKS
CastAI	Third-party EKS/GKE/AKS optimizer	Full cluster right-sizing + spot orchestration	Combines multiple optimization levers; paid but ROI typically positive
Baselining	Defining the “normal” utilization pattern for a workload	90-day CPU/RAM trend for right-sizing	Without baseline, right-sizing is guesswork; baseline evolves continuously

5. Architecture & Workflow Analysis

5.1 HealthCorp Environment — Before and After

BEFORE (Single Account, No Tagging)
┌─────────────────────────── AWS Account (Single) ─────────────────────────────┐
│  Production                  Dev                      Staging                │
│  ─────────────               ───                      ───────                │
│  170 EC2 (Intel)     +   untagged EC2     +    untagged EC2                  │
│  EKS (Intel nodes)   +   running 24×7     +    running 24×7                  │
│  RDS (multi-engine)                                                           │
│  ECR (container registry)                                                     │
│  S3 (no lifecycle)                                                            │
│  EBS (GP2, unattached volumes)                                                │
│  CloudWatch (unlimited log retention)                                         │
│  NAT GW, ALB, Elastic IPs                                                    │
│                                            Monthly Total: ~$98,000           │
└──────────────────────────────────────────────────────────────────────────────┘

AFTER (Multi-account, Tagged, Optimized)
┌── Prod Account ──────────────────────────────┐
│  EC2 (AMD/ARM, right-sized, tagged)          │
│  EKS (Graviton nodes, Karpenter + CastAI)    │
│  RDS (right-sized, PostgreSQL preferred)      │
│  Harbor (replaces ECR)                        │
│  S3 (lifecycle policies → Glacier)           │
│  EBS (GP3, no idle volumes)                  │
│  CloudWatch (retention set, logs exported)   │
│  NAT GW → VPC Endpoints where possible       │
│  Savings Plans (correctly sized 3yr)         │
└──────────────────────────────────────────────┘
┌── Non-Prod Account ──────────────────────────┐
│  EC2 (AMD, smaller, auto-stop 10PM–6AM IST)  │
│  Tagging: ENV=dev/staging, LOB=<team>        │
└──────────────────────────────────────────────┘
                 Monthly Total: ~$20,000 (target)

5.2 Layered Cost Optimization Workflow

START
  │
  ├─ Export full inventory (AWS SDK script → Excel workbook)
  ├─ Enable: Cost Explorer + Compute Optimizer + Trusted Advisor + Anomaly Detection
  ├─ Audit and enforce tagging on all resources
  │
  ▼
LAYER 1: COMPUTE
  ├─ EC2:
  │    ├─ Categorize: standalone / ASG / EKS node group
  │    ├─ Architecture: Intel→AMD (all safe); Intel→ARM (post-assessment)
  │    ├─ Right-size (90-day baseline → Compute Optimizer)
  │    ├─ Schedule non-prod shutdown (10PM–6AM)
  │    ├─ Spot for non-critical batch/CI
  │    └─ Savings Plans (compute or EC2; 3yr all-upfront if stable)
  ├─ EKS:
  │    ├─ Karpenter (node autoscaler)
  │    ├─ CastAI (holistic optimizer)
  │    └─ Kubecost (visibility)
  └─ Lambda/ECS (Fargate):
       └─ Memory right-sizing; move schedulers off Lambda to Jenkins
  │
  ▼
LAYER 2: NETWORK
  └─ NAT GW → S3/DynamoDB VPC Endpoints | EIP audit | ALB consolidation
  │
  ▼
LAYER 3: STORAGE / DATA
  └─ EBS: GP2→GP3 | delete idle volumes | shrink oversized | snapshot cleanup
     S3: lifecycle → Standard-IA → Glacier
  │
  ▼
LAYER 4: MONITORING / LOGGING
  └─ CloudWatch: retention policies | export to S3 | reduce debug log volume
  │
  ▼
LAYER 5: OTHER TOOLS
  └─ ECR→Harbor | RDS right-size | ElastiCache right-size | GuardDuty scope
  │
  ▼
VERIFY & REPORT
  └─ Cost Explorer: before/after per service/resource/day
     Client report: savings in $ and % per layer
     Document in Terraform; update tagging policies
     Establish ongoing monitoring (Anomaly Detection alerts)

5.3 EC2 Instance Migration Flow (Intel → AMD, Standalone)

1. ASSESS application compatibility (developer + AWS Porting Advisor)
2. IDENTIFY equivalent AMD instance type  (T3.medium → T3a.medium)
3. CREATE AMI backup  → wait for status = Available
4. STOP instance  → wait for status = stopped
5. CHANGE INSTANCE TYPE  → T3a.medium (confirm price shown in console)
6. START instance
7. MONITOR application 48–72 hrs  (error rate, latency, CloudWatch)
8. VERIFY cost in Cost Explorer  (before vs after per-day billing)
9. DELETE AMI backup  (after 1–2 weeks stable)
10. UPDATE Terraform/IaC  (reflect new instance type)
11. REPORT savings to client

6. Commands, Scripts & Configs

The scripts and Terraform modules were promised for upload post-session and are not in the transcript. What is captured below comes from what was described or shown.

AWS CLI — Describe and filter EC2 instances

# List all EC2 instances with their instance type, state, and tags:
aws ec2 describe-instances \
  --query 'Reservations[*].Instances[*].[InstanceId,InstanceType,State.Name,Tags[?Key==`ENV`].Value|[0],Tags[?Key==`Name`].Value|[0]]' \
  --output table

# Filter by tag ENV=dev (for non-prod shutdown script):
aws ec2 describe-instances \
  --filters "Name=tag:ENV,Values=dev" \
  --query 'Reservations[*].Instances[*].InstanceId' \
  --output text

# Stop all dev-tagged instances:
aws ec2 stop-instances \
  --instance-ids $(aws ec2 describe-instances \
    --filters "Name=tag:ENV,Values=dev" "Name=instance-state-name,Values=running" \
    --query 'Reservations[*].Instances[*].InstanceId' \
    --output text)

# Start all dev-tagged instances:
aws ec2 start-instances \
  --instance-ids $(aws ec2 describe-instances \
    --filters "Name=tag:ENV,Values=dev" "Name=instance-state-name,Values=stopped" \
    --query 'Reservations[*].Instances[*].InstanceId' \
    --output text)

Lambda scheduler (non-prod shutdown) — Python skeleton

import boto3

def lambda_handler(event, context):
    ec2 = boto3.client('ec2', region_name='us-east-1')
    action = event.get('action', 'stop')  # 'stop' or 'start'
    env_tag = 'dev'

    instances = ec2.describe_instances(
        Filters=[
            {'Name': 'tag:ENV', 'Values': [env_tag]},
            {'Name': 'instance-state-name',
             'Values': ['running'] if action == 'stop' else ['stopped']}
        ]
    )
    ids = [i['InstanceId']
           for r in instances['Reservations']
           for i in r['Instances']]

    if ids:
        if action == 'stop':
            ec2.stop_instances(InstanceIds=ids)
        else:
            ec2.start_instances(InstanceIds=ids)
    return {'action': action, 'instances': ids}

Terraform — EC2 instance type change (IaC best practice)

resource "aws_instance" "healthcorp_app" {
  ami           = "ami-xxxxxxxxxx"   # Ubuntu 22.04 ARM (for Graviton)
  instance_type = "t3a.medium"       # AMD; was t3.medium (Intel)

  tags = {
    Name  = "healthcorp-dev-app-server"
    Owner = "devops-team"
    ENV   = "dev"
    LOB   = "LOB-1"
  }
}

AWS Porting Advisor for Graviton

# Option 1: Python package
pip install aws-graviton-porting-advisor
porting-advisor --output text /path/to/your/application

# Option 2: Docker
docker run --rm -v /path/to/app:/app \
  public.ecr.aws/graviton-porting-advisor/porting-advisor \
  --output text /app

Cost Explorer CLI — per-instance daily cost

aws ce get-cost-and-usage \
  --time-period Start=2026-01-20,End=2026-01-25 \
  --granularity DAILY \
  --metrics "UnblendedCost" \
  --filter '{
    "And": [
      {"Dimensions": {"Key": "SERVICE", "Values": ["Amazon Elastic Compute Cloud - Compute"]}},
      {"Tags": {"Key": "Name", "Values": ["healthcorp-prod-app-server"]}}
    ]
  }' \
  --group-by Type=DIMENSION,Key=INSTANCE_TYPE

EC2 instance type change via AWS CLI

# Stop the instance
aws ec2 stop-instances --instance-ids i-0abcdef1234567890

# Wait for it to stop
aws ec2 wait instance-stopped --instance-ids i-0abcdef1234567890

# Change instance type
aws ec2 modify-instance-attribute \
  --instance-id i-0abcdef1234567890 \
  --instance-type '{"Value": "t3a.medium"}'

# Start the instance
aws ec2 start-instances --instance-ids i-0abcdef1234567890

EBS volume audit — find unattached volumes

aws ec2 describe-volumes \
  --filters "Name=status,Values=available" \
  --query 'Volumes[*].[VolumeId,Size,VolumeType,CreateTime]' \
  --output table

S3 lifecycle policy (JSON)

{
  "Rules": [
    {
      "ID": "LogsLifecycle",
      "Status": "Enabled",
      "Filter": {"Prefix": "logs/"},
      "Transitions": [
        {"Days": 30,  "StorageClass": "STANDARD_IA"},
        {"Days": 90,  "StorageClass": "GLACIER"},
        {"Days": 365, "StorageClass": "DEEP_ARCHIVE"}
      ],
      "Expiration": {"Days": 730}
    }
  ]
}

7. Tools & Technologies

Tool	Category	Purpose	Notes from session
AWS Cost Explorer	Native AWS	Visualize, filter, and analyze AWS spend by service/resource/tag/time	Enable first; used for post-migration validation
AWS Compute Optimizer	Native AWS	Right-size recommendations for EC2, Lambda, EKS	Based on CloudWatch utilization data
AWS Trusted Advisor	Native AWS	Cost + security + performance + limits + ops excellence audit	Paid tiers unlock more checks
AWS Cost Anomaly Detection	Native AWS	Alert on unexpected cost spikes	Set budgets and alerts per service
AWS Savings Plans Calculator	Native AWS	Size Savings Plan commitments correctly	Complex mathematics; dedicated 30-40 page doc
AWS Pricing Calculator	Native AWS	Estimate cost before provisioning or migration	Live-demoed: T3.medium vs T3a.medium comparison
AWS SDK (boto3/CLI)	Native AWS	Script inventory extraction; automate start/stop	Scripts promised for upload
AWS Porting Advisor for Graviton	Native AWS	Scan app dependencies for ARM incompatibility	Run before any Intel→ARM migration
Karpenter	Open-source (AWS)	EKS node autoscaler; provisions right-size cheapest nodes	Alternative to managed node groups
CastAI	Third-party SaaS	EKS/GKE/AKS cost optimization; spot orchestration; right-sizing	Also supports EC2 and DB optimization
Kubecost	Open-source / SaaS	Kubernetes cost visibility per namespace/workload	Essential for chargeback/showback
Harbor	Open-source	Self-hosted container registry	Replaces ECR; eliminates pull costs within cluster
Terraform / Pulumi	IaC	Manage all resource changes; auditability; state management	Production changes must go through IaC
Jenkins	CI/CD	Host non-prod scheduling scripts (avoids Lambda cost for schedulers)	Mentioned as lightweight alternative to Lambda for cron jobs
Kubeflow / MLflow	MLOps	ML pipeline orchestration and experiment tracking	Covered in final case study session

8. Real-World Production Usage

Engagement model: Cost optimization engagements are typically scoped separately from architectural design or application changes. The DevOps consultant/team focuses on infrastructure levers and respects application constraints set by the dev team.

Key production realities surfaced:

Do not automate production instance changes via scripts. Use IaC (Terraform) for auditability and state integrity. Scripts are fine for non-prod automation (stop/start scheduling).
Downtime windows are real. EC2 instance type changes require a stop → change → start cycle. Coordinate with stakeholders and schedule a downtime window. Typical downtime: ~1–2 minutes per instance.
Monitor after every change. 48–72 hrs minimum for architecture changes; 1 week for major right-sizing.
Rollback must be pre-staged. AMI backup must be in Available state before touching the instance.
Baselining is continuous, not a one-time exercise. Traffic patterns change with new features, seasonal peaks, and user growth. Re-baseline periodically.
FinOps is a team sport. FinOps team (or DevOps with FinOps responsibility) monitors, reports, and alerts. Application teams are the decision-makers for their own spend. Budget plans should be set before provisioning resources, not after.
Multi-account architecture is the right pattern. Separate AWS accounts for prod, non-prod, tooling, and FinOps. AWS Control Tower + Landing Zones are the standard enterprise pattern. Budgets and quotas are set per-account.

Cost engineering impact claims (HealthCorp):

Architecture migration (Intel→AMD) across 120 instances: ~$935/month saved.
Architecture migration (Intel→ARM) across eligible instances: ~$1,500/month saved.
EBS cleanup (idle volumes + GP2→GP3): ~$232/month saved.
Non-prod scheduling (dev/staging off 8+ hrs/night): ~30–50% of non-prod compute (substantial).
Total claimed: ~80% of $98K/month = ~$78,400/month reduction.

9. Interview Preparation

Beginner

Q1. What is AWS Cost Explorer and why is it the first tool you enable for cost optimization? A: Cost Explorer is AWS’s native spend visualization tool. It lets you filter by service, resource, tag, account, region, and time granularity. You enable it first because you need historical cost data before you can make any optimization decision — it’s the source of truth for current and trend spend, and the tool you use to validate that your changes actually reduced the bill.

Q2. What is the difference between Reserved Instances and Savings Plans? A: Reserved Instances commit you to a specific instance family, size, and region for 1–3 years; Savings Plans commit you to a $/hr compute spend and can apply across any instance family, size, OS, and region. Savings Plans are more flexible. Both offer the same discount levels at best (3yr all-upfront), but Savings Plans are generally preferred for mixed or evolving workloads. Neither applies to Spot instances.

Q3. What tags should every AWS resource have, and why? A: At minimum: Name (identification), Owner (accountability), ENV (environment — prod/dev/staging), and LOB (line of business for team-level cost attribution). Without tags, you cannot split costs by environment, automate shutdowns by tag, attribute spend to teams, or produce meaningful cost reports.

Q4. What is a Graviton instance and why would you use it? A: Graviton instances (suffix g, e.g., m6g.large) run on AWS-designed ARM processors. They are 30–40% cheaper than equivalent Intel instances. They require ARM-compatible application dependencies, which you verify using the AWS Porting Advisor for Graviton. For 99% of modern workloads (containerized apps, modern runtimes), they are drop-in replacements.

Intermediate

Q5. A client has a $98K/month AWS bill. Walk me through how you’d approach reducing it. A: (1) Export full inventory using AWS SDK scripts. (2) Enable Cost Explorer, Compute Optimizer, Trusted Advisor, Cost Anomaly Detection. (3) Audit and enforce tagging. (4) Work layer by layer — Compute first (usually the biggest driver): categorize EC2 by arch/sizing/scheduling opportunity; analyze EKS with Karpenter/CastAI; right-size Lambda. (5) Network: NAT Gateway → VPC endpoints; release unused EIPs. (6) Storage: GP2→GP3; delete idle EBS; S3 lifecycle. (7) Monitoring: CloudWatch retention + export to Glacier. (8) Other: ECR→Harbor; RDS right-size; Savings Plans after 90-day baseline. Validate every change in Cost Explorer.

Q6. How do you migrate 170 EC2 instances from Intel to AMD with zero unplanned downtime? A: (1) Categorize instances into standalone/ASG/EKS node group — each has a different migration procedure. (2) Verify application compatibility (AWS Porting Advisor + developer sign-off). (3) For each instance: take AMI backup → wait for Available → agree maintenance window with stakeholder → stop instance → change instance type to a equivalent → start → monitor 48–72 hrs. (4) For ASG: update Launch Template with AMD AMI/instance type → trigger rolling replacement. (5) For EKS: cordon + drain old nodes → update node group → new nodes come up on AMD. (6) Verify via Cost Explorer. (7) Delete AMI backups. (8) Update Terraform.

Q7. When is it appropriate to stop EC2 instances to save cost, and when is it not? A: Stop/schedule shutdowns only for non-production instances (dev, staging, testing). Use tag-based automation: ENV=dev → stop at 10 PM IST, start at 6 AM IST. Production instances should never be automatically stopped — use autoscaling to handle load variations instead. Non-prod scheduling typically saves 30–65% of non-prod compute cost (14–16 hours off per day).

Q8. What is MaxSessions / right-sizing, and how do you determine the correct instance size? A: Right-sizing means matching instance CPU, RAM, and storage to actual utilization. The process: collect CloudWatch metrics over a baseline period (90–365 days); use Compute Optimizer’s recommendations; apply a safety buffer (never size below peak + 20% headroom). Trigger for re-baseline: new feature release, traffic pattern change, or when anomaly detection alerts on sustained CPU spike. Baselining is continuous, not a one-time exercise.

Advanced

Q9. Explain the tradeoffs between Compute Savings Plans, EC2 Savings Plans, and Reserved Instances. A: Compute Savings Plans offer the most flexibility — apply to any EC2 family/region/OS, Lambda, and Fargate, but the discount is slightly lower. EC2 Savings Plans are more restrictive (specific family + region) but offer deeper discounts. Reserved Instances are most restrictive (specific instance type) and the deepest discount, but can be sold on the Marketplace if unused. General rule: use Compute Savings Plans for dynamic environments; EC2 Savings Plans or RIs for stable, long-running workloads where the instance family is unlikely to change.

Q10. What are the risks of buying a Savings Plan without doing the mathematics first? A: Over-commitment: you pay for unused compute hours for 1–3 years with no refund or resale option (unlike Reserved Instances). Under-commitment: you leave money on the table — On-Demand pricing for the uncovered portion. The correct process: analyze the past 90-day On-Demand baseline (removing already-covered spend), project the stable portion that won’t scale down, and commit to that $/hr amount, leaving headroom for growth. AWS’s Savings Plans recommendations in Cost Explorer help but should be validated manually.

Q11. How does Karpenter differ from a standard EKS managed node group with Cluster Autoscaler? A: Standard Cluster Autoscaler scales the number of nodes within a fixed node group (fixed instance type). Karpenter provisions the right instance type and size for the specific pods waiting to be scheduled — it reads pod resource requests and picks the cheapest instance that fits, including Spot if appropriate. This eliminates over-provisioning at the node level. Combined with CastAI, which also handles multi-cloud and DB optimization, you get significantly better cost visibility and control than native autoscaling alone.

Q12. How do you implement FinOps for a multi-account AWS organization with 20 accounts? A: Use AWS Organizations + Control Tower. Each account has its own budget and alerts (AWS Budgets). Cost is aggregated in a centralized billing/FinOps account using Cost Explorer’s multi-account view. Tag policies (Service Control Policies) enforce mandatory tags across all accounts. Quotas are set per-account using Service Quotas. FinOps team monitors and reports; application teams own spend decisions. Chargeback/showback happens via tag-based cost allocation reports exported to S3 and loaded into a BI tool.

10. Exam & Certification Notes

For AWS Solutions Architect / DevOps Professional:

Cost-related services: Cost Explorer, Compute Optimizer, Trusted Advisor (Cost pillar), Cost Anomaly Detection, Savings Plans, Reserved Instances — know the differences and when to use each.
Instance naming conventions: No suffix = Intel; a suffix = AMD; g suffix = Graviton/ARM. Commonly tested: “Which is cheapest for same workload?” → ARM.
EBS volume types: GP2 (older, burstable IOPS) vs GP3 (newer, cheaper, predictable 3000 IOPS baseline). Migration from GP2 to GP3 is a well-known exam optimization scenario.
S3 storage classes: Standard → Standard-IA (30+ day old, infrequently accessed) → Glacier (90+ days, archive) → Glacier Deep Archive (1 year+, coldest). Know lifecycle policy syntax.
Savings Plans vs Reserved Instances: flexibility (Savings Plans) vs discount depth (RIs) — exam frequently tests the distinction.
Auto Scaling vs Karpenter: exam favors native autoscaling; real-world favors Karpenter. Know both.
Tagging best practices: mandatory for cost allocation; enforced via Tag Policies in AWS Organizations.

Potential trick questions:

“Savings Plans apply to Spot instances” → False. On-Demand only.
”GP2 has higher IOPS than GP3” → False for the baseline. GP3 provides 3000 IOPS baseline (vs GP2’s burst-only IOPS). GP3 wins on both cost and performance for most workloads.
”A stopped EC2 instance incurs no charges” → False. EBS volumes attached to a stopped instance are still billed.
”Reserve Instances can be applied across instance families” → False. That’s Compute Savings Plans.

11. Cheat Sheet

Start every engagement: Export inventory → Enable Cost Explorer + Compute Optimizer + Trusted Advisor + Anomaly Detection → Enforce tagging (Name/Owner/ENV/LOB)

EC2 architecture cost ladder (cheapest to most expensive, same performance): ARM/Graviton (g) < AMD (a) < Intel (no suffix) — ~30–40% from Intel→ARM; ~15–20% from Intel→AMD

Instance naming: t3.medium = Intel | t3a.medium = AMD | m6g.large = Graviton/ARM

Standalone instance migration (Intel→AMD): AMI backup → stop → change instance type → start → monitor 48–72h → verify Cost Explorer → delete AMI

Migration complexity matrix:

	Intel→AMD	AMD→ARM
Standalone	Easiest	Hardest
ASG	Medium	Medium
EKS node group	Hardest	Easiest

Non-prod scheduling: ENV=dev/staging → stop 10 PM IST, start 6 AM IST (via Jenkins script or Lambda + EventBridge)

Savings Plans rule: Analyze 90-day On-Demand baseline → commit to stable $/hr → 3yr all-upfront = deepest discount → never buy blindly

EBS quick wins: GP2 → GP3 (cheaper + faster) | delete unattached (status=available) | right-size (assigned vs used)

S3 lifecycle: Standard → Standard-IA (30d) → Glacier (90d) → Deep Archive (365d)

Key tools:

Cost Explorer — always open first
Compute Optimizer — right-size recommendations
Trusted Advisor — idle resources + security
Anomaly Detection — spike alerts
Karpenter — EKS node optimization
CastAI — full cluster cost optimization
Kubecost — Kubernetes cost visibility
Harbor — replace ECR for in-cluster pulls
AWS Porting Advisor — ARM compatibility check

Layer order: Compute → Network → Storage → Monitoring/Logging → Other tools

12. Gaps, Assumptions & Incomplete Areas

Deliberately deferred (promised in future sessions):

ASG-managed instance migration demo (Intel→AMD and AMD→ARM)
EKS node group migration demo
Karpenter and CastAI live implementation and comparison
Savings Plans mathematics deep-dive (30–40 page document)
Networking cost optimization (NAT Gateway, VPC Endpoints, etc.)
CloudWatch log cost optimization (Polomial live client sessions)
RDS, ElastiCache, DocumentDB right-sizing
Security tooling cost (GuardDuty, Inspector, WAF)
ECR→Harbor migration walkthrough
Non-prod scheduling script walkthrough (Lambda + EventBridge vs Jenkins)
Terraform modules for instance changes
AWS SDK inventory extraction scripts
Architectural diagram for HealthCorp (to be created by instructor)
Multi-account architecture / AWS Organizations / Control Tower

Transcription artifacts:

“Light cell” / “light sail” = Lightsail
”Carpenter” = Karpenter
”Cast AI / Castai” = CastAI
”Cube cost” = Kubecost
”Studioious” / other garbled name = StormForge (likely, context-fit)
“Terapform” = Terraform
”Genkins” = Jenkins
”Kota” / “kota” = quota
”LOB” = Line of Business (tag)
“Click ops” = ClickOps (managing cloud via console UI rather than IaC)
“Polomial” = Polymial or similar (blockchain client name — exact spelling unclear)

Claims to treat as heuristics, not facts:

“99% of workloads are AMD/ARM compatible” — accurate for modern containerized apps; less true for legacy JNI, compiled native binaries, or hardware-specific libraries. Always verify.
”80% cost reduction achieved” — real claim but unverified; no external audit cited.
”Intel→AMD saves 15–20%” — directionally accurate per AWS pricing; exact delta varies by family and region.
”Intel→ARM saves 30–40%” — directionally accurate; same caveat.

Open questions from attendees (unanswered or partially answered):

How to implement budget restrictions per-account in a multi-account AWS org (Control Tower / SCP / Budgets — answered by an attendee, not the instructor in depth).
How to simulate production-level traffic in the playground for realistic testing.
What specific CloudWatch metrics constitute “overprovisioning” vs “underprovisioning” with exact thresholds.

13. Gap-Fill — What the Session Left Unfinished, Completed Here

These are filled from AWS engineering knowledge and FinOps practice. Clearly labelled as gap-fill, not transcript content.

GAP 1 — The Savings Plans Mathematics (the 30-page document, summarized)

The instructor deferred this to a dedicated document. Here is the core methodology.

Step 1: Determine your On-Demand spend baseline. Go to Cost Explorer → Service: EC2/Lambda/Fargate → Payment Option: On-Demand. Filter out already-covered spend (existing RIs or Savings Plans). Take the average $/hour over the past 90 days. This is your coverage baseline.

Step 2: Identify the stable portion. Stable = compute that runs consistently 24×7 without scale-in. For a workload averaging $10/hr with a trough of $6/hr: the stable floor is $6/hr. Commit to at most the trough, not the average — this avoids over-commitment during low-usage periods.

Step 3: Choose the right plan type.

Workload changes instance family often → Compute Savings Plan
Workload is stable in one region/family → EC2 Instance Savings Plan
Includes Lambda/Fargate → Compute Savings Plan only

Step 4: Choose term and payment.

3-year All Upfront:     deepest discount (~57% vs On-Demand for some families)
1-year All Upfront:     middle (~40%)
1-year No Upfront:      smallest (~30%) but no cash outlay

For stable production workloads with predictable usage: 3-year all-upfront maximizes ROI. For uncertainty: start with 1-year, no-upfront, and upgrade.

Step 5: Validate using AWS Recommendations. Cost Explorer → Savings Plans → Recommendations. AWS shows the recommended commitment, estimated savings, and coverage percentage. Cross-check manually.

Step 6: Purchase and monitor coverage. After purchase, monitor Savings Plans utilization (target: >80%) and coverage rate (target: >90% of On-Demand spend covered). Low utilization = over-bought. Low coverage = room to buy more.

Warning: Do not purchase Savings Plans during the first 30–60 days of a new environment. Wait until usage patterns stabilize.

GAP 2 — ASG Instance Migration (Intel→AMD)

The session described the category but did not demonstrate it. Here is the correct procedure.

For instances in an Auto Scaling Group:

Identify the current Launch Template or Launch Configuration used by the ASG.
Create a new Launch Template version with the AMD equivalent instance type.
Update the ASG to use the new Launch Template version.

Trigger a rolling instance refresh:

aws autoscaling start-instance-refresh \
  --auto-scaling-group-name <asg-name> \
  --preferences MinHealthyPercentage=80,InstanceWarmup=300

ASG gradually replaces old Intel instances with new AMD instances, maintaining minimum healthy count.
Monitor CloudWatch + application metrics during refresh.
If issues arise, cancel the refresh: aws autoscaling cancel-instance-refresh --auto-scaling-group-name <asg-name>

Terraform approach:

resource "aws_launch_template" "app" {
  name_prefix   = "healthcorp-app-"
  instance_type = "t3a.medium"   # Changed from t3.medium
  image_id      = "ami-xxxxxxxxxx"
  # ... rest of config unchanged
}

resource "aws_autoscaling_group" "app" {
  # ...
  launch_template {
    id      = aws_launch_template.app.id
    version = "$Latest"
  }
  instance_refresh {
    strategy = "Rolling"
    preferences {
      min_healthy_percentage = 80
    }
  }
}

GAP 3 — EKS Node Group Migration (Intel→AMD and AMD→ARM)

Intel→AMD (most complex):

Create a new managed node group with AMD instance type and AMD-compatible AMI.
Cordon the old Intel nodes to prevent new pod scheduling: kubectl cordon <node-name>
Drain the old Intel nodes to migrate pods safely: kubectl drain <node-name> --ignore-daemonsets --delete-emptydir-data
Pods are rescheduled onto new AMD nodes by the scheduler.
Once all pods are rescheduled and old nodes are empty, delete the old node group.

AMD→ARM (simplest — this is why the complexity matrix is inverted):

Update the managed node group to use a Graviton AMI and ARM instance type.
EKS handles the rolling replacement of nodes automatically (similar to ASG instance refresh).
Ensure all container images are multi-arch (linux/amd64 + linux/arm64) or ARM-only.
Check all DaemonSets for ARM compatibility (e.g., monitoring agents, security agents may have x86-only binaries).

Multi-arch container images:

# Build and push multi-arch image:
docker buildx create --use
docker buildx build \
  --platform linux/amd64,linux/arm64 \
  -t your-registry/your-app:latest \
  --push .

GAP 4 — NAT Gateway Cost Reduction (the Networking Layer)

NAT Gateway is often a surprise large cost item, especially as logging and inter-service traffic grows.

The problem: Any EC2 or EKS pod in a private subnet making an outbound request to the internet (S3, ECR, DynamoDB, external APIs) routes through the NAT Gateway. You pay $0.045/GB of data processed.

The fix — VPC Endpoints (Gateway + Interface):

Service	Endpoint type	Cost	Impact
S3	Gateway endpoint	Free	Eliminates S3 NAT cost entirely
DynamoDB	Gateway endpoint	Free	Eliminates DynamoDB NAT cost entirely
ECR	Interface endpoint	~$0.01/GB	Reduces ECR pull NAT cost (worth it at scale)
Secrets Manager	Interface endpoint	~$0.01/GB	Reduces Secrets Manager NAT cost
CloudWatch Logs	Interface endpoint	~$0.01/GB	Reduces logging NAT cost

# Create a free S3 Gateway endpoint (eliminates S3→NAT traffic):
aws ec2 create-vpc-endpoint \
  --vpc-id vpc-xxxxxxxxxx \
  --service-name com.amazonaws.us-east-1.s3 \
  --route-table-ids rtb-xxxxxxxxxx

# Terraform:
resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.${var.region}.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = [aws_route_table.private.id]
}

For large logging volumes (e.g., Polomial’s $15K/month CloudWatch bill): the primary driver is often EKS pods → CloudWatch Logs via NAT Gateway. A CloudWatch Logs VPC Interface Endpoint removes that NAT cost and often reduces the CloudWatch bill simultaneously.

GAP 5 — The Overprovisioning Detection Framework

An attendee pressed on this without getting a clear answer. Here is the engineering standard:

Metric thresholds for right-sizing decisions:

Metric	Overprovisioned signal	Underprovisioned signal
CPU utilization	P99 < 40% over 90d	P99 > 85% consistently
Memory utilization	P99 < 40% over 90d	P99 > 85% consistently
Network throughput	< 10% of bandwidth available	Sustained saturation
EBS IOPS	< 20% of provisioned IOPS	IOPS wait time > 1ms sustained
Storage	Used < 50% of provisioned	Used > 85% of provisioned

Safety buffer: Always keep 30–40% headroom above the measured P99 peak. Do not size to the average — size to the peak + buffer.

Baselining cadence:

New workloads: wait 30 days before right-sizing.
Stable workloads: re-evaluate every 6 months or after any significant traffic event.
Before any performance-impacting change: document the current metrics.

Tools:

AWS Compute Optimizer: automated recommendations with confidence level.
AWS CloudWatch: pull P50/P99 CPU/RAM over 90 days for each instance.
Third-party: Spot.io, StormForge, CastAI (all provide automated continuous right-sizing).

GAP 6 — Multi-Account AWS Strategy (Control Tower + FinOps)

The attendee question about budget restrictions per-account was answered partially by a peer. Full picture:

AWS Organization (Management/Root Account)
├── FinOps/Billing Account
│   ├── Cost Explorer (consolidated, all accounts)
│   ├── Budgets (per-account, per-service, per-tag)
│   └── Cost Anomaly Detection (org-wide)
├── Security Account
│   ├── CloudTrail (org-wide)
│   ├── GuardDuty (delegated admin)
│   └── Security Hub
├── Shared Services Account
│   ├── Transit Gateway
│   ├── Shared VPCs
│   └── ECR / Harbor
├── Production Account
│   ├── Budget: $45,000/month
│   ├── Alert: 80% threshold → notify FinOps
│   └── SCP: cannot create instances > r6.2xlarge without approval
├── Staging Account
│   └── Budget: $5,000/month; auto-stop all instances at 11 PM UTC
└── Dev Account
    └── Budget: $3,000/month; auto-stop all instances at 10 PM IST

Budget enforcement (not just alerting): AWS Budgets can trigger Lambda actions to stop instances or send SNS notifications when thresholds are breached. SCPs (Service Control Policies) can prevent resource creation above a certain size or prevent resources from being created in non-approved regions.

FinOps chargeback workflow:

All accounts tag resources with LOB.
Cost Allocation Tags report in Cost Explorer groups by LOB.
Monthly report exported to S3 → loaded into BI tool (QuickSight, Tableau).
Each LOB team receives their cost statement and must justify deviations > 10% from budget.