SRE Labs (Advanced Track) — Project Call: HealthCorp AWS Cost Optimization (Session 2)

Structured educational resource covering sre labs (advanced track) — project call: healthcorp aws cost optimization (session 2).

senior 45 min read 11 sections

#kubernetes#cloud-k8s#aws#gcp

ASG & EKS Node Group Migration, Right-Sizing, Spot, Savings Plans, and EKS Security Scanning (kube-hunter/kubescape/kube-bench)

2. Table of Contents

Executive Summary
Table of Contents
Detailed Structured Notes
- 3.1 Session Recap & Agenda
- 3.2 ASG Intel→AMD Migration — Live Demo
- 3.3 EKS Node Group Intel→AMD Migration — Live Demo
- 3.4 AMD→ARM Migration — Approach & Assignment
- 3.5 Non-Production Environment Shutdown
- 3.6 Right-Sizing — The P-R-C Framework
- 3.7 On-Demand → Spot Migration
- 3.8 Savings Plans (Recap + Compute vs. EC2)
- 3.9 EKS Security Scanning — kube-hunter, kubescape, kube-bench
- 3.10 Extended Q&A Highlights
- 3.11 Program Logistics
Key Concepts Table
Architecture & Workflow Analysis
Commands & Configurations
Tools & Technologies
Real-World Production Usage
Interview Preparation (Beginner / Intermediate / Advanced)
Exam & Certification Notes
Cheat Sheet
Gaps & Assumptions

3. Detailed Structured Notes

3.1 Session Recap & Agenda

Quick recap of the prior session: HealthCorp’s infrastructure was divided into layers (compute, storage, monitoring/logging, networking, DB/security), and EC2 cost optimization began with architecture migration — starting with the standalone Intel→AMD case, achieving roughly a 30–40% cost saving opportunity purely from the architecture change.

This session’s agenda:

Complete architecture migration: ASG and EKS node group categories (Intel→AMD live demo), plus an overview of AMD→ARM.
Non-production environment shutdown.
Right-sizing instances.
On-demand → Spot migration.
Savings plans (recap/deepening).
EKS security scanning (kube-hunter, kubescape, kube-bench) as a prerequisite to EKS cost optimization.
(Deferred to next session due to time: full EKS cost optimization — Karpenter, CastAI, Kubecost.)

3.2 ASG Intel→AMD Migration — Live Demo

Core concept: An Auto Scaling Group (ASG) doesn’t define instance configuration directly — it references a Launch Template (LT), which defines instance type, VPC/subnet config, security groups, and everything else about the instances the ASG creates. To migrate the ASG’s instances to a different architecture, you don’t edit the ASG directly — you create a new version of its launch template with the updated instance type, then point the ASG at that new version.

Step-by-step procedure demonstrated:

Navigate to the ASG, identify its attached Launch Template.
Open the launch template → Actions → Modify template (Create new version).
Give the new version a clear name/description (e.g., “Intel to AMD LT version”).
The only configuration value that changes is the instance type — e.g., T3.medium → T3a.medium (found by searching “AMD equivalent of T3.medium”). All other settings (VPC, security groups, etc.) are carried over unchanged from the base version.
Create the new template version.
Go back to the ASG’s Launch Template configuration and update the ASG to use the new version (this doesn’t yet create any new instances — it only changes what the ASG would use going forward).
Go to the ASG’s Instance Refresh tab and Start Instance Refresh.
Critical choice here: you’re offered options controlling how the refresh sequences termination vs. launch. For a production-safe, zero-downtime migration, choose the option where new (AMD) instances are launched and confirmed healthy before the old (Intel) instances are terminated — this was explicitly clarified in Q&A as “launch and then terminate,” as opposed to simultaneous “terminate and launch,” which is more cost-efficient (shorter overlap) but risks a brief capacity gap.
Health/warm-up configuration: minimum healthy percentage should be set to at least 80% for production-appropriate safety, and instance warm-up time should be a minimum of 5 minutes, giving new instances time to become fully healthy before old ones are torn down.
The refresh process takes roughly 5 minutes; the ASG automatically creates new AMD instances, verifies health, then terminates the old Intel instances.
Rollback: simply re-point the ASG at the previous (Intel) launch template version and run another instance refresh — the old LT version is retained specifically to serve as a rollback path, so it should not be deleted.
Post-migration verification: confirm actual cost impact in Cost Explorer, filtered by the specific instance ID, comparing daily cost before/after — e.g., an instance costing ~$20/day before, dropping to ~$16–17/day after, translating to roughly $100–200/month in savings for that single instance once verified.
Observation period: minimum 72 hours of monitoring post-migration before considering the change final.

3.3 EKS Node Group Intel→AMD Migration — Live Demo

This is explicitly the most complex of the three migration categories, and — unlike the standalone and ASG cases — requires roughly 2–3 minutes of downtime, because it fundamentally depends on the Kubernetes scheduler’s normal pod-rescheduling behavior, which is inherently sequential.

Prerequisite analysis (must be done before any changes):

Determine whether the workloads on this node group are stateless or stateful. Stateful workloads make this migration substantially harder; the demo and general guidance apply cleanly to stateless workloads.
Inspect the node group’s current configuration — specifically its Kubernetes labels and taints (these determine which pods the scheduler will place on it) and its underlying ASG name (every EKS node group is backed by an ASG under the hood).
Identify the node group’s subnets — via aws eks describe-nodegroup (or equivalent), the new node group must use the same subnets.
Identify the node group’s IAM node role — each cluster/node group has its own; needed for creating the replacement node group. Important gotcha: when creating a new node group via CLI, an instance profile is auto-created for you — the launch template used for the new node group must have “do not include instance profile” explicitly selected, or the creation command will error out.

Migration steps:

Create a new Launch Template version (same process as the ASG case) with the instance type changed to the AMD equivalent, and the instance-profile setting corrected as noted above.
Create a brand-new node group from scratch (not modifying the existing one) — using: the same subnets identified above, the new LT version, the same scaling configuration (min/max/desired, copied directly from the existing ASG’s settings), and the correct node role.
At this point, you have two node groups running simultaneously (Intel and AMD) — no applications have moved yet.
Apply the same labels and taints from the old (Intel) node group onto the new (AMD) node group, so the Kubernetes scheduler will treat them as equally eligible placement targets based on the app’s own scheduling constraints.
Drain the old (Intel) node group: kubectl drain — this evicts all running pods from those nodes.
Cordon the old (Intel) node group: marks it as unschedulable, so the scheduler will never place new pods there again.
Result: as pods are evicted from the Intel nodes, the scheduler — finding no eligible Intel nodes (now cordoned) but finding the new AMD node group with matching labels/taints — automatically reschedules those pods onto the AMD node group. This rescheduling window is the source of the ~2–3 minutes of downtime.
Rollback procedure (if needed): reverse the same sequence — drain the AMD node group, cordon it, then uncordon the Intel node group — triggering the scheduler to move pods back, with a similar ~2–3 minute downtime window.
Post-migration verification: same as before — confirm actual savings via Cost Explorer filtered by instance ID.

Explicit clarification on application impact (Intel→AMD specifically): since Intel and AMD share the same underlying x86 architecture, there is no application compatibility risk in this specific migration — unlike a migration to ARM/Graviton (a genuinely different architecture, “64-bit” in the instructor’s simplified framing — see Gaps & Assumptions in the earlier session’s package for a note on this same simplification), which does require dependency compatibility verification.

3.4 AMD→ARM Migration — Approach & Assignment

Application compatibility check is mandatory and must come first. Two complementary approaches:
1. Developer consultation — ask the developers directly whether the application has any known architecture-specific dependencies. If they confirm none, you can generally proceed with more confidence.
2. Tooling verification — AWS Porting Advisor for Graviton. Installable as a local package (scans your codebase, e.g., a GitHub repo, generating an HTML report) or run as a Docker image. It flags specific lines/dependencies with architecture-specific issues (e.g., a “pre-processor error” tied to x86-specific code) that would block clean ARM compatibility.
Neither approach alone is sufficient — the tool catches technical dependency issues but doesn’t understand application intent; the developer knows intent but might miss subtle technical dependencies. Use both.
Migration path guidance reaffirmed: direct Intel→ARM is possible but carries a larger blast radius; the staged Intel→AMD→ARM path is safer. (Consistent with the framework established in the first HealthCorp session.)
Migration-category difficulty reverses for AMD→ARM (as previously established): standalone is now the hardest category — because, unlike Intel→AMD (a simple in-place instance-type swap), migrating a standalone instance to ARM effectively requires creating a new instance from scratch and manually migrating the application to it (there’s no simple “change instance type” path across genuinely different architectures for a standalone instance the way there is within the x86 family). ASG and EKS node group migrations remain comparatively straightforward, following essentially the same procedures already demonstrated for Intel→AMD.
Live assignment given to participants: using an already-migrated AMD instance (from the standalone case in the prior session), perform and record the AMD→ARM migration for standalone, ASG, and EKS node group cases independently (individually or in squads), using a simple demo application (e.g., nginx) — submitted as an assignment deliverable.
A follow-up live demo specifically on diagnosing and fixing ARM-incompatible application code (using the Porting Advisor’s output) was requested by multiple participants as the genuinely valuable “hard part” of this whole migration category, and the instructor committed to covering this in the Tuesday doubt class, with a recording to be shared afterward.

3.5 Non-Production Environment Shutdown

Rationale: non-production (dev/QA/staging) workloads often run 24/7 despite only being needed during working hours — a direct, low-complexity cost-saving opportunity.
Implementation options demonstrated:
1. Lambda + EventBridge: Python scripts (Lambda doesn’t support Bash) that start/stop instances filtered by an environment tag (e.g., nonprod), triggered on a schedule via EventBridge — example schedule given: start at 8:00 a.m. IST, stop at 10:00 p.m. IST.
2. Jenkins-hosted scripts (Bash or Python) + Jenkins scheduler: an alternative that avoids the incremental Lambda/EventBridge cost, appropriate for organizations that don’t want to introduce additional billed resources purely to save cost elsewhere.
Important real-world caveat, directly tied to HealthCorp: not every organization has a clean day/night working-hours pattern — HealthCorp’s own engineering team worked round the clock, meaning the naive “shut down outside business hours” pattern didn’t directly apply. The team had to specifically identify a narrower, legitimate optimization scope within that constraint, ultimately saving roughly $900 — smaller than a full 24/7-workload client might see, but still meaningful, and a good illustration that this technique requires understanding the actual business’s operating pattern rather than applying a generic template.

3.6 Right-Sizing — The P-R-C Framework

Core framework introduced: balance three pillars — Performance, Reliability, Cost (P-R-C) — when right-sizing any instance. Optimizing purely for cost while ignoring performance/reliability is explicitly framed as irresponsible.
Primary data sources:
- AWS Compute Optimizer — analyzes historical CPU/memory utilization trends and recommends specific instance-type/size changes with an estimated savings figure (e.g., a live example: downsizing from 4 vCPU/32GB to 2 vCPU/16GB, saving ~$37 based on utilization history).
- GCP equivalent, for cross-reference: Active Assist — described as functioning the same way, based on the same kind of historical utilization data.
Explicit guidance: never right-size based on average utilization alone. Always factor in a 20–30% buffer above observed averages to absorb traffic variability, rather than sizing tightly to the mean — sizing to average, without buffer, risks an outage during any traffic spike.
Explicit, firsthand cautionary example — never right-size during a cloud-to-cloud migration: the instructor described personally working a migration for an organization (referred to as “L&T” in the transcript) from GCP to AWS, where the team also attempted cost optimization/right-sizing simultaneously with the migration — and this caused real production incidents, because different cloud providers have different instance-family capacity characteristics and specifications, even when instance types are nominally “equivalent.” Rule stated explicitly: do not attempt right-sizing optimization at the same time as a cross-cloud migration — separate these activities.
Additional caution: avoid right-sizing when an application’s traffic pattern is highly uncertain/volatile (the example given: baseline traffic around 20,000 but spiking to 100,000–200,000 at certain times) — in these cases, historical averages are a poor guide, and aggressive downsizing risks being caught out by the next spike.
General caution against overly aggressive, repeated downsizing cycles (explicitly called out as a startup anti-pattern): repeatedly shrinking instance sizes week over week without adequate buffer eventually produces chaos when a real traffic event occurs.

3.7 On-Demand → Spot Migration

Mechanical steps (straightforward):

Take an AMI backup (same as the standalone architecture-migration procedure).
Launch a new instance from that AMI.
In the launch configuration’s Advanced Details, change the purchasing option from “None” (on-demand) to Spot.

Where and how to actually use spot — the more important, nuanced part of this section:

Best, uncontroversial use cases: stateless microservices, event consumers/background workers, data engineering/ETL processing jobs, CI/CD build agents (GitHub Actions runners, Jenkins agents), and — without any real caveat — all of dev, QA, and test environments.
Production use is explicitly not off-limits, contrary to a common assumption — but requires deliberate architectural design:
- Minimum 2+ replicas for any production workload placed on spot.
- HPA (Horizontal Pod Autoscaler) and PodDisruptionBudgets (PDBs) properly configured.
- A mixed node pool design — the instructor’s example: 70% spot / 30% on-demand, explicitly clarified (in Q&A) as a node-pool-level split, not an application-level split — i.e., you don’t decide “app A goes on spot, app B goes on-demand”; instead you create two node pools with different purchasing models, and the Kubernetes scheduler places pods according to normal scheduling rules/pressure, naturally distributing load across both pools.
- A fallback instance-type chain: e.g., if T3a.medium capacity is unavailable, fall back to T3.medium, and if that’s also unavailable, fall back to a larger instance type — preventing a spot capacity shortage from causing an outage.
- Graceful shutdown handling in the application/infrastructure to handle spot interruption notices cleanly.
Real companies cited as using spot extensively in production: Pinterest and Slack (per their own published infrastructure strategy blogs) run most of their workloads on spot; Netflix is also cited (per their engineering blog) as a production spot user.
The actual risk isn’t “spot in production” — it’s bad architecture plus spot. The instructor’s explicit framing: a well-architected system (correct fallbacks, correct replica counts, correct disruption budgets) is safe with spot in production; a poorly-architected system is risky even in non-production, and adding spot to bad architecture compounds the risk (“2x chaos”).

3.8 Savings Plans (Recap + Compute vs. EC2)

Two AWS Savings Plan types, re-clarified with a direct comparison question from a participant:
- Compute Savings Plan — the broader option; covers EC2, Lambda, and ECS/Fargate collectively under one commitment.
- EC2 Instance Savings Plan — narrower; covers EC2 only, and specifically only on-demand usage (not spot).
Neither Savings Plan type applies to Spot instances — reaffirmed from the first HealthCorp session.
Design principle: deliberately plan which instances are stable/long-running enough to commit to (candidates for savings-plan coverage via on-demand) versus which are non-critical enough to move to spot — building a considered combination rather than defaulting everything to one purchasing model.
A detailed (~30–40 page) internal documentation guide on savings-plan mathematics/methodology was referenced again (previously mentioned in the first HealthCorp session) as available for participants to study independently, alongside a recorded walkthrough.
Cross-cloud equivalence noted: GCP’s Committed Use Discount (CUD) serves the same purpose/role as AWS Savings Plans.

3.9 EKS Security Scanning — kube-hunter, kubescape, kube-bench

This section was led by co-instructor Ravi, explicitly framed as a prerequisite step before EKS cost optimization — establish the security baseline first.

Tool 1 — kube-hunter (penetration testing):

Purpose: simulates an actual attack on the cluster from the perspective of a compromised pod — i.e., “if an attacker gets access to even a simple pod, what could they do?”
Mechanism: deployed as a scheduled CronJob (demonstrated running daily at 2:00 a.m.); can also be triggered as a one-off manual Job via kubectl create job <name> --from=cronjob/<cronjob-name>.
Output: identifies specific vulnerabilities/exploitation attempts the simulated attacker was able to perform — the live example surfaced five categories including spoofing attempts, access to sensitive interfaces, and attempts to access the API server and credentials.
Remediation workflow: each finding has a specific vulnerability ID (e.g., KHV00002) that can be looked up on the vendor’s (Aqua Security’s) site for detailed remediation guidance.

Tool 2 — kubescape (comprehensive misconfiguration scanning):

Purpose: described as the most comprehensive of the three tools — runs a broad set of control checks against deployed workloads and produces detailed findings with remediation guidance and reference links.
Live example findings on one scanned pod (14 checks run, 8 failed): no CPU limits set (risk: a runaway workload can starve other pods of resources), no memory limits set, no network policy hardening (ingress/egress not restricted), and — flagged as a very common, usually unintentional issue — the container was running as root, which the tool explicitly recommends changing to non-root.
Instructor’s explicit recommendation: if an organization can only realistically adopt one of these three tools, kubescape is the one to prioritize, given its breadth and actionable remediation output.

Tool 3 — kube-bench (CIS benchmark auditing):

Purpose: a benchmarking/auditing tool following the CIS (Center for Internet Security) benchmark — an industry-standard reference.
Output style: simple pass/fail/warning results per check (e.g., confirming the kubeconfig file has appropriately restrictive permissions set to 644, or flagging as a warning that a hostname-override argument isn’t set on kubelet) — less detailed than kubescape’s remediation-oriented output, more of a compliance-audit checklist format.
Live example result: 13 checks passed, 3 warnings, zero failures — framed as an acceptable-but-not-perfect posture (“I wouldn’t say the best because we still have some warnings, but no failures, which is a good sign”).

Operational guidance (applies to all three tools):

All are free and open-source, published by the vendor Aqua Security.
Deployed as CronJobs, so the cluster is scanned on an ongoing, automated basis, not just once.
Resource overhead is minimal — job executions complete in roughly 1–5 minutes, addressing a common objection (“won’t this create load on our cluster?”).
If there’s organizational resistance to running scans during business hours, they can be scheduled during low-traffic windows (e.g., weekends, nights) — but logs must still be reviewed afterward (forwarded to persistent storage like S3, or into an existing monitoring/logging pipeline, since pod logs disappear once the job pod completes).
These tools scan both infrastructure and application-deployed workloads — confirmed directly in Q&A (a participant asked specifically whether application endpoints are scanned; confirmed yes, since kubescape’s example finding was on an application deployment, not a system component).
Recommended to run scans in all environments, but the typical real-world remediation workflow is: fix issues in lower environments first, validate, then apply the same remediation to production.
Can be integrated into infrastructure CI/CD pipelines (explicitly distinguished from application CI/CD) so that security scanning is part of infrastructure provisioning from the start, not bolted on later.
Framed as an increasingly common hard requirement in regulated industries (finance was cited specifically) — some organizations require a clean security scan report as a gating criterion before allowing production deployment.

3.10 Extended Q&A Highlights

Terraform/IaC vs. console/CLI demos: multiple participants raised that production changes are virtually always made through Terraform, not the console — the instructor agreed this is correct practice, clarified that the console/CLI demos in this session are for teaching the underlying logic of each migration (which transfers directly to a Terraform implementation), and confirmed a separately recorded Terraform-based demo of the same migrations exists and would be uploaded/shared to the drive, along with the actual Terraform codebase used.
Handling capacity unavailability during an ASG spot/AMD rollout: don’t block the migration — use multi-AZ node groups, diversified instance types, explicit fallback instance-type chains, and PodDisruptionBudgets to prevent over-draining any single AZ/node group during the transition.
When should an organization actually pursue cost optimization? Named trigger scenarios: a sudden, unexplained cost spike; chronically overprovisioned infrastructure with low utilization (example thresholds given: CPU utilization under ~20%, memory under ~40%); and specifically after any major infrastructure transition — on-prem→cloud migration, VM→EKS/containerization, or monolith→microservices — since these transitions often leave behind provisioning decisions made under different assumptions than the new architecture actually needs.
Spot instance node-pool vs. application-level clarification (detailed in Section 3.7) — a participant’s follow-up concern about all replicas of one application accidentally landing on spot nodes simultaneously was addressed via the fallback-instance-type-chain mechanism, not by avoiding spot.
Advanced Track assignment structure changing starting Week 2: previously, Advanced Track participants were given a pre-built environment to work in; from Week 2 onward, participants are expected to build the environment themselves (using provided environment-creation commands/scripts) before performing the assignment’s troubleshooting/optimization activities — a deliberate increase in hands-on responsibility.
Multiple individual access/lab issues (a stuck Prometheus node-exporter pod, unresolved EKS cluster access) were deferred to dedicated 1:1 follow-up sessions rather than resolved in the group call.

4. Key Concepts Table

Concept	Explanation	Example	Why It Matters
Launch Template (LT) versioning	ASGs and EKS node groups don’t hold instance config directly — they reference a Launch Template, and new versions of that template are how instance configuration changes are safely staged	Creating LT version 3 with `T3a.medium` instead of `T3.medium`, then pointing the ASG at version 3	The old version remains available as an instant rollback path — versioning is what makes this migration pattern safe
ASG Instance Refresh (launch-then-terminate vs. terminate-and-launch)	Two sequencing options for how an ASG replaces instances during a config change	Choosing “launch new, verify healthy, THEN terminate old” for zero-downtime production migrations	The sequencing choice directly determines whether the migration is zero-downtime or has a brief capacity gap
Drain + Cordon (Kubernetes)	`drain` evicts running pods from a node; `cordon` marks a node unschedulable for future pods	Draining and cordoning the Intel EKS node group to force pods onto the new AMD node group	The core mechanism behind EKS node group architecture migration — not a config swap, but a scheduler-driven pod relocation
Migration-difficulty inversion (standalone vs. ASG vs. EKS, by direction)	Standalone is easiest for Intel→AMD but hardest for AMD→ARM; EKS/ASG show the opposite pattern	Confirmed again in this session for AMD→ARM	A non-intuitive but consistently reinforced fact worth memorizing deliberately, not inferring by analogy
P-R-C Framework (Performance, Reliability, Cost)	A three-pillar balance to maintain when right-sizing any instance	Never right-size purely for cost without checking performance/reliability impact	Prevents cost optimization from degrading the system it’s meant to make more efficient
Never right-size during a cloud migration	Different cloud providers have different instance-family capacity characteristics even for “equivalent” instance types	A GCP→AWS migration that also attempted right-sizing caused real incidents (firsthand account)	A hard-won, specific operational rule — worth remembering as a standalone caution, not just general “be careful” advice
Spot node-pool split (infrastructure-level, not application-level)	A “70% spot / 30% on-demand” mix refers to node pool composition, with the Kubernetes scheduler naturally distributing pods across both — not a manual per-application assignment	Two node pools (spot-heavy, on-demand-heavy) in one EKS cluster	A common point of confusion; understanding this correctly is necessary to actually design a safe spot-inclusive production architecture
Fallback instance-type chain (for spot capacity gaps)	A prioritized list of instance types to fall back through if the preferred spot type is unavailable	`T3a.medium` unavailable → try `T3.medium` → try a larger type	Prevents a spot capacity shortage in one specific instance type from causing a scheduling failure/outage
CIS Benchmark auditing (kube-bench)	Industry-standard security configuration checks, evaluated as simple pass/fail/warning	Checking kubeconfig file permissions, kubelet hostname-override argument	A recognized, standard reference point often required for compliance/audit purposes
Vulnerability ID → vendor remediation lookup (kube-hunter)	Each identified vulnerability has a specific ID mappable to vendor documentation for exact remediation steps	`KHV00002` → looked up on Aqua Security’s site	Turns a generic “you have a vulnerability” finding into an actionable, specific fix

5. Architecture & Workflow Analysis

5.1 ASG Migration Workflow (Zero-Downtime)

1. Identify ASG's current Launch Template (LT)
        |
        v
2. Create NEW LT version (only instance type changed: Intel -> AMD equivalent)
        |
        v
3. Point ASG at new LT version (config updated, no new instances yet)
        |
        v
4. Start Instance Refresh
   - Choose: launch new -> verify healthy -> THEN terminate old (zero downtime)
   - Min healthy % >= 80%, warm-up >= 5 min
        |
        v
5. New AMD instances created, health-checked
        |
        v
6. Old Intel instances terminated automatically
        |
        v
7. Verify savings in Cost Explorer (filter by instance ID)
        |
        v
8. Rollback path (if needed): re-point ASG to OLD LT version, refresh again

5.2 EKS Node Group Migration Workflow (~2-3 min downtime)

PRE-REQ ANALYSIS:
  - Stateless workload? (required for straightforward path)
  - Existing node group labels/taints
  - Subnets (must match on new node group)
  - Node IAM role
        |
        v
1. Create new LT version (AMD instance type + "no instance profile" flag)
        |
        v
2. Create BRAND NEW node group from scratch
   (same subnets, new LT, same min/max/desired scaling config)
        |
        v
3. Apply SAME labels/taints to new (AMD) node group as old (Intel) node group
        |
        v
   NOW: Two node groups running in parallel, no apps moved yet
        |
        v
4. kubectl drain <Intel node group>   -- evicts running pods
        |
        v
5. kubectl cordon <Intel node group>  -- marks unschedulable
        |
        v
6. Scheduler automatically places evicted pods onto AMD node group
   (matching labels/taints, now the only eligible schedulable target)
        |
        v
   ~2-3 min of downtime during this rescheduling window
        |
        v
7. Verify savings in Cost Explorer
        |
        v
Rollback (if needed): drain+cordon AMD -> uncordon Intel -> scheduler reverses

5.3 Migration Difficulty Matrix (Full, Both Directions)

                    Intel -> AMD              AMD -> ARM
                    -----------              -----------
Standalone            EASIEST                  HARDEST
                  (simple stop/change-        (must manually create new
                   type/start; zero            instance + manually
                   downtime)                   migrate app; different
                                                architecture entirely)

ASG                   MEDIUM                   MEDIUM
                  (LT versioning +          (same LT versioning +
                   instance refresh;         instance refresh process,
                   zero downtime)            AMD->ARM app compatibility
                                              already verified)

EKS Node Group        HARDEST                  EASIEST
                  (new node group +         (Kubernetes drain/cordon
                   drain/cordon;             mechanism naturally
                   ~2-3 min downtime)        handles the swap once new
                                              ARM node group exists)

5.4 Spot Instance Production Architecture Pattern

EKS Cluster
        |
   -------------------------------
   |                             |
Node Pool A                  Node Pool B
(~70% Spot)                  (~30% On-Demand)
   |                             |
Fallback chain:              Stable baseline
T3a.medium ->                capacity
T3.medium ->
larger type
   |                             |
   -------------------------------
                |
                v
        Kubernetes Scheduler
        (distributes pods per normal
         scheduling rules/pressure --
         NOT a manual per-app assignment)
                |
                v
   Production workload requirements:
   - 2+ replicas minimum
   - HPA configured
   - PodDisruptionBudget configured
   - Graceful shutdown on spot interruption

5.5 EKS Security Scanning Pipeline

EKS Cluster
        |
   CronJob: kube-hunter (daily, e.g. 2 AM)
   CronJob: kubescape
   CronJob: kube-bench
        |
        v
Each produces findings:
  kube-hunter  -> vulnerability IDs (e.g. KHV00002) -> vendor lookup -> remediation
  kubescape    -> control-check failures + built-in remediation guidance
  kube-bench   -> CIS benchmark pass/fail/warning checklist
        |
        v
Logs forwarded to persistent storage (S3 / monitoring pipeline)
   (since pod logs disappear once job completes)
        |
        v
Remediation workflow: fix in LOWER environments first -> validate -> apply to PROD

6. Commands & Configurations

Command / Config	Purpose	Explanation
Launch Template → Actions → Modify template → Create new version	Create a new instance-config version without altering the live/active version	Used for both the ASG and EKS node group Intel→AMD migrations
ASG → Launch Template configuration → Edit → select new LT version → Update	Point an ASG at the new launch template version	Doesn’t create new instances by itself — only updates what the ASG will use
ASG → Instance Refresh → Start Instance Refresh	Trigger the ASG to replace its instances according to the current (new) LT version	Choose the launch-then-terminate sequencing for zero-downtime; set min healthy % ≥80%, warm-up ≥5 min
`aws eks describe-nodegroup --cluster-name <cluster> --nodegroup-name <name> --region <region>` (pattern, output as text)	Retrieve the subnets attached to an existing EKS node group	Used to ensure the new (AMD) node group uses identical subnets
`eksctl create nodegroup` / `aws eks create-nodegroup` (pattern) — with min/max/desired matching the existing ASG, correct node role, new LT version, “do not include instance profile” set	Create a brand-new EKS node group from scratch	The core step in EKS node group migration — building the parallel AMD node group before any pods are moved
Applying labels/taints to a node group (via Console UI or `kubectl label node` / node group config)	Make the new node group’s scheduling eligibility match the old node group’s	Necessary so the scheduler treats the new node group as a valid placement target for existing workloads
`kubectl drain <node>`	Evict all running pods from a node/node group	First step in forcing workload migration off the old node group
`kubectl cordon <node>`	Mark a node/node group as unschedulable for future pods	Prevents the scheduler from placing any new pods back on the node group being retired
`kubectl uncordon <node>`	Reverse a cordon, making a node schedulable again	Used as part of the rollback procedure
AMI creation from an existing instance, then Launch instance from AMI → Advanced Details → Purchasing option: Spot	Migrate an instance from on-demand to spot pricing	The mechanical steps for the on-demand → spot migration covered in this session
AWS Cost Explorer — filter by service (EC2) + instance ID + daily granularity	Verify actual before/after cost impact of any migration	Standard verification step reused across every migration type in this series
AWS Porting Advisor for Graviton (local package install, or Docker image)	Scan application code/dependencies for ARM-architecture compatibility issues	Generates HTML reports flagging specific incompatible lines/dependencies
`kubectl create job <manual-job-name> --from=cronjob/<cronjob-name>`	Trigger a one-off manual run of a scheduled CronJob (e.g., a security scan)	Used to run kube-hunter/kubescape/kube-bench on demand, outside their normal schedule
`kubectl logs <pod-name>`	Retrieve logs from a completed scan job’s pod	Used to review kube-hunter/kubescape/kube-bench findings after a job completes

7. Tools & Technologies

AWS Compute Optimizer

Purpose: Analyzes historical CPU/memory utilization and recommends specific right-sizing changes with estimated savings.
When to use it: As the primary, free, native data source for any right-sizing decision — cross-referenced against actual metrics rather than trusted blindly.
Cross-cloud equivalent: GCP’s Active Assist.

AWS Porting Advisor for Graviton

Purpose: Scans application code and dependencies for ARM (Graviton) architecture compatibility.
When to use it: Before any AMD→ARM (or Intel→ARM) migration, as one of two required verification steps (alongside direct developer consultation).
Advantages: Runs locally as a package or via Docker; produces specific, actionable HTML reports.

kube-hunter

Purpose: Kubernetes penetration-testing tool; simulates an attacker operating from within a compromised pod.
When to use it: As a scheduled (CronJob) or on-demand scan to identify exploitable cluster vulnerabilities from an attacker’s perspective.
Vendor: Aqua Security (free, open-source).

kubescape

Purpose: Comprehensive Kubernetes misconfiguration and security-posture scanning tool, with detailed remediation guidance.
When to use it: As the single highest-priority tool to adopt if only one of the three security tools can realistically be implemented, given its breadth and actionable output.
Vendor: Aqua Security (free, open-source).

kube-bench

Purpose: CIS (Center for Internet Security) benchmark auditing tool for Kubernetes clusters.
When to use it: For standard, recognized compliance-style pass/fail/warning auditing against an industry-standard benchmark.
Vendor: Aqua Security (free, open-source).

Kubernetes `drain` / `cordon` / `uncordon`

Purpose: Core Kubernetes node-lifecycle management commands for safely removing a node from scheduling eligibility.
When to use them: Central to the EKS node group architecture-migration mechanism demonstrated in this session, and broadly useful for any node maintenance/decommissioning workflow.

Karpenter / CastAI / Kubecost (previewed for the next session)

Purpose: EKS-specific cluster autoscaling and cost-optimization tools — same category of tooling previously covered in the FintechPlatform GCP sessions, now being applied to HealthCorp’s AWS/EKS environment.
Status: Deferred to the next session due to time; a recorded backup demo was noted as already available in the drive.

8. Real-World Production Usage

The “launch-then-terminate” ASG instance refresh pattern is a directly reusable, production-grade zero-downtime technique — this is exactly how real teams roll out instance-type or AMI changes across an ASG without a capacity gap, and understanding the health-check/warm-up parameters (≥80% healthy, ≥5 min warm-up) is directly applicable to any real ASG-managed fleet, not just this specific migration scenario.
The drain/cordon-based EKS node group migration pattern is the standard, correct way to retire or replace a node group in any real Kubernetes cluster — not just for architecture migration, but for any node-group-level infrastructure change (e.g., upgrading node AMIs, changing instance types for cost, moving to a different node group entirely).
The firsthand “never right-size during a cloud migration” caution is a genuinely valuable, hard-won lesson that isn’t always obvious from first principles — it’s exactly the kind of operational knowledge that distinguishes an engineer who’s actually run a migration from one who’s only read about the theory.
The spot-in-production discussion, grounded in real company examples (Pinterest, Slack, Netflix), corrects a common industry misconception — many engineers default to “spot is only for non-production,” when in practice, well-known, sophisticated engineering organizations run substantial production workloads on spot, provided the architecture (fallback chains, PDBs, replica counts) is actually designed for it.
Running free, open-source security scanning tools (kube-hunter, kubescape, kube-bench) as scheduled CronJobs is a low-cost, high-value practice directly transferable to any real Kubernetes cluster — the “clean security scan as a production deployment gate” pattern cited for finance-sector organizations is a real, common compliance requirement worth being familiar with regardless of industry.
The clarification that spot/on-demand splits happen at the node-pool level, not the application level, reflects the actual mental model needed to design real Kubernetes infrastructure with mixed purchasing options — getting this wrong (trying to manually pin specific applications to specific purchasing models) leads to unnecessarily complex, hard-to-maintain scheduling configuration.

9. Interview Preparation

Beginner Questions

Q1: What’s the difference between modifying an ASG’s Launch Template directly versus creating a new Launch Template version? A: A Launch Template supports versioning — rather than editing the existing (active) version in place, you create a new version with the desired changes (e.g., a different instance type), while the old version remains fully intact and available. You then point the ASG at the new version explicitly. This preserves the old version as an instant rollback path if the new configuration causes problems.

Q2: What do kubectl drain and kubectl cordon each do, and why are both needed when migrating a node group? A: drain evicts all currently-running pods from a node (or node group), forcing the Kubernetes scheduler to reschedule them elsewhere. cordon marks a node as unschedulable, preventing any new pods from being placed there in the future. Both are needed together: draining alone would remove current pods, but without cordoning, the scheduler could still place new pods back on the same (soon-to-be-retired) nodes.

Q3: Why might a Compute Savings Plan be preferable to an EC2 Instance Savings Plan for an organization using multiple AWS compute services? A: A Compute Savings Plan applies its commitment discount flexibly across EC2, Lambda, and ECS/Fargate collectively, whereas an EC2 Instance Savings Plan only covers EC2 usage. If an organization uses more than just EC2 for compute, a Compute Savings Plan provides broader coverage from a single commitment, rather than needing separate plans (or leaving non-EC2 compute uncovered).

Intermediate Questions

Q4: Walk through why migrating an EKS node group’s architecture requires downtime, while migrating a standalone EC2 instance or an ASG-managed instance typically doesn’t. A: Standalone instances can simply be stopped, have their instance type changed in place, and restarted — a direct, self-contained operation. ASG migrations can achieve zero downtime by launching and health-verifying new instances before terminating old ones (the ASG manages this overlap directly). EKS node group migration is fundamentally different: it depends on the Kubernetes scheduler naturally rescheduling pods after the old node group is drained and cordoned — there’s an inherent window between pods being evicted from the old nodes and being successfully placed and started on the new node group, which is where the ~2–3 minutes of downtime comes from. This is a scheduling-driven process, not a simple infrastructure swap.

Q5: Explain why an engineer might deliberately avoid right-sizing instances during a cloud-to-cloud migration, even though both activities individually seem like sound cost-optimization practice. A: Different cloud providers implement instance families with different underlying capacity characteristics and specifications, even when instance types are marketed as roughly “equivalent” (e.g., a GCP machine type and an AWS instance type with similar vCPU/RAM numbers can still behave differently under real workload conditions). Attempting to simultaneously migrate to a new provider and right-size instances compounds two large sources of behavioral change at once, making it much harder to isolate the cause if something goes wrong — and can directly cause incidents, since you’re both changing the underlying platform and reducing headroom at the same time. The safer practice is to complete the migration first, stabilize, observe real-world behavior on the new platform, and only then evaluate right-sizing separately.

Q6: A team wants to use spot instances for a production Kubernetes workload but is worried about interruption risk. What architectural safeguards would you recommend? A: At minimum: run at least 2 replicas so a single spot interruption doesn’t take the workload fully offline; configure a Horizontal Pod Autoscaler so capacity can adjust to compensate; configure a PodDisruptionBudget so voluntary disruptions (including node draining) don’t remove too many replicas simultaneously; design a fallback instance-type chain so that if your preferred spot instance type becomes unavailable, the system automatically falls back to alternative types rather than failing to schedule; and implement graceful shutdown handling in the application so it can respond cleanly to spot interruption notices. This mirrors how real, sophisticated organizations (Pinterest, Slack, Netflix, per their own published engineering blogs) run substantial production workloads on spot successfully.

Advanced Questions

Q7: Design a safe, staged process for migrating a stateless EKS workload’s underlying node group from Intel to a different processor architecture, including how you’d verify success and how you’d roll back if needed. A: First, confirm the workload is genuinely stateless (a hard prerequisite for a clean migration), and capture the existing node group’s full configuration — labels, taints, subnets, IAM node role, and ASG scaling parameters (min/max/desired). Create a new Launch Template version with the target architecture’s instance type, ensuring the “no instance profile” setting is correctly applied if creating the node group via CLI/eksctl (since node group creation auto-generates one). Create a brand-new node group from scratch using the captured subnet/role/scaling configuration and the new LT version, then apply the same labels and taints as the original node group so it’s an eligible scheduling target for the existing workloads. At this point both node groups coexist with no workload movement yet. To cut over: drain the old node group (evicting its pods) and cordon it (preventing future scheduling there) — the scheduler will then place the evicted pods onto the new, matching, still-schedulable node group, causing a brief (roughly 2–3 minute) downtime window while rescheduling completes. Verify success by confirming application health post-migration and cross-checking actual cost impact in Cost Explorer, filtered by the relevant instance IDs, over a 72-hour observation window. If a rollback is needed, reverse the sequence: drain and cordon the new node group, then uncordon the old node group, letting the scheduler move workloads back — accepting a similar downtime window for the rollback itself.

Q8: How would you decide which specific EC2 instances in a large fleet are good candidates for Savings Plan coverage versus Spot versus on-demand, as part of a comprehensive cost-optimization plan? A: Categorize instances by workload characteristics and criticality rather than treating the fleet uniformly. Instances supporting stable, long-running, business-critical workloads with predictable utilization are strong Savings Plan candidates (locking in a discount against a spend commitment you’re confident you’ll actually use). Instances supporting stateless, fault-tolerant, or easily-restartable workloads (background workers, CI/CD agents, batch/ETL processing, non-critical microservices with adequate replica counts and fallback logic) are strong Spot candidates, since interruption risk is either irrelevant or well-mitigated by the workload’s own design. Instances that don’t clearly fit either category — highly variable/uncertain traffic patterns, workloads you don’t yet have enough historical data on, or resources currently mid-migration — should remain on-demand until you have enough operational confidence to reclassify them, since committing prematurely (via Savings Plan) or exposing an unproven workload to interruption risk (via Spot) both carry real downside. This categorization should be revisited periodically as workload behavior and business needs evolve, rather than treated as a one-time decision.

Q9: What’s the strategic value of running kube-hunter, kubescape, and kube-bench together, rather than choosing just one, and how would you prioritize adoption if resource/time constraints only allowed a phased rollout? A: Each tool answers a genuinely different question: kube-hunter answers “if an attacker got into my cluster via a compromised pod, what could they actually do?” (an offensive/attacker-perspective simulation); kubescape answers “what specific misconfigurations exist across my deployed workloads, and how do I fix each one?” (a comprehensive, remediation-focused defensive audit); kube-bench answers “does my cluster meet a recognized, industry-standard security baseline?” (a compliance/audit-oriented checklist against the CIS benchmark). Together they cover offensive testing, comprehensive defensive remediation, and standardized compliance auditing — three complementary angles rather than redundant coverage. For a phased rollout under real constraints, prioritize kubescape first (as explicitly recommended in this session) given its breadth and directly actionable remediation guidance, since it provides the most immediate risk-reduction value per unit of adoption effort; add kube-bench next if compliance/audit requirements are a factor (since its output format maps directly onto formal audit documentation); and add kube-hunter last, once the more foundational misconfiguration and compliance gaps are already being addressed, since its attacker-simulation findings are most valuable once the “easy” defensive gaps have already been closed.

10. Exam & Certification Notes

(Relevant to AWS Certified Solutions Architect / SysOps Administrator / DevOps Engineer certifications, and Certified Kubernetes Security Specialist (CKS)-adjacent content.)

ASG Instance Refresh mechanics: Know that Instance Refresh is the AWS-native mechanism for rolling out Launch Template changes across an existing ASG, and that its sequencing behavior (health-check-before-terminate vs. simultaneous replace) directly determines whether the operation is zero-downtime — a commonly tested operational nuance.
Launch Template versioning: Understand that Launch Templates are versioned resources, and that an ASG references a specific version — changing which version an ASG uses is itself the mechanism for rolling out instance-config changes, distinct from editing a Launch Configuration (the older, non-versioned AWS construct, largely superseded by Launch Templates).
kubectl drain vs. cordon vs. uncordon: A standard, frequently tested Kubernetes operational distinction — drain evicts existing pods (and by default cordons the node too), cordon alone only prevents future scheduling without evicting anything currently running, and uncordon reverses a cordon.
CIS Benchmarks for Kubernetes: Know that CIS (Center for Internet Security) publishes a recognized Kubernetes benchmark, and that tools like kube-bench specifically audit against it — relevant for CKS and general Kubernetes security certification content.
Savings Plans — Compute vs. EC2 Instance scope: A frequently tested distinction on AWS certifications — Compute Savings Plans apply across EC2/Lambda/Fargate; EC2 Instance Savings Plans apply only to EC2, and neither applies to Spot.
Spot Instance interruption handling: Know that Spot Instances can receive a two-minute interruption notice before reclamation, and that production-grade Spot usage requires application-level graceful-shutdown handling to respond to this notice — a standard exam topic when Spot appears in a scenario question.

11. Cheat Sheet

Migration Difficulty Matrix (memorize the inversion):

	Intel→AMD	AMD→ARM
Standalone	Easiest	Hardest
ASG	Medium	Medium
EKS Node Group	Hardest	Easiest

ASG Migration — Zero Downtime, 4 Steps:

New Launch Template version (change instance type only)
Point ASG at new version
Instance Refresh: launch-new → verify-healthy → THEN terminate-old
Verify savings in Cost Explorer

EKS Node Group Migration — ~2-3 min downtime, 4 Steps:

New node group from scratch (same subnets/role/scaling; matching labels/taints)
kubectl drain old node group
kubectl cordon old node group → scheduler moves pods to new node group
Verify savings in Cost Explorer

Right-Sizing Rules:

Balance P-R-C: Performance, Reliability, Cost
Never size to average — always keep 20–30% buffer
NEVER right-size during a cloud-to-cloud migration
Avoid right-sizing when traffic is highly volatile/uncertain

Spot in Production — Safe If:

2+ replicas minimum
HPA + PodDisruptionBudget configured
Fallback instance-type chain defined
Graceful shutdown handling implemented
Split is at the node-pool level, not per-application

Savings Plans Quick Reference:

Type	Covers	Applies to Spot?
Compute Savings Plan	EC2 + Lambda + ECS/Fargate	No
EC2 Instance Savings Plan	EC2 only	No

EKS Security Scanning Tools (all free, Aqua Security):

Tool	Answers	Output Style
kube-hunter	”What could an attacker do from inside a pod?”	Vulnerability IDs → vendor remediation lookup
kubescape	”What’s misconfigured, and how do I fix it?”	Detailed control checks + remediation
kube-bench	”Do we meet the CIS benchmark?”	Pass/fail/warning checklist
If only one: choose kubescape.

12. Gaps & Assumptions

Exact CLI command syntax for several steps (the aws eks describe-nodegroup subnet lookup, the exact eksctl/aws eks create-nodegroup node-group-creation command, the exact instance-profile-exclusion flag) was demonstrated live on screen but not always fully dictated verbatim in the transcript audio. This document presents these using standard, conventional AWS CLI/eksctl syntax consistent with what was described — verify exact current flags against official AWS documentation before use.
”L&T” as the organization name in the GCP→AWS migration cautionary story is preserved as heard in the transcript — this is an anecdotal reference from the instructor’s own past work experience, not part of the HealthCorp engagement itself, and is not independently verifiable from this transcript alone.
The “64-bit” framing for ARM/Graviton (used again in this session, echoing the same simplification noted in the first HealthCorp session’s package) is imprecise — both Intel/AMD (x86-64) and ARM/Graviton are 64-bit architectures; the actual distinguishing factor is the instruction set architecture (x86 vs. ARM), not bit-width. Preserved as stated for fidelity to the source, flagged here to avoid repeating it as fact in a technical context.
Advanced Track assignment structural change (participants now expected to build their own environment starting Week 2) was announced in this session but its full implications/instructions weren’t detailed in this transcript — participants should refer to the actual Week 2 assignment materials for specifics.
The Terraform-based recorded demo and codebase referenced multiple times in this session (as something already created and to be shared/uploaded) is not part of this transcript — this document describes its existence and purpose as stated, not its content.
EKS security scan example figures (e.g., “14 checks, 8 failed” for kubescape; “13 passed, 3 warnings” for kube-bench) are specific to the demo cluster shown live in this session, not HealthCorp’s actual production cluster — treat these as illustrative of the tools’ output format, not as HealthCorp-specific findings.
This document consolidates a long, time-constrained session (the instructor explicitly extended by 15–20 minutes past the original schedule and still deferred full EKS cost optimization to a follow-up call) — content has been reorganized topically for clarity rather than presented in strict chronological order, consistent with the approach used for prior packages in this series.