SRE Labs (Advanced Track) — EKS Cost Optimization: Karpenter & CastAI

Structured educational resource covering sre labs (advanced track) — eks cost optimization: karpenter & castai.

senior 45 min read 12 sections

#kubernetes#cloud-k8s#aws#dns

Complete Learning Package (Karpenter · CastAI · Capacity Planning · Kubernetes Autoscaling)

Source: 2026-02-07-19-07-59.md — A weekday session (7 February). Presenter: Ravi (guest DevOps/SRE practitioner, also the EKS security scanning presenter from Session 2). Covers Karpenter theory + installation steps and CastAI overview + partial demo. Both live demos failed; the instructor promised a rescheduled dedicated demo session.

Session context: This is the EKS cost optimization session that was deferred from Project Call 2. It follows the HealthCorp cost optimization arc and the EKS security scanning content (Kube-hunter, Kubescape, Kube-bench).

Honesty note: The session is shorter than usual (~43K chars vs. the typical 100–120K) because both demos failed and significant time was spent in Q&A. Theory content is solid; the practical implementation detail that would have come from a working demo is missing. I have filled the gaps in Section 13.

2. Table of Contents

Executive Summary
Table of Contents
Detailed Structured Notes
- 3.1 AI in DevOps — Where It Adds Value
- 3.2 Capacity Planning Fundamentals (Min / Desired / Max)
- 3.3 How to Determine Maximum Capacity — Worked Example
- 3.4 Why Switch from Cluster Autoscaler to Karpenter
- 3.5 What Karpenter Is
- 3.6 Karpenter — 6-Step Installation Process
- 3.7 Karpenter — EC2NodeClass and NodePool
- 3.8 Karpenter — Advanced Q&A (OS Patching, Blue-Green, Pod Identity)
- 3.9 Karpenter — When to Use OSI vs V7 Framework (Recap)
- 3.10 CastAI — What It Is and How It Works
- 3.11 CastAI — Dashboard and Recommendation Types
- 3.12 CastAI — Pricing and When It Makes Sense
- 3.13 Goldilocks — Brief Mention
- 3.14 Demo Status and What Was Promised
Key Concepts Table
Architecture & Workflow Analysis
Commands, Configs & Manifests
Tools & Technologies
Real-World Production Usage
Interview Preparation (Beginner / Intermediate / Advanced)
Exam & Certification Notes
Cheat Sheet
Gaps, Assumptions & Incomplete Areas
Gap-Fill — What the Session Left Unfinished, Completed Here

3. Detailed Structured Notes

3.1 AI in DevOps — Where It Adds Value

The session opened with a brief but useful exchange on when AI is and isn’t the right tool for DevOps automation.

The thumb rule (stated by presenter):

“The thumb rule for any automation — whether it is RPA, script-based, or AI — is: what is the repetitive, routine activity that is rule-based or logic-based (not business-based)? Identify such scenarios and then ask: do I need a straightforward script, or do I need an AI angle?”

When AI adds value over plain automation:

When the user needs a conversational / plain-English interface — AI translates natural language into actual tasks/programs. This is the primary differentiator.
When decisions depend on historical data and complex logic (e.g., “which component inside this application is failing?” — if you’ve collected one year of incident/fix data, an AI engine can pattern-match against that history).
When managing scale — e.g., one engineer handling 10 accounts without AI vs. 15 accounts with AI (concrete example from an attendee’s experience).

When plain automation is sufficient:

Rule-based, logic-based tasks with deterministic outcomes (e.g., daily backups, server start/stop schedules).
The logic doesn’t require natural-language interface or historical pattern matching.

Practical example given (attendee): Client sends infrastructure requests (start servers, create small Terraform snippets) via email. AI agent picks up the request, executes it, and returns a result. Engineer now handles 50% more accounts with the same effort. Result: client reduces headcount need; engineer gets higher workload without proportional time increase.

3.2 Capacity Planning Fundamentals (Min / Desired / Max)

Before covering Karpenter, the presenter insisted on establishing this foundational concept because it underpins all autoscaling decisions.

Minimum Capacity

The number of nodes that must be running when the cluster starts. Purpose:

Run Kubernetes core components (CoreDNS, kube-proxy, etc.).
Provide baseline capacity for critical application processes.

Rule of thumb: size minimum to run core K8s components + your most critical services (login, database, front-end).

Desired Capacity

Capacity needed for normal operating load (excluding peak):

Identify critical processes (login service, web front-end, database).
Size for typical concurrent usage.
This is the “steady state” node count.

Maximum Capacity

The ceiling — the most nodes the cluster will ever provision. This is the most nuanced setting.

How NOT to determine max: “Just set it high” — this causes chronic overprovisioning.

How to determine max correctly: (see Section 3.3)

3.3 How to Determine Maximum Capacity — Worked Example

Worked example: Banking internet banking platform

Step	Action	Result
1. Find absolute upper bound	How many customers does the bank have?	50,000 customers
2. Apply trend factor	Not all customers log in simultaneously; check historical peak simultaneous sessions from production logs	Observed peak: 20,000 simultaneous
3. Apply growth factor	Customer base grew from 30K → 50K (~1.67x). Scale the peak proportionally: 20,000 × (50,000 / 30,000) ≈ 33,000	Projected peak: ~33,000
4. Add buffer	Add 20% to account for planning-to-implementation lag and business growth	33,000 × 1.20 ≈ ~40,000 equivalent capacity
5. Convert to infrastructure	Run load tests to find how many nodes support 40,000 concurrent sessions	e.g., 20 nodes at max

Key principles:

Never size for the absolute maximum (all 50,000 customers logging in simultaneously) — it never happens and leads to massive overprovisioning.
Always add a 20% buffer at planning time, because implementation takes months and business grows.
Inputs come from two teams: Application team (CPU/RAM/disk/IO demand per session) + Business stakeholders (upcoming campaigns, regulatory events, seasonal peaks).
Planning is cross-functional. DevOps/infra cannot make this decision alone.

For e-commerce / seasonal businesses:

Seasonal peaks (Black Friday, Diwali, New Year campaigns) must be explicitly factored in.
Historic trend data: minimum one year needed to capture seasonal variation.
Performance/load testing in non-prod validates the infrastructure demand per user.

The problem this creates (leading into Karpenter): You design for peak, but peak happens only a few times a year. The rest of the time, ~30% of provisioned resources sit idle — wasted cost.

3.4 Why Switch from Cluster Autoscaler to Karpenter

Three structural problems with Cluster Autoscaler (CA) / native ASG-based scaling:

Problem	Cluster Autoscaler	Karpenter
Provisioning speed	ASG takes 3–5 minutes to spin up a new node	Karpenter directly calls EC2 API; nodes ready in ~60 seconds
Overprovisioning	You pre-configure min/max/desired → ~30% of capacity sits idle at off-peak	Karpenter provisions exactly what the workload requests, no pre-configuration needed
Management overhead	DevOps engineer must tune min/max/desired per node pool	Karpenter takes decisions based on demand; DevOps only defines broad constraints

The core difference: Cluster Autoscaler scales within a fixed node pool (predefined instance type, pre-configured). Karpenter reads the pending pod’s resource requests and provisions the cheapest, right-sized instance type that satisfies those requests — dynamically, without a predefined pool.

Clarification from Q&A: Cluster Autoscaler is not “native” to Kubernetes — it also requires installation. But it is the older, more established, cloud-provider-recommended autoscaler. Karpenter is newer (developed by AWS, now open-sourced for multi-cloud) and more efficient.

3.5 What Karpenter Is

Type: Open-source Kubernetes node autoscaler.
Developed by: AWS (originally AWS-only); now multi-cloud.
Supported platforms: EKS (AWS), AKS (Azure Kubernetes Service), GKE (Google Kubernetes Engine).
How it works:
1. Watches for pods in Pending state (unschedulable due to insufficient capacity).
2. Reads the pod’s resources.requests (CPU + memory).
3. Selects the cheapest instance type from your defined set that can satisfy the request.
4. Directly calls the EC2 (or cloud equivalent) API to provision the node.
5. When load decreases, consolidates pods onto fewer nodes and terminates idle nodes → cost saving.
Key resource types:
- EC2NodeClass: defines the pool of possible instance types, AMI, subnets, security groups, and instance families.
- NodePool: defines scheduling constraints, disruption policies, and resource limits; references an EC2NodeClass.
Installation method: Helm.
Namespace: Can be karpenter namespace or kube-system (latest docs recommend kube-system).
Scope: Per-cluster. If you have 5 clusters, install Karpenter on each separately.
Does not replace: HPA (Horizontal Pod Autoscaler) or VPA (Vertical Pod Autoscaler) — Karpenter handles node-level scaling; HPA/VPA handle pod-level scaling. They are complementary.

3.6 Karpenter — 6-Step Installation Process

The presenter walked through these steps in notepad (no live terminal demo):

Step 1 — Install Karpenter via Helm

helm repo add karpenter https://charts.karpenter.sh
helm repo update

# Install (key parameters):
helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" \
  --namespace kube-system \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.clusterEndpoint=${CLUSTER_ENDPOINT}" \
  --set serviceAccount.create=false \
  --set "settings.interruptionQueue=${INTERRUPTION_QUEUE}" \
  --set "settings.aws.defaultInstanceProfile=KarpenterNodeInstanceProfile-${CLUSTER_NAME}"

Key parameters explained:

serviceAccount.create=false — service account will be created separately with IAM role attached.
clusterEndpoint — your EKS cluster’s API endpoint.
settings.aws.defaultInstanceProfile — naming convention: KarpenterNodeInstanceProfile-<cluster-name>.
interruptionQueue — SQS queue name for handling Spot interruption notices.
version — always check the official Karpenter docs for the latest stable version (1.9 or 1.10 at time of session).

Step 2 — Create IAM Role and Attach Policies

Karpenter needs permission to create and delete EC2 instances:

# Required policies to attach to the Karpenter IAM role:
# 1. AmazonEKSClusterPolicy
# 2. AmazonEKSWorkerNodePolicy
# (Additional permissions: EC2:RunInstances, EC2:TerminateInstances, EC2:DescribeInstances,
#  IAM:PassRole, SQS:* for interruption queue)

Then bind the IAM role to the Karpenter service account (IRSA or Pod Identity).

Step 3 — Tag Subnets

Karpenter discovers subnets using a specific tag:

# Tag all subnets that Karpenter should use:
aws ec2 create-tags \
  --resources <subnet-id-1> <subnet-id-2> <subnet-id-3> \
  --tags Key="karpenter.sh/discovery",Value="${CLUSTER_NAME}"

Why: Karpenter looks for this tag to find which subnets it can launch instances into. Without it, Karpenter cannot provision nodes.

Step 4 — Tag Security Groups

# Tag the cluster's security group:
aws ec2 create-tags \
  --resources <cluster-security-group-id> \
  --tags Key="karpenter.sh/discovery",Value="${CLUSTER_NAME}"

Step 5 — Create EC2NodeClass

Defines the template for nodes Karpenter can create:

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2           # Amazon Linux 2 (or Bottlerocket, Ubuntu, etc.)
  role: "KarpenterNodeRole-${CLUSTER_NAME}"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
  instanceProfile: "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"

Key design decision (from presenter): In the instance family list, include a mix of Spot and On-Demand plus a mix of small-to-large SKUs. This gives Karpenter maximum flexibility to find the cheapest available instance. Example mix:

  # in NodePool spec (see Step 6):
  - key: "karpenter.sh/capacity-type"
    operator: In
    values: ["spot", "on-demand"]
  - key: node.kubernetes.io/instance-type
    operator: In
    values: ["t3.medium", "t3a.medium", "t3.large", "t3a.large",
             "m5.large", "m5a.large", "c5.large", "c5a.large"]

Step 6 — Create NodePool

Defines the scheduling constraints Karpenter uses when selecting nodes:

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      nodeClassRef:
        name: default
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64", "arm64"]
  limits:
    cpu: 1000
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s

Once these 6 steps are applied, Karpenter is active and will begin managing node provisioning automatically.

Production note: Always use Terraform, CloudFormation, or equivalent IaC for these steps. Manual/CLI approach is for understanding only.

3.7 Karpenter — EC2NodeClass and NodePool Detail

EC2NodeClass is Karpenter’s equivalent of a Launch Template — it defines what nodes can look like. Key fields:

amiFamily: which OS/image family (AL2, AL2023, Bottlerocket, Ubuntu, Custom).
subnetSelectorTerms / securityGroupSelectorTerms: uses the karpenter.sh/discovery tags set in Steps 3–4.
instanceProfile: IAM instance profile for the nodes.

NodePool defines how Karpenter schedules. Key fields:

requirements: constraints on instance types, capacity types (Spot/On-Demand), architecture.
limits: hard cap on total CPU/memory Karpenter can provision (prevents runaway cost).
disruption: consolidation policy — when to bin-pack pods onto fewer nodes and terminate idle nodes.

Consolidation (cost saving mechanism): When pods finish or scale down, Karpenter’s consolidation policy terminates nodes that are no longer needed and re-schedules remaining pods onto fewer nodes. This is the primary cost-saving mechanism beyond right-sizing provisioning.

3.8 Karpenter — Advanced Q&A (OS Patching, Blue-Green, Pod Identity)

These topics emerged from the Q&A and represent real production complexity:

OS Patching / Custom AMI with Karpenter

The scenario (from an attendee): Their org maintains custom hardened AMIs (security-hardened, specific kernel versions). Every major kernel update triggers a new AMI build. With Cluster Autoscaler, they use a blue-green node pool strategy: spin up new node pool with new AMI, drain old nodes, migrate workloads. How does this change with Karpenter?

Answer (composite from presenter + attendees):

Karpenter uses the AMI defined in EC2NodeClass. To update the AMI, update the EC2NodeClass spec (change AMI ID or AMI family selector). Karpenter will begin using the new AMI for any new nodes it provisions.
Challenge: Karpenter may be running 2 nodes today and 15 nodes tomorrow. You don’t control exactly when nodes are created, so you can’t do a “batch AMI update” the same way.
Strategy 1 (Multi-NodePool): Maintain two NodePools with different EC2NodeClasses (different AMI IDs). Use labels/taints to control which workloads land where. This gives you blue-green capability within Karpenter.
Strategy 2 (Blue-Green at Deployment level): Split deployments across two availability zones or node pools via nodeAffinity / nodeSelector. Update AMI in one NodePool → Karpenter uses new AMI for that AZ. Once validated, update the other.
Key insight from presenter: “Karpenter is flexible and intelligent. Whatever scenario you’re describing is achievable in Karpenter. It requires researching the right combination of NodePools, affinities, and taints.”
Honest caveat: Presenter said “I don’t have a direct answer” to the exact question and offered to follow up. This is genuinely an advanced topic without a single canonical answer.

Pod Identity Support

Question: Does Karpenter support Pod Identity (newer AWS identity mechanism, replacing IRSA)?

Answer: Not definitively confirmed in session. The presenter checked the documentation and didn’t find explicit Pod Identity support listed. However, because Karpenter still needs to create nodes that join the cluster with the correct IAM permissions, the underlying IAM patterns (IRSA or Pod Identity) should work — but this requires verification against current Karpenter documentation.

Karpenter with Kubernetes Federation (KubeFed)

Question: Does Karpenter support multi-cluster federation?

Answer: Karpenter operates at the individual cluster level. For a federation of 5 clusters, you install Karpenter on each cluster independently. There is no cross-cluster Karpenter coordination.

3.10 CastAI — What It Is and How It Works

Type: SaaS platform for Kubernetes cost management and right-sizing.

Key distinction from Karpenter:

Karpenter is an autoscaler — it provisions/terminates nodes based on demand.
CastAI is a cost optimizer and recommendation engine — it analyzes your cluster and tells you what to change.
CastAI does not provide autoscaling in the same way Karpenter does. They are complementary: Karpenter handles scaling; CastAI handles cost visibility and optimization recommendations.

How it works:

Connect CastAI to your cluster by running a provided script (creates authorization/RBAC for CastAI’s agent).
CastAI agent monitors your cluster in real time (pod resource requests vs. actual usage, node utilization, etc.).
CastAI dashboard shows recommendations categorized by type and potential savings.
DevOps engineer reviews recommendations, applies judgment, and implements changes.

Supported platforms: EKS, AKS, GKE, and on-premises (via “on-prem” option in connection flow).

Access: console.cast.ai — free signup with company email. No personal email trial.

3.11 CastAI — Dashboard and Recommendation Types

The presenter showed CastAI’s pre-built demo cluster (their own training cluster couldn’t connect). Key recommendation categories visible:

Category	What CastAI Identifies	Demo Savings
Spot Migration	Workloads running On-Demand that could safely run on Spot	$5,000/month saved (from $27,000 total)
Workload Right-Sizing	Workloads with resource requests significantly above actual usage (CPU/RAM wastage)	~$5,000/month additional saved
Hibernation / Auto-Shutdown	Workloads or namespaces unused during specific hours (nights, weekends)	Schedule-based; saves % of idle time cost

Real-time vs. trend-based recommendations:

Recommendations appear in real time (as soon as CastAI detects over-requested resources).
But: DevOps should not act on real-time recommendations immediately. Wait at least one month of trend data before applying right-sizing changes — a workload might genuinely need high CPU at certain times.

Stateful workload awareness:

CastAI recognizes stateful workloads (e.g., databases with PersistentVolumeClaims).
For stateful workloads, CastAI will not recommend Spot migration — it keeps them on On-Demand. This is an important safety feature.

Comparison to Datadog:

Both provide cluster-level cost and performance insights.
Datadog is enterprise-grade, more expensive, broader scope (APM + logs + cost).
CastAI is focused specifically on Kubernetes cost optimization; lighter-weight than Datadog for this use case.
CastAI does not provide autoscaling. Datadog does not either (for nodes). Both are recommendation/visibility tools, not scalers.

3.12 CastAI — Pricing and When It Makes Sense

Pricing model (at time of session):

Free tier: available; no personal email (requires company/org email); limited period.
Paid: based on number of clusters + GPU workloads; approximately $1,000+/month depending on usage.

When the paid tier makes sense:

ROI calculation: if CastAI recommendations save $10,000/month → paying $1,000/month for CastAI is justified.
If potential savings are only a few hundred dollars → no ROI for paid tier.
Free tier still provides recommendations (just time-limited trial).

Alternatives mentioned:

In-house reporting tools (many mature organizations have custom scripts/dashboards).
Goldilocks (open-source, cluster-level right-sizing recommendations) — mentioned briefly.
Datadog (enterprise, broader scope).

3.13 Goldilocks — Brief Mention

What it is: Open-source Kubernetes right-sizing tool from Fairwinds. Provides CPU and memory recommendations per workload/namespace based on Vertical Pod Autoscaler (VPA) metrics.

Scope: Cluster-level; scans all pods and recommends optimal resources.requests and resources.limits.

Primary use case mentioned: One attendee is using it to address JVM memory issues — Java services requesting 6 GB RAM but not using it; Goldilocks identifies the wastage.

Distinction: Goldilocks is a cost recommendation tool, not a performance tuning tool. If your goal is thread/memory optimization for performance (not cost), you need APM-level tooling.

Similar tools:

CastAI (SaaS, broader feature set).
Datadog (enterprise, APM + cost).
AWS Compute Optimizer (native AWS; EKS + EC2 right-sizing).
Kubecost (open-source; cost visibility per namespace/workload).

3.14 Demo Status and What Was Promised

What failed:

Karpenter demo on training cluster: NodeClaim was created but the node never spun up. Cause not identified during session. Promised: dedicated rescheduled demo session.
CastAI agent on training cluster: Could not connect. Fell back to CastAI’s own demo cluster to show the UI.

What was shown:

Karpenter: theory + notepad walkthrough of the 6 installation steps.
CastAI: UI walkthrough using CastAI’s demo cluster (recommendations visible).

Promised for rescheduled session:

Karpenter working demo: simulate load, watch Karpenter provision nodes, watch consolidation remove idle nodes.
CastAI working demo on training cluster: see real recommendations for participants’ own deployments.
Notification via Discord and WhatsApp.

4. Key Concepts Table

Concept	Explanation	Example	Why It Matters
Minimum capacity	Nodes needed to run K8s core components and critical services at cluster start	CoreDNS + kube-proxy + login service = min 2 nodes	Cluster won’t function below this
Desired capacity	Normal operating node count for typical load	3–5 nodes for normal business hours	Sets steady-state cost baseline
Maximum capacity	Upper ceiling for autoscaling; sized for peak load + growth buffer	Historical peak × growth factor + 20% buffer	Prevents unbounded scaling (cost runaway) while handling real peaks
20% buffer rule	Always add 20% headroom during capacity planning	Peak needs 20 nodes → plan for 24	Accounts for business growth between planning and implementation
Seasonal peak	Highest demand period; must be explicitly factored into max capacity	E-commerce: holiday sales; banking: tax filing season	Sizing for average misses these events → outage
Cluster Autoscaler (CA)	K8s native node autoscaler; scales within predefined ASG / node pools	Min 2, max 10 nodes in a node group	Established, stable, but slow (3–5 min) and requires manual config
Karpenter	AWS-developed open-source K8s node autoscaler; provisions right-sized nodes on demand	Pod requests 2 CPU → Karpenter picks cheapest 2-vCPU instance	Faster (~60s), no pre-config needed, right-sized by default
EC2NodeClass	Karpenter resource defining node template (AMI, subnets, SGs, instance families)	`amiFamily: AL2, subnetSelector: karpenter.sh/discovery`	Karpenter uses this to know what kinds of nodes it can create
NodePool	Karpenter resource defining scheduling constraints and disruption policy	Requirements: Spot or On-Demand; limits: max 1000 CPU	Controls which workloads land where and when idle nodes are removed
Consolidation	Karpenter feature: terminates underutilized nodes and reschedules pods onto fewer nodes	After traffic drops: 8 nodes → 3 nodes in 30 seconds	Primary mechanism for off-peak cost saving
`karpenter.sh/discovery` tag	Tag Karpenter uses to find which subnets and SGs it can use	`Key=karpenter.sh/discovery, Value=my-cluster`	Without this tag, Karpenter cannot provision nodes
Overprovisioning	Resources provisioned but not utilized; wasted spend	~30% of cluster resources idle at off-peak with static max config	CA-based clusters structurally overprovision for peak
CastAI	SaaS Kubernetes cost management platform; provides right-sizing and Spot migration recommendations	”This workload requests 6 CPU, uses 0.5 CPU → downsize to 1 CPU”	Removes manual analysis; identifies waste automatically
Workload right-sizing (CastAI)	Adjusting pod resource requests/limits to match actual usage	JVM requesting 6 GB RAM, using 2 GB → recommend 2.5 GB	Directly reduces node size requirements → cost down
Spot migration recommendation (CastAI)	Identifies stateless workloads safe to run on Spot	”These 12 deployments can move to Spot; save $5K/month”	CastAI knows which workloads are stateful → skips those
Hibernation (CastAI)	Schedule workloads or namespaces to scale to zero during known off-hours	Dev namespace: scale to 0 at 10 PM, back up at 8 AM	Equivalent to EC2 non-prod scheduling but at pod/namespace level
Goldilocks	Open-source Kubernetes right-sizing tool (Fairwinds)	Scans all pods; recommends optimal requests/limits per workload	Free alternative to CastAI’s right-sizing component
Pod Identity	Newer AWS mechanism for IAM identity assignment to pods (replaces IRSA)	Replaces `eks.amazonaws.com/role-arn` annotation	More secure and scalable than IRSA; Karpenter compatibility unconfirmed at session

5. Architecture & Workflow Analysis

5.1 Cluster Autoscaler vs Karpenter Flow

CLUSTER AUTOSCALER (legacy)

Pod → Pending
  ↓ (K8s scheduler cannot find suitable node)
Cluster Autoscaler detects unschedulable pod
  ↓ (3–5 minutes)
ASG adds node of predefined type (fixed size from node group)
  ↓
Node joins cluster → pod scheduled

Problems:
  - Fixed node type (may be too big or too small)
  - 3–5 min provisioning lag
  - ~30% idle capacity at off-peak
  - Manual min/max/desired config per node group

────────────────────────────────────────────────

KARPENTER (modern)

Pod → Pending
  ↓ (K8s scheduler cannot find suitable node)
Karpenter reads pod's resources.requests
  (e.g., cpu: "500m", memory: "256Mi")
  ↓ (selects cheapest instance from NodePool that fits)
Karpenter directly calls EC2 API
  ↓ (~60 seconds)
Node joins cluster → pod scheduled on right-sized node

When load drops:
  Karpenter consolidation → bin-packs pods → terminates idle nodes

Benefits:
  - Right-sized nodes for each workload
  - ~60s provisioning
  - Consolidation saves off-peak cost
  - No manual min/max/desired tuning

5.2 Karpenter Setup Architecture (6 steps)

EKS Cluster
│
├─ Step 1: helm install karpenter (kube-system namespace)
│   └─ Karpenter controller pod running
│
├─ Step 2: IAM Role (Karpenter controller)
│   └─ Permissions: EC2:RunInstances, EC2:TerminateInstances, IAM:PassRole, SQS:*
│
├─ Step 3: Tag subnets
│   └─ Key: karpenter.sh/discovery, Value: <cluster-name>
│
├─ Step 4: Tag security groups
│   └─ Key: karpenter.sh/discovery, Value: <cluster-name>
│
├─ Step 5: EC2NodeClass (YAML applied to cluster)
│   └─ Defines: amiFamily, role, subnetSelector, sgSelector, instanceProfile
│
└─ Step 6: NodePool (YAML applied to cluster)
    └─ References EC2NodeClass; defines requirements (Spot/On-Demand, arch), limits, disruption policy

→ Karpenter is now active and managing node provisioning.

5.3 CastAI Integration Architecture

EKS Cluster
│
└─ CastAI agent (installed via script: kubectl apply -f <castai-script>)
    │
    └─ Sends cluster metrics to CastAI SaaS platform (console.cast.ai)
        │
        └─ CastAI analysis engine:
            ├─ Compares pod resource.requests vs actual usage
            ├─ Identifies stateless vs stateful workloads
            ├─ Computes On-Demand → Spot migration safety
            └─ Generates recommendations:
                ├─ Workload right-sizing (adjust requests/limits)
                ├─ Spot migration (for stateless workloads only)
                └─ Hibernation schedule (dev/staging off-hours)

DevOps engineer reviews → validates trend (≥1 month) → applies changes

5.4 Karpenter Node Lifecycle

PROVISIONING:
Pending pod detected
  → Karpenter selects cheapest fitting instance type from NodePool requirements
  → Karpenter calls EC2 CreateInstance
  → Node boots, kubelet registers with cluster
  → Pod scheduled and running
  (Total: ~60 seconds)

CONSOLIDATION (cost saving):
Load decreases → pods idle or removed
  → Karpenter detects underutilized nodes (based on consolidationPolicy)
  → Karpenter cordons + drains underutilized node (pods reschedule onto remaining nodes)
  → Karpenter terminates the empty node
  → Cluster shrinks to minimum needed capacity

SPOT INTERRUPTION:
AWS sends 2-minute Spot interruption notice → SQS interruption queue
  → Karpenter receives notice
  → Karpenter cordon + drains the Spot node gracefully
  → Provisions replacement node (On-Demand fallback or another Spot in different AZ)
  → Pods reschedule onto new node

6. Commands, Configs & Manifests

Karpenter — Environment Variables Setup

export KARPENTER_VERSION="1.0.0"   # Check official docs for latest stable
export CLUSTER_NAME="my-eks-cluster"
export CLUSTER_ENDPOINT=$(aws eks describe-cluster \
  --name "${CLUSTER_NAME}" \
  --query "cluster.endpoint" --output text)
export KARPENTER_IAM_ROLE_ARN="arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME}"
export INTERRUPTION_QUEUE="${CLUSTER_NAME}"

Karpenter — Helm Installation

helm registry logout public.ecr.aws || true

helm upgrade --install karpenter \
  oci://public.ecr.aws/karpenter/karpenter \
  --version "${KARPENTER_VERSION}" \
  --namespace kube-system \
  --create-namespace \
  --set "settings.clusterName=${CLUSTER_NAME}" \
  --set "settings.clusterEndpoint=${CLUSTER_ENDPOINT}" \
  --set serviceAccount.create=false \
  --set "serviceAccount.name=karpenter" \
  --set "settings.interruptionQueue=${INTERRUPTION_QUEUE}" \
  --wait

# Verify:
kubectl get pods -n kube-system | grep karpenter

Karpenter — Subnet and Security Group Tagging

# Tag subnets:
for SUBNET_ID in $(aws eks describe-cluster --name "${CLUSTER_NAME}" \
  --query "cluster.resourcesVpcConfig.subnetIds" --output text); do
  aws ec2 create-tags --resources "${SUBNET_ID}" \
    --tags Key="karpenter.sh/discovery",Value="${CLUSTER_NAME}"
done

# Tag cluster security group:
CLUSTER_SG=$(aws eks describe-cluster --name "${CLUSTER_NAME}" \
  --query "cluster.resourcesVpcConfig.clusterSecurityGroupId" --output text)
aws ec2 create-tags --resources "${CLUSTER_SG}" \
  --tags Key="karpenter.sh/discovery",Value="${CLUSTER_NAME}"

EC2NodeClass Manifest

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  role: "KarpenterNodeRole-${CLUSTER_NAME}"
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: "${CLUSTER_NAME}"
  instanceProfile: "KarpenterNodeInstanceProfile-${CLUSTER_NAME}"
  tags:
    karpenter.sh/discovery: "${CLUSTER_NAME}"
    Environment: "production"

NodePool Manifest (Mixed Spot/On-Demand, AMD and ARM)

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    metadata:
      labels:
        managed-by: karpenter
    spec:
      nodeClassRef:
        apiVersion: karpenter.k8s.aws/v1beta1
        kind: EC2NodeClass
        name: default
      requirements:
        - key: "karpenter.sh/capacity-type"
          operator: In
          values: ["spot", "on-demand"]
        - key: "kubernetes.io/arch"
          operator: In
          values: ["amd64", "arm64"]
        - key: "node.kubernetes.io/instance-type"
          operator: In
          values:
            - t3.medium
            - t3a.medium
            - t3.large
            - t3a.large
            - m5.large
            - m5a.large
            - m6g.large        # Graviton/ARM
            - c5.large
            - c5a.large
            - c6g.large        # Graviton/ARM
      taints: []
  limits:
    cpu: 1000
    memory: 1000Gi
  disruption:
    consolidationPolicy: WhenUnderutilized
    consolidateAfter: 30s

Testing Karpenter — Simulate Load

# Deploy a test workload that requests more resources than available:
cat <<EOF | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: inflate
spec:
  replicas: 10
  selector:
    matchLabels:
      app: inflate
  template:
    metadata:
      labels:
        app: inflate
    spec:
      containers:
      - name: inflate
        image: public.ecr.aws/eks-distro/kubernetes/pause:3.7
        resources:
          requests:
            cpu: 1
            memory: 1.5Gi
EOF

# Watch Karpenter create nodes:
kubectl get nodes -w
kubectl logs -f -n kube-system -l app.kubernetes.io/name=karpenter

# Clean up (watch Karpenter consolidate and terminate nodes):
kubectl delete deployment inflate
kubectl get nodes -w

CastAI — Agent Installation

# Script provided by CastAI console after selecting your cloud provider:
# Example (actual script varies per cluster):
curl -fsSL https://api.cast.ai/v1/agent.yaml | \
  sed 's|CASTAI_API_TOKEN|<your-token>|g' | \
  kubectl apply -f -

# Verify agent is running:
kubectl get pods -n castai-agent

7. Tools & Technologies

Tool	Type	Purpose	Key notes from session
Karpenter	Open-source (AWS, CNCF)	Kubernetes node autoscaler	Replaces Cluster Autoscaler; ~60s provisioning; right-sized nodes; consolidation saves cost
Cluster Autoscaler	Open-source (K8s SIG)	Kubernetes node autoscaler (legacy)	3–5 min provisioning; requires static min/max/desired config; ~30% overprovisioning
Helm	Package manager	Install Karpenter and other K8s tools	`helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter`
EC2NodeClass	Karpenter CRD	Defines node template (AMI, SGs, subnets, instance families)	Must be applied before NodePool
NodePool	Karpenter CRD	Defines scheduling constraints and disruption policy	References EC2NodeClass
CastAI	SaaS platform	Kubernetes cost management and right-sizing	Free tier with org email; paid from ~$1K/month; supports EKS/AKS/GKE/on-prem
Goldilocks	Open-source (Fairwinds)	Workload right-sizing recommendations via VPA	Free; cluster-level; good for CPU/memory request tuning
Kubecost	Open-source/SaaS	Kubernetes cost visibility per namespace/workload	Mentioned as complementary to Karpenter; not covered in this session
Datadog	Enterprise SaaS	Observability + cost insights	Comparable to CastAI for K8s cost visibility; broader scope; more expensive
HPA (Horizontal Pod Autoscaler)	Kubernetes native	Scales pod count based on metrics	Complementary to Karpenter (pod-level vs node-level)
VPA (Vertical Pod Autoscaler)	Kubernetes native	Adjusts pod resource requests	Powers Goldilocks recommendations

8. Real-World Production Usage

Karpenter adoption:

Now the recommended autoscaler for EKS in most new deployments.
Organizations switching from Cluster Autoscaler primarily for: provisioning speed (event-driven apps, ML workloads needing burst capacity) and cost savings from consolidation.
Financial services and healthcare use cases: fast provisioning for peak periods (tax season, benefit enrollment) without maintaining massive static node pools year-round.

CastAI real-world use:

JVM applications: notorious for requesting large RAM allocations; CastAI consistently finds 50–70% of that allocation is unused.
Latency-sensitive workloads: CastAI correctly excludes these from Spot recommendations (keeps them On-Demand).
Multi-team clusters: CastAI’s per-namespace cost visibility enables chargeback to application teams.

Capacity planning (real engineering practice):

Cross-functional process: infra team + application team + business stakeholders.
Historical data: minimum 1 year for seasonal businesses, 90 days minimum for non-seasonal.
Load testing: mandatory before any major capacity decision; uses realistic workload profiles, not synthetic load.
Never trust “maximum conceivable” — use observed historical peak × growth factor + buffer.

OS patching with Karpenter (production pattern):

Maintain separate NodePools per AMI version.
Use node affinity/taints to control which deployments land on which NodePool.
Test new AMI on lower environments → validate → update EC2NodeClass in production NodePool.
Karpenter will only start using the new AMI for new nodes; existing nodes continue on old AMI until consolidated.

9. Interview Preparation

Beginner

Q1. What is the difference between Karpenter and the Kubernetes Cluster Autoscaler? A: Cluster Autoscaler scales within predefined node groups (fixed instance types, manual min/max/desired config) and takes 3–5 minutes to provision. Karpenter directly provisions nodes by reading pod resource requests, selects the cheapest fitting instance type dynamically, and provisions in ~60 seconds. Karpenter also consolidates idle nodes automatically; Cluster Autoscaler requires manual scale-down configuration. Karpenter was developed by AWS, open-sourced, and now supports EKS, AKS, and GKE.

Q2. How does Karpenter know which subnets and security groups to use when provisioning a new node? A: Karpenter discovers them via tags. You must tag all subnets and the cluster security group with karpenter.sh/discovery=<cluster-name>. The EC2NodeClass manifest uses subnetSelectorTerms and securityGroupSelectorTerms to find resources with this tag at runtime.

Q3. What is CastAI and how is it different from Karpenter? A: CastAI is a SaaS cost optimization platform for Kubernetes. It monitors your cluster and recommends workload right-sizing, Spot migration, and hibernation schedules. Karpenter is an autoscaler — it provisions and terminates nodes based on demand. CastAI is a recommendation and visibility tool — it tells you what to change but doesn’t itself provision nodes. They’re complementary: Karpenter handles scaling efficiently; CastAI identifies what still needs optimization.

Intermediate

Q4. Walk through the 6 steps to install Karpenter on an EKS cluster. A: (1) Install Karpenter controller via Helm (helm upgrade --install karpenter oci://public.ecr.aws/karpenter/karpenter ...). (2) Create an IAM role for the Karpenter controller with EC2 and IAM permissions; bind it to the Karpenter service account. (3) Tag all cluster subnets with karpenter.sh/discovery=<cluster-name>. (4) Tag the cluster security group with the same tag. (5) Create an EC2NodeClass manifest defining the AMI family, IAM instance profile, and subnet/SG selectors. (6) Create a NodePool manifest defining requirements (instance types, capacity type, architecture), resource limits, and disruption/consolidation policy. Apply the manifests — Karpenter is now active.

Q5. What does Karpenter’s consolidation feature do and why does it save cost? A: When load decreases (pods removed, workloads scaled down), Karpenter’s consolidation policy identifies underutilized nodes. It cordons and drains the node (evicting pods to other nodes), then terminates it. This bin-packs remaining workloads onto fewer, fully-utilized nodes. The result: you pay for fewer nodes during off-peak hours without manually adjusting min/max/desired configuration. Consolidation timescale is configurable (e.g., consolidateAfter: 30s).

Q6. You’re on-boarding Karpenter into a production EKS cluster that currently uses custom hardened AMIs and a blue-green patching strategy. How do you maintain this with Karpenter? A: Use multiple NodePool resources, each referencing a different EC2NodeClass with a different AMI ID. Use node affinity or taints on deployments to target specific NodePools. When a new AMI is validated in lower environments: update the EC2NodeClass for one NodePool → Karpenter uses the new AMI for any new nodes in that pool → existing nodes continue with old AMI until consolidated. Gradually shift deployments to the new NodePool. This gives blue-green AMI management within Karpenter at the cost of some deployment configuration complexity.

Advanced

Q7. Karpenter provisioning is taking longer than expected during a traffic spike. What would you investigate? A: (1) Check IAM permissions on the Karpenter controller role — missing EC2:RunInstances or IAM:PassRole causes silent failures (NodeClaim created but node never spins up — exactly what happened in the session demo). (2) Check EC2 instance capacity availability in the configured AZ — Spot capacity may be exhausted; diversify instance types in NodePool. (3) Check subnet CIDR capacity — if the subnet has no available IP addresses, new instances can’t get IPs. (4) Check karpenter.sh/discovery tags on subnets and SGs — missing tags cause Karpenter to find no valid launch configuration. (5) Check Karpenter controller logs: kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter.

Q8. Compare capacity planning approaches: traditional (static max) vs. Karpenter-managed. When is each appropriate? A: Traditional static max: appropriate for predictable, stable workloads where the peak load pattern is well understood and cost of provisioning complexity is low. Advantages: simple, predictable cost. Disadvantages: 30% chronic overprovisioning, slow to respond to unexpected spikes. Karpenter-managed: appropriate for dynamic workloads with variable load patterns, microservices architectures, and ML/batch workloads with burst requirements. Advantages: right-sized provisioning, fast (~60s), automatic consolidation, lower off-peak cost. Disadvantages: requires IAM and networking setup, newer tooling, complex OS patching strategy. For most modern K8s deployments, Karpenter is superior. For legacy monoliths on dedicated nodes, static configuration may be simpler.

10. Exam & Certification Notes

CKA (Certified Kubernetes Administrator):

Karpenter is not directly tested in CKA (too new / vendor-specific) — but node autoscaling concepts are.
NodePool and EC2NodeClass are Karpenter CRDs — know they are custom resources, not core K8s objects.
Pod scheduling: understand resources.requests drives scheduling decisions — this is how Karpenter determines what to provision.
Cluster Autoscaler concepts (scaling, min/max) are core K8s knowledge.

AWS Solutions Architect / DevOps Professional:

Karpenter is an official AWS best practice for EKS autoscaling.
Know the comparison: Karpenter vs. Managed Node Groups with Cluster Autoscaler.
IAM permissions model: Karpenter needs EC2:RunInstances, EC2:TerminateInstances, IAM:PassRole.
IRSA (IAM Roles for Service Accounts) — used to bind the IAM role to the Karpenter service account.
SQS interruption queue: how Karpenter handles Spot interruptions.

Common exam misconceptions:

“Karpenter replaces HPA” → False. Karpenter is node-level; HPA is pod-level. Both are needed.
”Karpenter only works on AWS” → False since open-sourcing. Supports EKS, AKS, GKE.
”CastAI is an autoscaler” → False. CastAI is a recommendation/visibility tool, not a scaler.
”Cluster Autoscaler is native to Kubernetes” → Debatable. It’s open-source but not in core Kubernetes; each cloud provider offers its own maintained version.

11. Cheat Sheet

Why Karpenter over Cluster Autoscaler: Cluster Autoscaler: 3–5 min provisioning, static config, ~30% overprovisioning Karpenter: ~60s provisioning, dynamic right-sizing, automatic consolidation

Karpenter 6-step setup:

1. helm install karpenter (kube-system, serviceAccount.create=false)
2. IAM Role → attach EC2 + IAM policies → bind to service account (IRSA)
3. Tag subnets: karpenter.sh/discovery=<cluster-name>
4. Tag cluster SG: karpenter.sh/discovery=<cluster-name>
5. kubectl apply -f ec2nodeclass.yaml (amiFamily, role, subnet/SG selectors)
6. kubectl apply -f nodepool.yaml (requirements, limits, consolidation policy)

Key tags (both required): Key: karpenter.sh/discovery Value: <your-cluster-name>

NodePool requirements (recommended mix): capacity-type: [spot, on-demand] arch: [amd64, arm64] instance-types: [diverse]

Testing Karpenter: kubectl apply -f inflate-deployment.yaml (high CPU requests) → watch kubectl get nodes -w

CastAI access: console.cast.ai (company email required)

CastAI recommendation types:

Spot migration (stateless workloads only — skips stateful)
Workload right-sizing (requests vs. actual usage)
Hibernation (schedule-based scale-to-zero)

When to act on CastAI recommendations: Wait ≥1 month of trend data; don’t act on real-time blips.

CastAI vs Karpenter: CastAI = recommendations/visibility; Karpenter = actual provisioning. Complementary.

Capacity planning max = historical peak × growth factor + 20% buffer (never use absolute max)

12. Gaps, Assumptions & Incomplete Areas

Both demos failed — details not captured:

Karpenter provisioning in action (NodeClaim created but node never joined).
CastAI real cluster connection (agent failed to connect).
Promised: dedicated rescheduled demo session (notify via Discord + WhatsApp).

Transcription artifacts:

“Carpenter / car printer / car” = Karpenter
”Casti / cast aai” = CastAI
”Terapform” = Terraform
”Cubernetes / cubernet” = Kubernetes
”Cube proxy / kube proxy” = kube-proxy
”Hem” = Helm
”IRS / IDCS / ODC” = IRSA (IAM Roles for Service Accounts) / OIDC (OpenID Connect)
“SKUs” = instance type SKUs (t3.medium, m5.large, etc.)
”RPA” = Robotic Process Automation (mentioned in AI context)

Content not covered (expected from agenda, deferred or not reached):

Karpenter live demo (failed).
CastAI live demo on training cluster (failed).
Kubecost (mentioned in Session 2 agenda, not covered in this session).
Interaction between Karpenter and HPA/VPA in detail.
Karpenter + CastAI together (the combination the original HealthCorp session referenced).
Karpenter Pod Identity support (unresolved in session).
Karpenter Spot interruption handling deep-dive.

Assumptions:

Version “1.9 or 1.10” mentioned — verify against Karpenter’s official changelog; version numbers are approximate from transcript.
The Karpenter demo failure was likely due to IAM role misconfiguration (NodeClaim created = Karpenter is running; node not joining = IAM/permissions or subnet/SG tag issue). This is consistent with the instructor’s own debugging pattern from the demo.

13. Gap-Fill — What the Session Left Unfinished, Completed Here

Filled from Karpenter and Kubernetes engineering knowledge. Clearly labelled as gap-fill.

GAP 1 — Why the Karpenter Demo Likely Failed (and How to Debug It)

The session showed: NodeClaim was created but no node appeared. This is the most common Karpenter installation failure pattern. Here is the diagnostic procedure:

# Step 1: Check Karpenter controller logs (most informative):
kubectl logs -n kube-system -l app.kubernetes.io/name=karpenter --all-containers=true | tail -100

# Common error patterns:
# "failed to launch instance: UnauthorizedOperation"
#   → IAM role missing EC2:RunInstances permission
# "failed to resolve instance profile"
#   → instanceProfile in EC2NodeClass does not exist or has wrong name
# "no subnets found"
#   → subnets not tagged with karpenter.sh/discovery=<cluster-name>
# "no security groups found"
#   → cluster SG not tagged with karpenter.sh/discovery=<cluster-name>
# "RequestExpired"
#   → clock skew between Karpenter pod and EC2 API

# Step 2: Check NodeClaim status:
kubectl get nodeclaim -o yaml
# Look at .status.conditions for error messages

# Step 3: Verify IAM permissions:
# List policies attached to KarpenterControllerRole:
aws iam list-attached-role-policies --role-name KarpenterControllerRole-<cluster-name>

# Step 4: Verify subnet tags:
aws ec2 describe-subnets --filters \
  "Name=tag:karpenter.sh/discovery,Values=<cluster-name>" \
  --query 'Subnets[*].[SubnetId,AvailabilityZone,CidrBlock]' \
  --output table

# Step 5: Verify SG tags:
aws ec2 describe-security-groups --filters \
  "Name=tag:karpenter.sh/discovery,Values=<cluster-name>" \
  --query 'SecurityGroups[*].[GroupId,GroupName]' \
  --output table

# Step 6: Verify instance profile exists:
aws iam get-instance-profile \
  --instance-profile-name KarpenterNodeInstanceProfile-<cluster-name>

Most likely root cause in the session: Missing IAM permission or untagged subnet/SG — both are easy to miss and both produce the “NodeClaim created, node never joins” pattern.

GAP 2 — Karpenter + HPA Together (the Right Architecture)

The session didn’t cover how Karpenter and HPA interact. This is critical for production:

Pod-level scaling (HPA):
  CPU/memory spike on pods
  → HPA adds more pod replicas
  → Replicas may be Pending (no node capacity)
  → Karpenter detects Pending pods
  → Karpenter provisions new node
  → Pods scheduled

Traffic decrease:
  → HPA scales down pods
  → Nodes become underutilized
  → Karpenter consolidation removes underutilized nodes

Combined effect:
  HPA handles pod-level elasticity (fast, ~10s)
  Karpenter handles node-level elasticity (fast, ~60s)
  Together: true end-to-end autoscaling with cost efficiency

Recommended configuration: Always deploy HPA alongside Karpenter. Without HPA, you might get extra nodes but not extra pods — and a single pod cannot scale to use all the resources on a new node.

GAP 3 — Karpenter Spot Interruption Handling (SQS Queue)

The session mentioned interruptionQueue without explaining it:

# Create SQS queue for interruption notices:
aws sqs create-queue --queue-name "${CLUSTER_NAME}" --region "${AWS_REGION}"

# Set permissions for EC2 to publish interruption notices:
aws sqs set-queue-attributes \
  --queue-url "https://sqs.${AWS_REGION}.amazonaws.com/${AWS_ACCOUNT_ID}/${CLUSTER_NAME}" \
  --attributes '{
    "Policy": "{\"Version\":\"2012-10-17\",\"Statement\":[{\"Effect\":\"Allow\",\"Principal\":{\"Service\":[\"events.amazonaws.com\",\"sqs.amazonaws.com\"]},\"Action\":\"sqs:SendMessage\",\"Resource\":\"arn:aws:sqs:${AWS_REGION}:${AWS_ACCOUNT_ID}:${CLUSTER_NAME}\"}]}"
  }'

# Create EventBridge rules to send Spot interruption events to the queue:
aws events put-rule \
  --name "KarpenterInterruptionQueueRule" \
  --event-pattern '{"source":["aws.ec2"],"detail-type":["EC2 Spot Instance Interruption Warning","EC2 Instance Rebalance Recommendation","EC2 Instance State-change Notification"]}'

aws events put-targets \
  --rule "KarpenterInterruptionQueueRule" \
  --targets "Id=1,Arn=arn:aws:sqs:${AWS_REGION}:${AWS_ACCOUNT_ID}:${CLUSTER_NAME}"

Flow when Spot is interrupted:

AWS sends 2-minute Spot interruption notice
  → EventBridge captures it
  → Routes to SQS queue (CLUSTER_NAME queue)
  → Karpenter reads the queue
  → Karpenter cordons the Spot node (no new pods)
  → Karpenter drains the node (evicts pods gracefully)
  → Pods reschedule onto remaining nodes or new node
  → AWS terminates the Spot instance
  (Total graceful window: 2 minutes — sufficient with graceful SIGTERM handling)

GAP 4 — CastAI Workload Right-Sizing: How It Calculates Recommendations

The session showed the CastAI UI but didn’t explain the calculation method. Here’s how CastAI determines right-sizing recommendations:

Data collected: Actual CPU and memory usage per pod over time (via metrics API, similar to how kubectl top pods works, but continuously).

Comparison: Actual P99 usage vs. resources.requests in the pod spec.

Recommendation logic:

Current pod spec:
  requests: { cpu: "2000m", memory: "4Gi" }
  limits:   { cpu: "4000m", memory: "8Gi" }

Observed actual usage over 30 days:
  CPU P99:    500m
  Memory P99: 1.5Gi

CastAI recommendation:
  requests: { cpu: "600m",  memory: "2Gi" }   # P99 + ~20% buffer
  limits:   { cpu: "1200m", memory: "4Gi" }   # 2x requests (common pattern)

Implication:
  Karpenter was provisioning m5.large nodes for this pod (needs 2 CPU)
  After right-sizing: t3a.medium is sufficient (needs 0.6 CPU)
  Cost saving: per-pod, node type drops significantly

Why wait 30 days before acting: Seasonal patterns, batch jobs running weekly, end-of-month processing — these all create legitimate CPU/memory spikes that are invisible in a 24-hour window. Acting on a 1-day sample risks OOM kills or CPU throttling.

GAP 5 — Karpenter IAM Setup (Complete)

The session said “create IAM role and attach policies” without showing the specifics:

# Step 1: Create IAM role for Karpenter controller (using IRSA):
cat <<EOF > karpenter-trust-policy.json
{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::${AWS_ACCOUNT_ID}:oidc-provider/${OIDC_PROVIDER}"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "${OIDC_PROVIDER}:aud": "sts.amazonaws.com",
        "${OIDC_PROVIDER}:sub": "system:serviceaccount:kube-system:karpenter"
      }
    }
  }]
}
EOF

aws iam create-role \
  --role-name KarpenterControllerRole-${CLUSTER_NAME} \
  --assume-role-policy-document file://karpenter-trust-policy.json

# Step 2: Attach required policies:
# Karpenter Controller Policy (create from AWS docs or CDK construct):
aws iam attach-role-policy \
  --role-name KarpenterControllerRole-${CLUSTER_NAME} \
  --policy-arn arn:aws:iam::${AWS_ACCOUNT_ID}:policy/KarpenterControllerPolicy-${CLUSTER_NAME}

# Step 3: Create the service account in K8s (pointing to the IAM role):
kubectl create serviceaccount karpenter -n kube-system
kubectl annotate serviceaccount karpenter -n kube-system \
  eks.amazonaws.com/role-arn=arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterControllerRole-${CLUSTER_NAME}

# Step 4: Create instance profile for Karpenter-managed nodes:
aws iam create-instance-profile \
  --instance-profile-name KarpenterNodeInstanceProfile-${CLUSTER_NAME}
aws iam add-role-to-instance-profile \
  --instance-profile-name KarpenterNodeInstanceProfile-${CLUSTER_NAME} \
  --role-name KarpenterNodeRole-${CLUSTER_NAME}

# Step 5: Add Karpenter node role to aws-auth ConfigMap:
kubectl edit configmap aws-auth -n kube-system
# Add:
# - groups:
#   - system:bootstrappers
#   - system:nodes
#   rolearn: arn:aws:iam::${AWS_ACCOUNT_ID}:role/KarpenterNodeRole-${CLUSTER_NAME}
#   username: system:node:{{EC2PrivateDNSName}}

GAP 6 — Capacity Planning: Translating Business Numbers to Infrastructure

The session gave the banking example conceptually. Here is the translation to actual infrastructure sizing:

BUSINESS INPUT:
  Total customers: 50,000
  Historical peak concurrent sessions: 20,000 (from prod logs)
  Projected customer growth: 50,000 / 30,000 = 1.67x
  Projected peak: 20,000 × 1.67 = 33,400 sessions
  Planning buffer: 20%
  Design capacity: 33,400 × 1.20 = ~40,000 sessions

APPLICATION INPUT (from app team):
  Resources per session:
    CPU per active session:    0.02 vCPU
    Memory per active session: 50 MB
  Sessions per pod:  200 concurrent sessions per pod replica
  Sessions per node: 5 pods per node = 1,000 sessions per node

INFRASTRUCTURE SIZING:
  Pods needed:  40,000 / 200 = 200 pods
  Nodes needed: 200 / 5      = 40 nodes (at peak)
  Node type:    m5.xlarge (4 vCPU, 16 GB) handles 5 pods × 200 sessions

KARPENTER CONFIGURATION:
  Max limit in NodePool:
    cpu: 160     (40 nodes × 4 vCPU)
    memory: 640Gi (40 nodes × 16 GB)
  Instance types in EC2NodeClass: m5.xlarge, m5a.xlarge, m6i.xlarge, m6a.xlarge
  (Karpenter will provision from this set based on availability and cost)

RESULT:
  During off-peak (5,000 sessions / 1,000 per node = 5 nodes):
    → Karpenter consolidates to 5 nodes → saves 35 × node cost
  During peak (40,000 sessions = 40 nodes):
    → Karpenter provisions up to 40 nodes → handles the load