SRE Labs (Advanced Track) — Shadowing Session: Live Client Engagement (Fintech/Web3 GCP Infrastructure)

Structured educational resource covering sre labs (advanced track) — shadowing session: live client engagement (fintech/web3 gcp infrastructure).

senior 45 min read 11 sections

#kubernetes#cloud-k8s#aws#gcp

Cost Optimization, Security Hardening & High Availability — Full Learning Package

2. Table of Contents

Executive Summary
Table of Contents
Detailed Structured Notes
- 3.1 Session Context & Access Setup (GitHub SSH, AWS IAM, EKS practice environment)
- 3.2 Client Overview
- 3.3 Phase 1 — Cost Optimization
- 3.4 Phase 2 — Security Hardening
- 3.5 Phase 3 — High Availability, Scalability & Final Cost Optimization
- 3.6 CastAI Live Demo Walkthrough
- 3.7 Engagement Constraints, Risks & Open Questions
Key Concepts Table
Architecture & Workflow Analysis
Commands & Configurations
Tools & Technologies
Real-World Production Usage
Interview Preparation (Beginner / Intermediate / Advanced)
Exam & Certification Notes
Cheat Sheet
Gaps & Assumptions

3. Detailed Structured Notes

3.1 Session Context & Access Setup

Practice Environment (Week 1 Kubernetes Assignment)

A dedicated EKS cluster was being provisioned for the cohort’s Week 1 assignment (separate from the live client project discussed later).
Structure: four namespaces, one per squad, each containing a small set of microservices and some stress-testing workloads for participants to practice on.
Cluster provisioned across two AWS regions: us-east-1 and ap-south-1.
Access model: usernames derived from the participant’s email (everything before the @), shared initial password, with the AWS account ID posted in the announcement channel.
Participants were told to enable MFA on their AWS accounts; the instructor had to add the IAM permission allowing users to self-enroll in MFA before this was possible.
IAM roles initially granted were scoped narrowly (EC2-related permissions only), with the plan to broaden access gradually as the co-instructor (Ravi) built out more of the AWS/Azure environment.

GitHub Access & SSH Key Setup

Multiple participants hit HTTPS password authentication errors when trying to clone the assignment repo — GitHub no longer supports password-based git clone/git push over HTTPS.

Correct fix (as explained live):

Generate an RSA SSH key pair (public + private key) locally.
Add the public key under your GitHub account settings → SSH and GPG keys (account-level setting, not repo-level).
Clone/interact with the repository using the SSH URL, authenticated via your private key.

This is a standard, correct GitHub authentication practice — worth internalizing even outside this specific course context: GitHub deprecated password auth for Git operations, so SSH keys (or PATs) are mandatory going forward.

Assignment submission workflow:

A shared GitHub organization/repo exists (referred to as something like “core infra cloud native”).
Participants create their own branch in that repo.
A README.md in the repo documents the required branch-naming convention and submission process.
A common early hiccup: branches not appearing until refreshing GitHub Actions/the branch list in the UI.

3.2 Client Overview

Attribute	Detail
Industry	Fintech — transactions related to blockchain and Web3
HQ / Operating base	Dubai
Team maturity	Originally built by an internal developer team (no dedicated DevOps/platform team); company has since transitioned toward a more formal, “M&A-style” structure while infra practices haven’t caught up
Primary cloud (current)	GCP
Target cloud footprint	50% GCP / 50% AWS (for high availability, not primarily cost)
Number of GCP accounts	13–14, separated by environment (dev, QA/pre-prod, prod, etc. — multi-account model, not a single shared account)
Environments identified	Prod, Pre-prod (“PRO”), Staging/QA, Dev — four environments total
Current traffic	~2 lakh (200,000) — likely requests/users, unit not explicitly confirmed in transcript
Projected traffic growth	5x within the current financial year
Acceptable downtime	Max ~30–60 minutes, Saturdays only — no other planned/unplanned downtime windows exist because the business runs 24/7 across multiple regions
Points of contact	No clear infra point of contact exists; only informal contact with a solution architect and a few engineering leads; no infrastructure documentation exists, especially for pre-prod
Career angle	Client has ~6 open DevOps/infra roles in Dubai; SRE Labs offered to refer program participants who want to pursue this after the 6-week engagement

Monthly cost snapshot at engagement start:

Environment	Monthly Cost	Notes
Production	~$40,000/month	Majority of this (~$20–22K) is compute cost
Pre-prod	~$8,000/month	Where the team starts implementation/testing first
Staging/Dev	Not fully broken out in the call, but in scope	Included in the overall optimization target

Target savings (initial framing, later revised as more detail emerged): An early figure floated was a $7.5K → $3.5K-ish delta with a target of saving ~$3K/month against roughly 8K in spend — this was an early, rough estimate stated before the actual client cost breakdown ($40K prod / $8K pre-prod) was confirmed later in the call. Treat the later $40K/$8K figures as the authoritative numbers.

Pre-prod GCP service inventory (as observed, not yet fully documented by the client):

GKE (Google Kubernetes Engine) — ~13+ nodes, ~32–39 microservices
GCS (Google Cloud Storage) — object storage
Cloud SQL — three separate instances
Compute Engine instances (GCP’s equivalent of EC2)
Spanner (per a passing mention — a globally distributed relational DB service)
Dataproc (managed Spark/Hadoop)
An ETL pipeline service (name not recalled by the speaker in the transcript)
Networking components: logging, and unspecified “networking costs”
Argo CD and Helm charts for deployment
KEDA (Kubernetes Event-Driven Autoscaling) — referenced/confirmed present in the infra config, likely tied to message-queue depth-based scaling
Kubecost — already present for Kubernetes cost visibility

3.3 Phase 1 — Cost Optimization

Stated scope: Optimize costs across pre-prod first (as a safe testbed), then apply validated changes to production. Explicitly framed as infrastructure-only changes, not application code changes — though the team acknowledged that some optimizations (e.g., ARM/Graviton migration) can require minor application-level compatibility changes.

Cost optimization ideas shared by participants (based on their own prior production experience) — preserved as a reference catalog:

S3 lifecycle policies — apply lifecycle rules to artifact buckets (CI/CD pipeline artifacts) and database backup buckets (e.g., retain backups for only 6 months) to avoid paying for storage nobody needs.
RDS instance family optimization — migrate from general-purpose instance types (e.g., T3a) to Graviton (ARM-based) RDS instances for better price/performance.
ECS right-sizing — reduce over-provisioned container CPU/memory allocations and reduce unnecessary horizontal replica counts in QA/dev/prod ECS services, based on actual observed utilization.
Non-prod scheduling — shut down dev/QA environments outside working hours (start/stop on a schedule) since those environments don’t need to run 24/7.
Legacy data cleanup — one participant described a monolith database holding ~4 TB of a decade of legacy/inactive user data; cleaning this up cascades into multiple savings:
- Smaller EBS volume → lower disk cost
- Smaller dataset → lower cross-region data transfer cost (their case: ~4–5 TB transferred daily from Singapore to ap-south-1 for backups, another ~12–15 TB/day retained in a Mumbai region for compliance/audit for 3 days)
- Smaller disk → ability to move off Aurora onto standard MySQL, improving DB efficiency and enabling further instance downsizing
Data warehouse optimization — rightsizing/optimizing a Redshift data warehouse.
Redis → Valkey migration — described as a newer AWS-supported open-source Redis-compatible engine offering roughly a 2/3 cost reduction versus ElastiCache for Redis (participant’s figure, stated informally — verify current AWS pricing directly before quoting this externally).
Intel → ARM (Graviton) migration for compute — a recurring theme; requires validating application/library compatibility with ARM before migrating.
EBS gp2 → gp3 migration — a “free” optimization (no code impact) that reduces storage cost/improves price-performance versus older gp2 volumes.
Elastic IP cleanup — removing unused/orphaned Elastic IPs that are billed even when unattached to running instances.
Savings Plans vs. Spot vs. On-Demand analysis — understanding how AWS Savings Plans work (as distinct from Spot pricing) as a lever for compute cost reduction.
Managed message queue vs. self-hosted (Kafka) trade-off — explicitly discussed as a decision with code-change implications (e.g., swapping a managed queue service for self-hosted Kafka isn’t a config-only change), so it needs a dedicated conversation with the client before being pursued.
Cluster autoscaling — the client’s GKE cluster currently has no cluster autoscaler at all. A proof-of-concept for CastAI had already been run in pre-prod and the client approved moving forward with it (~$200/quarter or similar low pricing, exact terms unclear in the call). Alternatives discussed: Karpenter (AWS-native, free), cluster-autoscaler, and (for GKE) potentially EKS Auto Mode-equivalent managed autoscaling — the counter-argument raised was that free, cloud-native autoscalers avoid third-party vendor lock-in.
GCP Recommender — GCP’s equivalent of AWS Trusted Advisor; provides built-in security and cost-optimization recommendations. Team needs to check whether it’s already enabled, and note that enabling some of its features may require a paid support tier, which needs business approval.

Planned execution order for Phase 1 (as stated near the end of the call):

Build a full inventory of the client’s cloud resources (see Section 3.6 for tooling).
Categorize resources and optimize in this order: network layer → compute layer → data layer.
Validate every change in pre-prod first, observe cost impact, then propagate to production.

3.4 Phase 2 — Security Hardening

Initial findings shared live (informal, pre-audit observations):

Some EC2/Compute instances have SSH ports open directly to the internet with public IPs attached.
Some public IPs are orphaned — allocated but not attached to any active gateway/resource (wasted spend + attack surface).
No existing architecture documentation to clarify network segmentation (public vs. private subnets, DMZ boundaries, database placement).

Recommended network hardening pattern discussed (participant-contributed, standard practice):

Per-environment (per-account) VPCs, each with its own NAT Gateway / Internet Gateway for controlled egress/ingress.
Private subnets for databases; public subnets only for app-facing/edge components.
Network ACLs (“nacls”) layered alongside security groups.
Explicit IP-to-subnet mapping table to track this deliberately rather than ad hoc.

SSH access hardening options discussed:

Restrict direct SSH entirely, funnel all access through a bastion host (jump box).
Recognize that even bastion hosts aren’t fully “safe” on their own — enterprises (especially regulated ones like banking) often go further with privileged access management (PAM) tools (e.g., CyberArk was named specifically) that grant short-lived, time-boxed SSH sessions (example given: sessions valid for as little as 10–30 minutes) rather than standing SSH access.
Trade-off raised: DevOps/platform teams that frequently recreate infrastructure (changing IPs constantly) can find it operationally painful to keep re-registering with a PAM tool like CyberArk — a real tension between security tooling and infrastructure-as-code/ephemeral infra practices.
CIS Benchmarks — a participant described prior experience implementing CIS Level 1 and Level 2 benchmarks at the account level for a project that was scoped around PCI-DSS compliance, and suggested similar benchmarks could apply here (including inside the Kubernetes cluster).

Formal security audit scope, broken into five sub-audits:

#	Audit Area	What’s Covered
1	Cloud account audit	All 13 GCP accounts scanned for misconfigurations, excessive permissions, exposed resources, etc.
2	CI/CD audit	Full pipeline review — Terraform, Docker images, secret/token handling (checking for hardcoded credentials), rollback capability, version management. Confirmed in scope: Terraform, Docker images, “everything end to end.” A participant asked whether IAM role auditing is included — implied yes, as part of the broader cloud account/CI-CD review. Ansible automation tooling was raised but tentatively deprioritized as covered under the “self-service tools” audit instead.
3	Platform/self-service tooling audit	Custom internal tools built by the client’s developers (described loosely as “sales tools”/self-service platform tooling) — audited for compliance and compatibility with the rest of the stack.
4	Architectural audit	Reviewing whether the current architecture is inefficient in ways that drive up cost or reduce reliability, and proposing changes. Confirmed to cover every element in the cloud account, not just compute — explicitly includes databases.
5	Backup audit	Reviewing existing backup practices (confirmed: S3 backups, DB backups, and backups tied to some ETL pipelines). Disaster Recovery (DR) was explicitly and openly negotiated as NOT yet confirmed in scope — the team acknowledged DR (full environment recreation time, RTO/RPO-style questions) is conceptually distinct from routine backups, and left it as an open item to formally scope with the client.

Security audit tooling named:

ScoutSuite — open-source multi-cloud security auditing tool; a scan had already been run against a dev account and produced an HTML report.
Prowler — open-source cloud security/compliance scanning tool (AWS-origin, also supports other clouds), used alongside ScoutSuite for the cloud account audit.
The instructor noted a general principle: some security tools are cloud-agnostic (work across AWS/GCP/Azure) while others are cloud-specific — and since this client’s environment is GCP-heavy, GCP-specific tools will surface more precise/actionable findings than generic multi-cloud tools.

3.5 Phase 3 — High Availability, Scalability & Final Cost Optimization

Trigger for this phase: Anticipated 5x traffic growth in the current financial year, combined with a near-zero-downtime operating constraint (the business runs continuously across 4 production regions and 3 pre-prod regions — there is no natural low-traffic maintenance window).

Scope:

Make the infrastructure resilient to failure and scalable ahead of the projected growth.
Execute a final round of cost optimization on production once the pre-prod-validated changes have been rolled forward.
Execute the planned multi-cloud shift — splitting infrastructure roughly 50/50 between GCP and AWS. The explicit stated purpose is high availability, not cost arbitrage: e.g., a given microservice might run two replicas on AWS and two replicas on GCP simultaneously, so a full outage of one cloud provider doesn’t take the whole service down.

Downtime/change-management constraint (applies across all three phases, emphasized repeatedly):

Only acceptable downtime window: up to ~30–60 minutes, on Saturdays.
Any migration/optimization must be validated as safe with effectively zero risk of breaking production.
The client is willing to run duplicate/parallel infrastructure temporarily (i.e., pay for both the old and new setup simultaneously) if that’s what it takes to guarantee a safe cutover — cost is secondary to stability during migrations.
Any change that would require application code changes (beyond minor compatibility patches, e.g., installing ARM-compatible libraries) is explicitly out of scope without a separate conversation and sign-off from the client. Infra-only changes (config, instance types, storage class, etc.) are the default mandate.

3.6 CastAI Live Demo Walkthrough

The instructor gave a live product tour of CastAI, a SaaS Kubernetes cost-optimization and autoscaling platform, using a demo cluster (not the actual client cluster). Key screens/features shown:

Cluster overview — total node count (demo showed ~100 nodes: ~70% on-demand / ~30% spot in the example), pod scheduling status (scheduled vs. unscheduled, with reasons surfaced for unscheduled pods — e.g., “not enough space in the cluster”).
CPU/memory/storage utilization view — requested vs. actually-used resources, with a time-range toggle to inspect utilization patterns over different days/times.
Cost monitoring — current month-to-date spend, forecasted spend, daily cost breakdown, and cost segmented by on-demand vs. spot vs. fallback instance categories.
Networking cost view — a dedicated tab for inspecting cross-AZ/egress-type networking costs.
Workload view — drills into individual workloads and their resource consumption; can also be filtered/viewed per namespace.
Savings recommendations — in the demo, CastAI reported a 94% cost-saving opportunity available if the workloads shown were shifted to spot instances (with a toggle to exclude spot if availability risk is unacceptable for a given workload).
Before/after optimization comparison — showed a concrete example moving from an Intel-based instance family (e.g., c5.4xlarge-style) to an AMD-based instance family, with compute cost dropping from ~$44,000/month to ~$5,000/month in that illustrative scenario (demo data, not the actual client’s numbers — do not treat as a guaranteed real-world savings ratio).
Node list — detailed view of every node, with labels and attached workloads.
Rebalancer / workload autoscaler — advanced features mentioned but not demoed in depth in this session (flagged as “we’ll explore this later”).
Optimizer — an extension for optimizing costs on managed services like RDS, beyond just Kubernetes compute.
Application monitoring — CastAI has also expanded into a broader observability/monitoring feature set beyond its original Kubernetes-cost-optimization core.
Cluster connection flow — to onboard a real cluster, you click “Connect Cluster,” choose the platform (e.g., GKE), and CastAI provides a command to run inside the cluster that installs the CastAI agent and grants it the access it needs to begin analyzing and (optionally) auto-remediating cost/scheduling issues.

How CastAI compares to alternatives (as discussed):

Functionally similar in spirit to Karpenter (AWS-native node autoscaler) and standard cluster-autoscaler, but CastAI adds a broader feature set: predictive cost modeling (uses ML models to forecast/optimize proactively rather than purely reactively), automatic rebalancing, an “Optimizer” for non-Kubernetes managed services, and (optionally) the ability to grant CastAI authority to automatically apply its recommended changes rather than just suggesting them.
Karpenter/cluster-autoscaler are free and avoid third-party vendor lock-in — explicitly raised as the main argument against paying for CastAI, especially since the client hadn’t yet fully explored native options.
CastAI (and similar third-party FinOps tools) give generic, use-case-agnostic recommendations — a human still has to apply business context (e.g., corporate/reserved-instance discounts that don’t show up in the tool’s “standard” pricing assumptions) before blindly acting on a suggestion.
Comparable alternative tools named in discussion: Kubecost (already present in this client’s environment) and a tool referred to only as “e-something” (name unclear/cut off in the transcript — likely a similarly-branded FinOps tool; not confirmed).

3.7 Engagement Constraints, Risks & Open Questions

No infrastructure ownership clarity — the client currently has no distinct “infra owner”; historically it’s been managed directly by application developers. The team explicitly flagged this as the biggest risk to the engagement: any change made without full understanding of a component’s role could cause a business-impacting outage, since there’s no clear escalation path or subject-matter owner to consult first.
Pre-prod carries production-equivalent authority — despite being called “pre-prod,” this environment is treated with the same care as production because of its criticality to the business, per the client’s own framing.
Shared cloud console access problem (unresolved during the call): the instructor wanted to give the whole cohort hands-on access to the client’s actual (read-only) GCP console for learning purposes, but Google’s account policies restrict how many concurrent devices/sessions can be logged into a single Gmail-based account at once (best guess in the call: roughly 3–4 concurrent sessions). Options discussed and left unresolved:
- Creating a shared Gmail account (blocked by Google’s session-concurrency policy).
- Sharing a GCP service account JSON key — rejected as a security risk, and impractical anyway since the client’s policy expires/rotates these credentials every 24–36 hours, requiring constant re-sharing.
- Attaching multiple individual users to a shared service account with view-only permissions — floated as a possible workaround, not confirmed as implemented by the end of the call.
Tool procurement constraints — introducing any third-party tool (even something like GCP’s built-in Recommender, which may require a paid support tier) requires business/procurement approval before use — this is a recurring theme across both cost and security tooling decisions.
Client agenda/prep gap — a participant gave direct, constructive feedback that sessions like this would be more valuable if a short agenda were shared in advance, so participants could pre-read and arrive prepared instead of following the discussion live and cold. The instructor agreed to start sharing a pre-session agenda going forward.

4. Key Concepts Table

Concept	Explanation	Example	Why It Matters
FinOps inventory-first approach	Before optimizing cost, build a complete inventory of every resource across every account/environment	Downloading all GCP resource metadata (instance type, size, CPU, storage) into a spreadsheet before touching anything	You can’t optimize what you haven’t measured; prevents accidentally breaking something you didn’t know existed
Cluster autoscaling (general)	Automatically adding/removing Kubernetes nodes based on real-time scheduling pressure and resource utilization	CastAI, Karpenter, native cluster-autoscaler	Prevents both over-provisioning (wasted cost) and under-provisioning (failed pod scheduling)
Graviton / ARM migration	Moving compute (EC2, RDS) from Intel/AMD (x86) to AWS Graviton (ARM) processors	RDS T3a → Graviton-based instance	Typically cheaper and more power-efficient per unit of compute, but requires validating software/library ARM compatibility
Lifecycle policies (S3/storage)	Rules that automatically transition or delete objects in storage after a defined period	Move CI/CD artifacts to cheaper storage tiers or delete backups after 6 months	Prevents storage costs from growing unbounded as historical data accumulates
Non-prod scheduling	Automatically stopping/starting non-production environments outside business hours	Dev/QA environments powered off nights and weekends	Non-prod environments often don’t need 24/7 uptime — this alone can cut related costs dramatically
CIS Benchmarks	Industry-standard security configuration baselines (Levels 1 and 2) published by the Center for Internet Security	Applying CIS Level 1/2 benchmarks to a cloud account for PCI-DSS compliance	Gives a recognized, auditable security baseline, often a compliance requirement in regulated industries
Bastion host	A hardened, tightly controlled server that acts as the single entry point for SSH access into a private network	All SSH traffic routed through one jump box instead of direct-to-instance	Reduces attack surface by eliminating direct internet-facing SSH; centralizes access logging
PAM (Privileged Access Management)	Tooling that grants short-lived, audited, just-in-time access to sensitive systems instead of standing credentials	CyberArk granting time-boxed (e.g., 10–30 minute) SSH sessions	Further reduces risk versus even a bastion host by removing long-lived access entirely; common in regulated industries like banking
Multi-cloud for HA (not cost)	Running redundant workload replicas across two different cloud providers specifically to survive a full provider-level outage	2 replicas of a microservice on AWS + 2 on GCP	Protects against the (rare but real) scenario of an entire cloud provider having an outage — a form of availability, not primarily cost, strategy
Cost-optimization layering (network → compute → data)	A structured order of operations for tackling cost optimization work	Fix orphaned/wasted network resources first, then rightsize compute, then optimize data storage/warehousing	Prevents wasted effort optimizing compute that’s about to be replaced by a network/architecture change
Zero/near-zero downtime migration constraint	Treating any infra change as needing to be effectively invisible to end users	Client allows only 30–60 min of downtime, only on Saturdays	Forces careful staging, dual-running infra, and pre-prod validation before any production change
KEDA (Kubernetes Event-Driven Autoscaling)	Scales Kubernetes workloads based on external event sources (e.g., queue depth) rather than just CPU/memory	Scaling consumers based on a message queue’s backlog depth	Enables scaling tied directly to actual business load signals, not just resource utilization proxies
Managed vs. self-hosted service trade-off	Choosing between a cloud-managed service (e.g., a managed queue) and self-hosting the equivalent (e.g., Kafka)	Managed message queue → self-hosted Kafka	Managed services reduce operational burden but can cost more; self-hosting can save money but often requires code/config changes and adds operational overhead

5. Architecture & Workflow Analysis

5.1 Client’s High-Level Environment Topology (as understood, not fully documented)

                     Client Organization (Fintech / Web3)
                                    |
        --------------------------------------------------------
        |                    |                    |            |
     Prod (GCP)          Pre-Prod (GCP)      Staging/QA (GCP)  Dev (GCP)
   ~$40K/month           ~$8K/month
        |                    |
   4 production          3 pre-prod
   regions                regions
        |                    |
   ------------          ------------
   |          |          |          |
  GKE       Cloud SQL   GKE        Cloud SQL
  Cluster   (x3)        Cluster    (x3)
   |                     |
  ~32-39                ~13+ nodes
  microservices
   |
  Argo CD / Helm  ---> deployed via CI/CD pipeline (audited in Phase 2)
   |
  KEDA (event-driven autoscaling) + Kubecost (cost visibility)

5.2 Planned Future-State Topology (Phase 3 target)

                        End Users
                            |
                    Global Traffic (5x growth expected)
                            |
        -----------------------------------------------
        |                                              |
     AWS (≈50%)                                    GCP (≈50%)
        |                                              |
  Microservice replicas                       Microservice replicas
  (e.g., 2 of 4)                               (e.g., 2 of 4)
        |                                              |
        -------------------  Data Layer  --------------
                    (Cloud SQL / RDS, S3/GCS, etc.
                     — exact cross-cloud data strategy
                     not detailed in this transcript)

Note (assumption): The transcript confirms the compute/replica layer will be split across AWS and GCP for HA, but does not detail how the data layer (databases) would be kept consistent across two clouds. This is flagged as a gap — real-world multi-cloud database strategy (active-active replication, single source of truth with cross-cloud read replicas, etc.) was not discussed.

5.3 Cost Optimization Workflow (Phase 1 execution order)

1. Build Inventory
   (GCP SDK script / console export → Excel/CSV of all resources)
        |
        v
2. Categorize by Layer
        |
   -----------------------
   |          |          |
 Network    Compute     Data
   |          |          |
 Fix open   Rightsize   Lifecycle
 SSH ports, ECS/Compute policies,
 orphaned   instances,  legacy data
 IPs        Graviton    cleanup,
            migration,  DB engine
            cluster     optimization
            autoscaling
   -----------------------
        |
        v
3. Validate in Pre-Prod
        |
        v
4. Measure Cost Impact
        |
        v
5. Roll Forward to Production
   (only within approved downtime window, or with zero-downtime technique)

5.4 Security Audit Workflow (Phase 2 execution order)

1. Cloud Account Audit  (all 13 GCP accounts)
   - Tools: ScoutSuite, Prowler
        |
        v
2. CI/CD Audit
   - Terraform, Docker images, secrets/tokens, rollback, versioning
        |
        v
3. Platform / Self-Service Tooling Audit
   - Internal developer-built tools
        |
        v
4. Architectural Audit
   - Full environment: compute + database + everything else
        |
        v
5. Backup Audit
   - S3, DB, ETL pipeline backups
   - DR: OPEN — not yet confirmed in scope

5.5 GitHub SSH Authentication Flow (participant onboarding)

Local Machine
   |
   1. Generate RSA key pair (public + private)
   |
   2. Upload PUBLIC key --> GitHub Account Settings --> SSH & GPG Keys
   |
   3. Clone repo via SSH URL (not HTTPS)
   |
   4. Git operations authenticate via PRIVATE key (stays local)
   |
   5. Create feature branch per README.md naming convention
   |
   6. Push work --> submit assignment via branch/PR

6. Commands & Configurations

This session was largely conceptual/architectural rather than hands-on scripting, so few literal commands were spoken on-screen. What follows are the concrete technical steps and syntax patterns that were explicitly described.

Command / Config	Purpose	Explanation
`ssh-keygen` (RSA key pair generation — implied, not typed verbatim on screen)	Generate a public/private key pair for GitHub authentication	Standard first step for SSH-based Git auth: create the key pair locally, keep the private key secret, and register the public key with GitHub.
GitHub → Settings → SSH and GPG keys → add public key	Register your SSH public key with your GitHub account	This is an account-level setting, not a per-repository setting — a common point of confusion the instructor explicitly clarified.
`git clone git@github.com:<org>/<repo>.git` (SSH-style URL, implied)	Clone the assignment repository using SSH instead of HTTPS	Required because GitHub no longer accepts password authentication over HTTPS for git operations.
GCP Cloud Console → instance list → Export to Excel/CSV	Manually export current inventory of cloud resources (instance name, type, storage, CPU)	The manual, UI-driven equivalent of the inventory script described below — used as a stopgap while the script isn’t built yet.
Planned Python inventory script (not yet written at time of call) using the GCP SDK	Programmatically pull the same inventory data (instance metadata) that the GCP Console UI’s “export” feature provides	Described as a short script (~6–7 lines) — automates what would otherwise be manual clicking through the console; the goal is a repeatable inventory pipeline.
CastAI “Connect Cluster” flow → copy provided install command → run inside target cluster	Installs the CastAI agent into a Kubernetes cluster (demoed for GKE)	Grants CastAI the access needed to begin analyzing cost/scheduling data and, if authorized, apply automated optimizations.

7. Tools & Technologies

CastAI

Purpose: Kubernetes cost optimization and cluster autoscaling SaaS platform.
When to use it: When a Kubernetes cluster has no autoscaler at all, or when teams want predictive/ML-driven cost optimization plus a broad feature set (rebalancing, workload monitoring, managed-service cost optimization) beyond what free/native tools offer.
Advantages: Predictive (ML-based) cost modeling; broad feature surface (cluster autoscaling, workload autoscaling, rebalancer, RDS-style “Optimizer,” application monitoring); can be authorized to auto-apply changes, not just recommend them.
Limitations: Third-party cost/vendor lock-in; generic recommendations still need human business-context validation (e.g., existing negotiated discounts); requires procurement/business approval to onboard.

Karpenter

Purpose: AWS-native, open-source Kubernetes node autoscaler.
When to use it: When you want a free, cloud-native alternative to third-party tools like CastAI, and your workloads run on AWS/EKS.
Advantages: Free; avoids vendor lock-in; native integration with AWS.
Limitations: Fewer bells and whistles than CastAI (no predictive ML cost modeling, no built-in managed-service optimizer in the same package) — was explicitly raised as the “why pay for CastAI” counter-argument.

Cluster Autoscaler (standard Kubernetes)

Purpose: The baseline, project-standard Kubernetes autoscaler.
When to use it: As a free default if you don’t need the more advanced predictive/rebalancing features of a commercial tool.
Limitations: Less sophisticated scheduling/rebalancing logic than CastAI or Karpenter, per the discussion.

ScoutSuite

Purpose: Open-source, multi-cloud security auditing tool that scans cloud account configurations for misconfigurations and risks.
When to use it: As part of a cloud account security audit, especially early-stage/first-pass scanning.
Advantages: Multi-cloud, produces a browsable HTML report.
Limitations: As a cloud-agnostic tool, may surface less precise, less actionable findings than a cloud-specific tool for a GCP-heavy environment.

Prowler

Purpose: Open-source cloud security and compliance scanning tool.
When to use it: Alongside ScoutSuite for cloud account audits; useful for compliance-oriented checks.
Advantages: Strong compliance-framework alignment (originated in the AWS ecosystem, also supports other clouds).
Limitations: Not detailed in depth in this transcript beyond being named as part of the toolkit.

GCP IAM Collector (name as given in the call; verify exact tool name independently)

Purpose: Referenced as a tool for understanding GCP service accounts and their relationships/permissions.
When to use it: When auditing whether service accounts are actively used by an application versus dormant/over-permissioned.
Note: Exact/official tool name uncertain from the transcript — treat as an assumption to verify.

GCP Recommender

Purpose: GCP’s built-in recommendations engine (GCP’s equivalent of AWS Trusted Advisor) — surfaces security and cost-optimization suggestions natively.
When to use it: As a first, “free-if-enabled” pass before adopting third-party tools, since it’s cloud-native.
Limitations: Some of its more advanced capabilities may require a paid GCP support tier, needing business approval.

Kubecost

Purpose: Kubernetes cost visibility and allocation tool.
When to use it: Already present in this client’s environment — used for granular cost breakdown by namespace/workload inside Kubernetes.

KEDA (Kubernetes Event-Driven Autoscaling)

Purpose: Scales workloads based on external event sources rather than just CPU/memory.
When to use it: When scaling needs to track a business signal like message-queue depth rather than raw compute utilization.

Argo CD

Purpose: GitOps continuous delivery tool for Kubernetes.
When to use it: Already in use by the client for deploying Helm-chart-based microservices to GKE.

AWS Valkey

Purpose: Open-source, AWS-supported Redis-compatible in-memory data store.
When to use it: As a cost-effective alternative to ElastiCache for Redis.
Advantages: Cited as offering roughly a 2/3 cost reduction versus Redis in the participant’s experience (informal figure — verify current pricing).
Limitations: Not detailed further in the transcript; standard migration/compatibility due diligence would apply.

CyberArk (referenced by name)

Purpose: Privileged Access Management (PAM) platform providing short-lived, audited access to sensitive systems.
When to use it: Regulated environments (e.g., banking) requiring tightly controlled, time-boxed SSH/administrative access.
Limitations: Operationally heavy for teams that frequently recreate/re-IP infrastructure (a tension explicitly raised in the discussion).

8. Real-World Production Usage

Enterprise use case — cost optimization at scale: A participant shared a real, first-hand case of saving ~$5 million on AWS over 3 years, taking monthly spend from ~$650K/month down significantly over 18–19 months, through a portfolio of techniques: Graviton migration for RDS, large-scale legacy data cleanup (reducing both storage and cross-region transfer costs), Redshift optimization, Redis→Valkey migration, and an ongoing AMD→ARM infrastructure migration. This is a strong illustration that meaningful FinOps savings usually come from compounding multiple smaller techniques, not one silver-bullet fix.
DevOps / Cloud best practice — inventory before action: Every experienced participant who spoke up echoed the same first move: build a full resource inventory before optimizing anything. This is treated as non-negotiable in real engagements, especially when no architecture documentation exists.
Security consideration — regulated industries and PAM: The CIS Benchmark / PCI-DSS / CyberArk discussion reflects a real pattern in banking and other regulated industries — security tooling requirements (short-lived credentialed access) can directly conflict with modern ephemeral infrastructure practices, and DevOps engineers are expected to navigate that tension, not just implement one side of it.
Cost optimization — network/data transfer often hides in plain sight: The Singapore→Mumbai cross-region backup example (4–5 TB/day, retained for 3 days, ~12–15 TB total) is a realistic illustration of how compliance-driven backup retention policies can become a hidden, compounding cost driver that’s easy to overlook if you only look at compute/storage line items in isolation.
Scalability consideration — multi-cloud for availability, not savings: The explicit framing here — that a 50/50 AWS/GCP split is about surviving a full-provider outage, not chasing cheaper unit pricing — is a useful real-world corrective to the common assumption that “multi-cloud = cost savings.” In practice it usually adds cost and operational complexity in exchange for resilience.
Change management under business constraints: The near-zero-downtime, Saturday-only maintenance window, and willingness to run duplicate infrastructure temporarily during migrations reflects how safety-first, business-constrained migrations actually get executed in production fintech environments — very different from a greenfield lab exercise.

9. Interview Preparation

Beginner Questions

Q1: What is a cluster autoscaler, and why does a Kubernetes cluster need one? A: A cluster autoscaler automatically adjusts the number of nodes in a Kubernetes cluster based on current scheduling demand — adding nodes when pods can’t be scheduled due to insufficient resources, and removing nodes when they’re underutilized. Without one, teams either over-provision (wasting money on idle capacity) or under-provision (causing pods to remain unscheduled and applications to fail to scale).

Q2: What’s the difference between GitHub authentication over HTTPS with a password versus SSH keys? A: GitHub deprecated password-based authentication for Git operations over HTTPS. The modern approach is to generate an SSH key pair, register the public key with your GitHub account (under account settings, not per-repository), and use the SSH remote URL to authenticate Git operations with your private key. Personal Access Tokens (PATs) are the equivalent alternative for HTTPS-based workflows.

Q3: What is an S3 (or GCS) lifecycle policy, and what problem does it solve? A: A lifecycle policy is a rule set that automatically transitions objects to cheaper storage classes or deletes them after a defined period. It solves the problem of storage costs growing indefinitely as data (like CI/CD artifacts or old backups) accumulates without anyone manually cleaning it up.

Intermediate Questions

Q4: Why would an organization choose ARM-based (Graviton) instances over x86 (Intel/AMD), and what’s the catch? A: Graviton instances typically offer better price-performance and lower power consumption for many workloads. The catch is compatibility: applications and their dependencies need to be validated (and sometimes recompiled or have specific ARM-compatible libraries installed) to run correctly on ARM architecture — so it’s not always a zero-code-change migration, even though it’s framed as “infrastructure-only.”

Q5: How would you structure a cost optimization engagement for a client with no existing infrastructure documentation? A: Start with a full inventory-building phase — extract every resource’s metadata (type, size, utilization) across all accounts/environments before making any changes. Then categorize resources by layer (typically network → compute → data) and validate every proposed change in a non-production environment first, measuring actual cost impact before rolling changes to production. This mitigates the risk of breaking something whose purpose isn’t understood.

Q6: What’s the difference between a bastion host and a full PAM (Privileged Access Management) solution like CyberArk, and when would you use each? A: A bastion host is a single, hardened entry point that all SSH traffic must pass through, reducing the attack surface versus exposing every instance directly to the internet. A PAM tool like CyberArk goes further by issuing short-lived, audited, just-in-time credentials/sessions (e.g., valid for only 10–30 minutes) rather than standing SSH access, even through the bastion. Regulated industries (banking, fintech under PCI-DSS) often require PAM-level controls; a bastion alone is a reasonable baseline for less regulated environments, though it introduces friction for teams with fast-changing, ephemeral infrastructure.

Q7: Explain the trade-off between using a managed cloud service versus self-hosting the equivalent open-source tool (e.g., a managed message queue vs. self-hosted Kafka). A: Managed services reduce operational overhead (patching, scaling, backups handled by the provider) but typically cost more and can create migration friction (application code often depends on managed-service-specific APIs/behaviors). Self-hosting can reduce direct service costs but shifts operational burden (uptime, scaling, security patching) onto the team, and — as raised in this transcript — often requires actual application code changes to interact with a different technology, unlike most pure infrastructure-layer optimizations.

Advanced Questions

Q8: A client wants a 50/50 multi-cloud split “for cost savings.” How would you push back or reframe this conversation? A: Clarify the actual goal first. Multi-cloud rarely reduces cost in practice — it usually increases operational complexity (duplicated tooling, cross-cloud networking, data consistency challenges) and often costs more overall due to lost volume discounts on either single cloud. The stronger, more defensible justification for multi-cloud is high availability — e.g., running redundant workload replicas across two providers so a full outage of one doesn’t take down the whole service — which is the framing this client actually used. If cost is the real driver, single-cloud rightsizing, reserved capacity, and architecture optimization usually deliver far more savings with far less complexity.

Q9: How would you scope a security audit for a client with 13+ separate cloud accounts and no architecture documentation, and what audit categories would you define? A: Break the audit into discrete, sequential categories rather than attempting one monolithic audit: (1) cloud account audit — permissions, exposed resources, misconfigurations across every account; (2) CI/CD audit — pipeline security, secret handling, Terraform/IaC review, rollback capability; (3) platform/internal-tooling audit — any custom self-service tools built by the org; (4) architectural audit — reviewing whether the design itself (not just configuration) is inefficient or insecure, covering compute and data layers; (5) backup and disaster recovery audit — with DR explicitly scoped as a separate conversation from routine backup review, since DR involves RTO/RPO-style questions (how fast can the full environment be recreated) that go beyond “do backups exist.”

Q10: What’s the risk profile of making infrastructure changes to a system with no clear ownership or point of contact, and how do you mitigate it? A: The core risk is that a seemingly safe infra-layer change (e.g., resizing an instance, changing a storage class) could unexpectedly break a business-critical dependency that isn’t documented anywhere and that no single person can confirm is safe to touch. Mitigation strategies discussed/implied in this transcript: build a complete inventory before any change; validate every change in a lower environment first (with pre-prod treated as production-critical here); constrain all changes to a narrow, pre-approved maintenance window; and be willing to run duplicate infrastructure in parallel during a migration so a rollback is instantaneous rather than a scramble.

Q11: Compare CastAI, Karpenter, and a standard Kubernetes cluster-autoscaler. When would you recommend each? A: All three solve node-level autoscaling, but differ in sophistication and cost. Standard cluster-autoscaler is the free, baseline Kubernetes-native option — reactive, rules-based. Karpenter (AWS-native) improves on this with faster, more flexible node provisioning, still free, still avoids vendor lock-in — a strong default recommendation for AWS-based clusters with no budget for third-party tooling. CastAI adds predictive, ML-driven cost optimization, a broader feature set (rebalancing, non-Kubernetes managed-service optimization via its “Optimizer,” application monitoring), and can be granted authority to auto-apply changes — appropriate when the organization values a broader, cross-cloud FinOps platform and is willing to pay for it and accept a degree of vendor dependency. The right choice depends on budget, existing tooling investment, cloud provider, and organizational appetite for third-party access to production clusters.

10. Exam & Certification Notes

(Relevant primarily to AWS/GCP cloud certifications and FinOps-adjacent credentials — DevOps/SRE/Cloud Practitioner-level exams frequently test these concepts.)

Graviton/ARM instances are a commonly tested AWS cost-optimization concept — know that they generally offer better price-performance but require compatibility validation.
gp2 vs. gp3 EBS volumes: gp3 is generally cheaper and allows independently configuring IOPS/throughput without needing to resize the volume — a frequently tested EBS optimization fact.
S3 lifecycle policies: Know the difference between transition actions (moving objects to cheaper storage classes like Glacier) and expiration actions (deleting objects outright) — both were referenced conceptually in this session.
Savings Plans vs. Reserved Instances vs. Spot Instances: This session distinguished Savings Plans from Spot pricing as two separate cost levers — a common exam distinction (Savings Plans = compute usage commitment discount; Spot = using spare capacity at variable, interruptible pricing).
CIS Benchmarks: Know that they come in tiers (commonly Level 1 = less restrictive/general-purpose baseline, Level 2 = more restrictive, often for higher-security/compliance environments like PCI-DSS) — a common trick-question distinction (candidates confuse “Level 2 is optional” with “Level 2 is for higher-assurance environments”).
Bastion host vs. PAM: Exams may test the layering of these concepts — a bastion host is a network-architecture control; PAM (e.g., session brokering, credential vaulting) is an identity/access-management control. They’re complementary, not substitutes.
KEDA: Know that it scales based on external event sources (e.g., queue depth, custom metrics) as opposed to core Kubernetes HPA’s default reliance on CPU/memory metrics — a common differentiator question.
Multi-cloud vs. multi-region: A frequently confused distinction — multi-region (within one cloud provider) protects against a regional outage; multi-cloud (across providers) protects against a full provider-level outage but adds significant operational complexity. This session’s use case was explicitly the latter, for HA purposes.

11. Cheat Sheet

Cost Optimization Levers (roughly ordered by “free/easy” → “requires more validation”):

gp2 → gp3 (EBS) — no code impact
Remove orphaned Elastic IPs — no code impact
S3/GCS lifecycle policies on artifacts & backups — no code impact
Non-prod scheduling (start/stop outside business hours) — no code impact
Right-size ECS/Compute instances based on actual utilization — low risk
Cluster autoscaling (Karpenter/cluster-autoscaler/CastAI) — low-to-moderate setup effort
Graviton/ARM migration (compute + RDS) — requires compatibility validation
Legacy data cleanup — requires careful validation (compliance/retention checks)
Redis → Valkey — requires migration testing
Managed service → self-hosted (e.g., Kafka) — requires code changes; separate scoping conversation

Security Audit Checklist (5 categories):

Cloud account audit (ScoutSuite, Prowler)
CI/CD audit (Terraform, Docker images, secrets, rollback, versioning, IAM roles)
Platform/self-service tooling audit
Architectural audit (compute + data + everything)
Backup audit (confirm DR is scoped separately if needed)

SSH Access Hardening Ladder (weakest → strongest):

Direct SSH to instance, open to internet ❌
SSH restricted to specific IP ranges
Bastion host as single entry point
PAM tool (e.g., CyberArk) — short-lived, audited, time-boxed sessions

Multi-Cloud Reality Check:

Multi-cloud ≠ automatically cheaper
Multi-cloud = availability/resilience strategy (surviving a full provider outage)
Multi-region (single cloud) = simpler way to address regional outages

GitHub SSH Setup (3 steps):

Generate SSH key pair locally
Add public key → GitHub account settings → SSH and GPG keys
Clone/push via SSH URL

12. Gaps & Assumptions

Unit for “2 lakh” traffic figure: The transcript states current traffic as “somewhere around two lakh” with an expected 5x increase, but does not specify the unit (requests/second, daily active users, transactions/day, etc.). Treated as an unconfirmed figure — do not assume a specific unit without checking directly with the source.
CastAI pricing: Stated informally as “around $200” possibly per quarter — the instructor themselves said they weren’t fully sure of the exact pricing structure. Not a reliable figure for external quoting.
”GCP IAM Collector” tool name: The instructor named a tool for auditing GCP service accounts but seemed uncertain of the exact name while speaking. Flagged as needing independent verification before relying on it.
Redis → Valkey “2/3 cost reduction” figure: Stated informally by a participant based on their own experience, not verified against current official AWS pricing in this session.
Data-layer strategy for the planned multi-cloud split: The transcript confirms compute/replica-level HA across AWS+GCP but never addresses how databases would stay consistent across two cloud providers (active-active replication? single source of truth? read replicas only?). This is a meaningful open gap in the plan as described.
Ansible audit scope: Left ambiguous — mentioned as potentially needed for the CI/CD audit, then tentatively folded into the “self-service tools” audit instead, without a final decision stated in the transcript.
DR (Disaster Recovery) scope: Explicitly and openly still unresolved at the time of this call — flagged by the team themselves as needing a dedicated scoping conversation with the client, separate from the routine backup audit.
Shared cloud console access for the cohort: No resolution was reached during the call; multiple options were floated (shared Gmail, service account JSON sharing, multi-user service account attachment) but none were confirmed as the final approach.
”E-something” alternative FinOps tool mentioned in passing during the CastAI-alternatives discussion — name was unclear/cut off in the transcript and could not be confidently identified.
Speaker/role attribution: As with the kickoff call, several speaker turns in this transcript are ambiguous due to auto-transcription artifacts (background noise, overlapping speech, unclear names). Technical content was preserved faithfully; specific speaker attribution for individual quotes was intentionally omitted where uncertain.
This is a “shadowing session,” not a scripted lecture: Content flow follows a live, sometimes meandering client conversation rather than a structured curriculum module — some topics (e.g., cost optimization scope) were revisited and refined multiple times across the call as new information emerged. This document consolidates and reconciles those revisions (e.g., using the later, confirmed $40K/$8K cost figures over the earlier rough estimate) rather than presenting them in raw chronological/contradictory form.