SRE Labs (Advanced Track) — Shadowing Session: GCP Inventory Building & Cloud Logging Cost Optimization

Structured educational resource covering sre labs (advanced track) — shadowing session: gcp inventory building & cloud logging cost optimization.

senior 45 min read 11 sections

#cloud-k8s#aws#gcp#cost-optimization

Client: FintechPlatform (Dubai-based Fintech) — Hands-On CLI Walkthrough & Live Cost Analysis

2. Table of Contents

Executive Summary
Table of Contents
Detailed Structured Notes
- 3.1 GCP Startup Credits Program (Context, Not Client-Specific)
- 3.2 Client Infrastructure Snapshot
- 3.3 90-Day Billing Analysis by SKU
- 3.4 Building the GCP Inventory (Full CLI Walkthrough)
- 3.5 GCP Built-In Cost/Security Tools (Active Assist & Security Command Center)
- 3.6 Cloud Logging Deep Dive & Optimization Plan
- 3.7 Client Q&A — Rollback, Bucket Confusion, Application-Level Filtering
- 3.8 Session Wrap-Up & Next Steps
Key Concepts Table
Architecture & Workflow Analysis
Commands & Configurations
Tools & Technologies
Real-World Production Usage
Interview Preparation (Beginner / Intermediate / Advanced)
Exam & Certification Notes
Cheat Sheet
Gaps & Assumptions

3. Detailed Structured Notes

3.1 GCP Startup Credits Program (Context, Not Client-Specific)

Before diving into the client work, the instructor gave a general briefing on GCP’s startup credit program — presented as generally useful knowledge for anyone running or planning to run their own venture, not specific to this engagement:

Eligibility: Startups without significant funding/pre-seed companies, or newly registered organizations.
Credit ceiling: Roughly $350,000 USD (~3.5 Crore INR), valid for 2 years.
Extension: Can be extended to 3 years by directly engaging GCP’s sales team.
AI usage increases credit eligibility: Startups actively using GCP AI services (e.g., Vertex AI for MLOps workloads) can qualify for additional credit beyond the baseline program.
Application requirements: Business registration details, GST number (for India-based entities) or equivalent documentation (for other regions, e.g., Dubai-based entities have a different document set), business address, and a description of the business.
Process: GCP’s sales team reviews the application and verifies the nature of the business; the overall process takes about 1 month to complete.
Client’s actual situation: FintechPlatform had already received and fully exhausted a $100K credit grant, and is now on a paid billing plan. Because they can now demonstrate genuine AI/MLOps usage (Vertex AI), the instructor expects GCP to approve an additional ~$250K in credits by around March.

3.2 Client Infrastructure Snapshot (This Specific GCP Project)

Attribute	Detail
Project name (as referenced)	FintechPlatform (specific sub-project under the client’s broader GCP footprint)
Monthly billing	~$8,000/month
Core infra components	Cloud Logging, Cloud Monitoring, a self-hosted Prometheus + Grafana monitoring stack, GKE, various VM (Compute Engine) instances, GCS buckets, Redis (Memorystore), Cloud SQL
Billing analysis window used	Last 90 days, filtered by SKU (Stock Keeping Unit — GCP’s per-service/per-resource billing line-item identifier)

3.3 90-Day Billing Analysis by SKU

The instructor used GCP’s billing console, filtered by SKU, to break down where the $8K/month is actually going:

Cost Component	90-Day Cost	% of Total Cost	Notes
Cloud Logging (log storage)	~$15,000	44%	~30,000 GB (30 TB) of log storage over 90 days — by far the single largest cost driver
VMs (Compute Engine)	~$7,500	21%	Includes at least one T2 AMD instance running in the Finland region (~~$3,600 over 3 months) and another single instance (~~$1,900 over 3 months)
Cloud Monitoring	~$3,800	11%	Second-largest single line item after logging
Cloud SQL	~$1,500	—
Redis (Memorystore)	~$1,400	—
Networking	(not fully quantified in the transcript)	—	Discussion was interrupted by a screen-share/Discord auto-timeout issue before the exact figure was stated

Key framing from the instructor: This client’s infrastructure is not particularly compute-heavy — the overwhelming majority of avoidable cost is concentrated in logging and monitoring, not in oversized compute resources. This directly determined the prioritization: logging first (highest cost + most straightforward fix), monitoring second (related to logging and next-highest cost).

3.4 Building the GCP Inventory (Full CLI Walkthrough)

Why this matters: Before making any optimization change, the team builds a full point-in-time inventory of every resource in the account. This serves two purposes: (1) a working dataset to analyze for optimization opportunities, and (2) a defensible audit trail — if the client later adds new infrastructure and complains that costs “haven’t gone down,” the team can point to the exact inventory snapshot that scoped their original work, showing that anything added afterward falls outside their responsibility.

Step-by-step process (as demonstrated live):

Install the Google Cloud SDK (gcloud CLI) — required for any local/CLI interaction with GCP.
Verify installation: gcloud version
Authenticate: gcloud auth login — opens a browser window; must log in with the Google account tied to your GCP access.
Set the active project: gcloud config set project <PROJECT_ID> — in this case, the FintechPlatform project ID.
Handle pre-existing local configs: If your machine already has gcloud configured for a different project, you need to clear the existing Application Default Credentials (ADC) and reconfigure — otherwise you’ll run into conflicts or see the wrong project’s data.
Confirm configuration: run a token-related check command (referred to as “token printing” in the call — functionally equivalent to gcloud auth print-access-token) to confirm the CLI is correctly authenticated and pointed at the right project.

Local folder structure created for the inventory:

fintechplatform-gcp-inventory/
├── compute/     (VM instance data)
├── gke/         (GKE cluster + node pool data)
├── ops/         (logging + monitoring inventory)
├── db/          (Cloud SQL + Redis inventory)
├── network/     (external IPs, load balancer forwarding rules)
└── storage/     (GCS bucket configuration)

Created with a simple mkdir -p command chain for each subdirectory.

Data pulled per category (fields and rationale):

Compute (VMs): name, zone, machine type, status, labels, tags, network interfaces, attached IP, disk size — exported to gcp_instances.csv.
GKE: cluster name, location, status, node pool configuration — exported to a clusters CSV; separately, a node pool inventory (node pool name, region, cluster name) and a node labels/taints inventory — explicitly called out as important for cost analysis, since node pool sizing and taints affect what workloads land where (and therefore what you’re paying for).
Cloud SQL: instance name, database engine/version, region, tier, disk size, availability configuration (regional vs. multi-regional), status.
Redis (Memorystore): instance name, size/tier, API version, and related config data.
GCS buckets: bucket-level configuration inventory.
Networking: external IP inventory (which instances have public IPs attached) and load balancer forwarding rules inventory.
Logging/Monitoring: log bucket/group inventory and monitoring configuration data.

Consolidation step: All the individual CSV inventories were merged into a single Excel workbook using a Python script built with pandas and the openpyxl module — giving the team one unified spreadsheet to work from instead of switching between multiple files/tabs in the GCP console.

3.5 GCP Built-In Cost/Security Tools (Active Assist & Security Command Center)

Active Assist (introduced as GCP’s equivalent to AWS Trusted Advisor):

Cost recommendations tab: In this client’s account, Active Assist identified:
- A Committed Use Discount (CUD) opportunity on compute (GCP’s equivalent of an AWS Savings Plan) — estimated savings of ~$500/month.
- A rightsizing opportunity on one instance — estimated savings of ~$35/month.
- Total identified savings: ~$800/month from these two recommendations alone.
Security recommendations tab: Flagged gaps including:
- Enabling the user password policy on Cloud SQL.
- Enabling database auditing.
- Enabling the instance password policy.
Reliability recommendations tab:
- Add a Pod Disruption Budget (PDB) for GKE deployments.
- Increase backup retention on Cloud SQL.
- Create a backup plan for the GKE cluster.
- Set up a maintenance window on the GKE cluster.
Performance recommendations tab: Analyzes application traffic patterns and surfaces performance optimization factors (not detailed further in the transcript).
Key differentiator vs. AWS Trusted Advisor: GCP’s Active Assist is free; AWS Trusted Advisor’s full recommendation set requires a paid support tier.

Security Command Center (SCC):

A paid GCP service (unlike Active Assist) that provides a risk overview dashboard, monitors inbound/outbound traffic, detects threats (e.g., DoS-style attacks), and can actively block malicious activity.
Specific client relevance: The client currently deals with bots scraping data from their product, and is manually and repeatedly blocking offending IPs at the ingress layer. SCC was identified as a candidate to formalize and automate this (rather than continuing ad hoc manual IP blocking) — the instructor plans to enable and configure SCC as part of the security-enhancement track of this engagement.
Recommended specifically because the client is a fintech organization handling sensitive data — the instructor’s framing was that this class of business justifies the added SCC cost given the risk profile.

3.6 Cloud Logging Deep Dive & Optimization Plan

Step 1 — Inspect existing log buckets:

Using a gcloud logging buckets list command (location scoped to global), the team found two log buckets:

Bucket ID	Purpose	Current Retention
`_Default` (referred to as “default”)	Production/application/infrastructure logs	30 days
`_Required` (referred to as “required”)	Audit logs — includes all data in the Default bucket plus all third-party tool interaction logs, needed for compliance/audit calls	400 days

Important clarification surfaced by a participant’s question and confirmed live: The _Required bucket is effectively a superset of the _Default bucket’s content — it contains everything the default bucket has, plus third-party integration/interaction logs needed specifically for compliance and audit purposes. The 400-day retention on this bucket is a fixed compliance baseline that cannot be reduced.

Step 2 — Confirm actual retention needs with the client:

The instructor explicitly asked the client how long they actually need production/application log data retained (separate from the fixed 400-day audit requirement). The client’s answer: 7–14 days is sufficient for their product logging needs.

Step 3 — Calculate the cost impact of reducing retention:

Reducing the _Default bucket’s retention from 30 days → 14 days is expected to cut that bucket’s storage cost by roughly 50%.
The math walked through live: at the current logging volume, retaining 30 days of data works out to storing significantly more total data than retaining 14 days would — directly translating to a proportional reduction in the $15,000/90-day (~$5,000/month) logging storage bill.
⚠️ Note on this calculation (see Section 12, Gaps & Assumptions): the specific per-day volume figure used live in the call (a “30 × 30 = 900” calculation) appears inconsistent with the ~30,000 GB/90-day figure stated earlier in the same session. The direction and general magnitude of the savings (cutting retention roughly in half yields roughly half the storage cost) is sound and directionally reliable; treat the specific intermediate numbers with caution and verify against the client’s actual billing data rather than the live-call arithmetic.

Step 4 — Categorize logs and identify what to drop:

The team built a log categorization (via a Python script) breaking logs down into categories including:

Application logs
Infrastructure logs
Debug logs
Info logs
Stdout logs (“STO log” in the transcript — standard output logs)

This categorization was shared with the client, who reviewed it and identified specific gaps/waste.

Step 5 — Agreed optimization actions:

Change retention on the _Default bucket from 30 → 14 days.
Drop debug and info logs entirely — confirmed as non-essential and not part of any audit requirement. A dedicated script will be written to filter these out.
Drop stdout logs — client explicitly confirmed these are not needed.
Exclude kube-system namespace logs from GKE, along with other “noisy” container/application logs — requires first auditing what each container is actually logging, then selectively excluding non-essential log sources at the GKE/cluster level.
Build a rollback/safety script — see Section 3.7 for the important nuance on what this script can and can’t actually do.
Implementation timing: All of the above is planned to be implemented starting the following Monday, in the next working session — this call was scoping/planning only, not execution.

3.7 Client Q&A — Rollback, Bucket Confusion, Application-Level Filtering

Several participant questions surfaced important operational nuances worth preserving:

On the “rollback script” — a critical clarification: A participant pressed on a subtle but important point: if you write a script to filter/drop certain log lines, and later want to “roll back” and restore them, where does the removed data actually come from? The instructor’s answer, after being pushed on this: there is no true data restoration — once historical log lines are deleted, they’re gone permanently (in ~99% of scenarios). The “rollback script” is better understood as a future-facing safety mechanism: if the team later decides they need a category of logs they previously excluded, the script provides a quick way to stop excluding that category and start capturing it again going forward — it does not retroactively recover data that was already deleted before the script existed. This is a genuinely important distinction to internalize: log filtering and log restoration are not symmetric operations.

On bucket confusion (Default vs. Required): Multiple participants were confused about what distinguishes the two log buckets. Clarified: the _Default bucket holds standard production/application/infrastructure logs; the _Required bucket holds everything the Default bucket has, plus all third-party tool integration/interaction logs needed specifically for compliance and audit review — i.e., _Required is a strict superset used for audit purposes, and its 400-day retention is non-negotiable.

On whether log buckets and storage (GCS) buckets are the same thing: Confirmed: yes — GCP log buckets are GCS buckets under the hood; there’s no separate distinct storage mechanism to reason about. In this client’s account, these two log buckets are the only GCS buckets in use — no other unrelated data is being stored there.

On whether logging changes could instead happen at the application/developer level (i.e., stop generating unnecessary logs in the first place, rather than filtering them downstream): A participant raised this as the arguably “more correct” long-term fix. The instructor confirmed they had already proposed exactly this to the client — but the client explicitly declined to make changes at the application/development level at this time. As a result, all log volume reduction has to be implemented at the cloud/infrastructure layer (log filtering/exclusion rules, retention policy), not by changing what the application emits. This is a realistic example of a technically “better” solution being blocked by client constraints, requiring the team to solve the problem within the boundary they’re actually given.

On documentation/best practices: A participant asked whether there’s Google-published guidance on log optimization best practices to reference. The instructor committed to finding and sharing the relevant GCP documentation link in the group afterward (not resolved within the call itself).

3.8 Session Wrap-Up & Next Steps

Recap given at the end of the call:

Inventory of the current infrastructure has been captured locally and will double as the audit-trail baseline for the engagement.
Cloud Logging has been identified and confirmed as the first cost-optimization target (highest cost, most tractable fix).
The specific logging optimization plan (retention change + category filtering) has been scoped and is ready for implementation.

Planned for the next session:

Implement the logging optimization (retention change to 14 days + dropping debug/info/stdout/kube-system logs).
Move to Cloud Monitoring cost optimization (next-highest cost item after logging).
Demo a full infrastructure security scan — the instructor mentioned being able to run a comprehensive security scan of the entire cloud infrastructure “in one go with one script” — details deferred to the next session.

Recurring call schedule mentioned:

Drill call (hands-on): 7:00 p.m. IST
Theory call: 9:00 p.m. IST
Participants can attend whichever matches their need — hands-on practice vs. conceptual/theoretical questions.

4. Key Concepts Table

Concept	Explanation	Example	Why It Matters
SKU-based billing analysis	GCP bills are broken down by SKU (Stock Keeping Unit) — a granular identifier for each billable resource/service line item	Filtering the 90-day billing view by SKU to isolate “log storage” as its own line item	This is the fastest way to find out exactly where cloud spend is actually going, rather than guessing based on intuition about which services “should” be expensive
Log bucket retention policy	The number of days GCP retains log data in a given log bucket before automatically deleting it	Reducing the `_Default` bucket’s retention from 30 to 14 days	Directly and proportionally reduces log storage cost — one of the simplest, lowest-risk cost levers available
`_Default` vs. `_Required` log buckets	GCP’s built-in log bucket structure: `_Default` for general logs, `_Required` for audit-mandated logs with a fixed (typically non-configurable) retention floor	`_Required` retained at 400 days regardless of cost, because it’s compliance-mandated	Understanding this distinction prevents accidentally proposing changes to a bucket that legally/contractually cannot be touched
Log category filtering	Selectively excluding certain types of logs (debug, info, stdout, noisy system namespaces) from being ingested/stored at all	Excluding `kube-system` namespace logs from a GKE cluster	Reduces both storage cost and log noise, making genuinely important logs easier to find
GCP Active Assist	GCP’s free, built-in recommendation engine covering cost, security, reliability, and performance	Surfacing a $500/month CUD opportunity and a $35/month rightsizing opportunity	A no-cost first step for any GCP cost/security audit — should be checked before reaching for paid third-party tools
Committed Use Discount (CUD)	GCP’s equivalent of AWS Savings Plans / Reserved Instances — a discount in exchange for committing to a certain usage level over time	A CUD recommendation surfaced by Active Assist saving ~$500/month	A standard, low-effort cost lever once usage patterns are stable enough to commit to
Security Command Center (SCC)	GCP’s paid security monitoring and threat-detection service	Considered for automating IP-blocking against bots scraping the client’s product	Provides centralized, automated threat detection/blocking instead of manual, reactive IP blocking
Inventory-as-audit-trail	Capturing a full point-in-time snapshot of infrastructure before making changes, to later prove what was/wasn’t in scope	If the client later adds new resources and complains costs haven’t dropped, the team can point to the original inventory to show the new resources are outside the original optimization scope	Protects the consulting team from being held accountable for cost growth outside their actual work
Asymmetry of log filtering vs. restoration	Deleting/filtering log data is generally irreversible; a “rollback script” can only prevent the same exclusion going forward, not recover already-deleted historical data	Team clarifies with the client that dropped debug/info logs cannot be truly “brought back” once deleted	Sets correct client expectations and avoids overpromising what a “rollback plan” can actually deliver
Application-level vs. infrastructure-level fixes	The same problem (excessive logging) can theoretically be fixed at the source (application code) or downstream (infra-level filtering)	Client declines to change what the application logs; team must solve it entirely via cloud-side filtering/retention	A recurring real-world constraint: the “cleanest” technical fix isn’t always the one you’re permitted to implement

5. Architecture & Workflow Analysis

5.1 GCP Inventory-Building Workflow

1. Install & Authenticate gcloud CLI
   gcloud version → gcloud auth login → gcloud config set project <ID>
        |
        v
2. Create Local Folder Structure
   fintechplatform-gcp-inventory/
     ├── compute/  ├── gke/  ├── ops/  ├── db/  ├── network/  ├── storage/
        |
        v
3. Pull Resource Data Per Category (gcloud CLI, CSV output)
   Compute → gcp_instances.csv
   GKE     → gke_clusters.csv + node_pools.csv + node_labels.csv
   DB      → cloud_sql.csv + redis.csv
   Network → external_ips.csv + lb_forwarding_rules.csv
   Storage → gcs_buckets.csv
   Ops     → log_buckets.csv + monitoring_config.csv
        |
        v
4. Consolidate with Python (pandas + openpyxl)
   All CSVs --> single unified Excel workbook
        |
        v
5. Use as Baseline for:
   (a) Cost optimization analysis
   (b) Defensible "before" snapshot / audit trail

5.2 Cost Optimization Prioritization Flow (This Engagement)

90-Day SKU Billing Analysis
        |
        v
Rank cost contributors:
  1. Cloud Logging   (44%, ~$15,000/90 days)  <-- highest priority
  2. VMs             (21%, ~$7,500/90 days)
  3. Cloud Monitoring(11%, ~$3,800/90 days)   <-- next priority (related to logging)
  4. Cloud SQL       (~$1,500/90 days)
  5. Redis           (~$1,400/90 days)
  6. Networking      (not fully quantified)
        |
        v
Target #1 first: Cloud Logging
        |
        v
    -----------------------------
    |                            |
Retention Reduction         Log Category Filtering
(30 days -> 14 days          (drop debug/info/stdout,
 for _Default bucket)         exclude kube-system logs)
    |                            |
    -----------------------------
        |
        v
Client sign-off --> Implement in next session --> Move to Cloud Monitoring

5.3 GCP Log Bucket Structure (This Client)

GCP Project: FintechPlatform
        |
   Cloud Logging
        |
   -------------------------
   |                        |
_Default bucket        _Required bucket
(30 -> 14 days)         (400 days, FIXED)
   |                        |
Production logs         Everything in _Default
Application logs        PLUS
Infra logs               Third-party tool
Debug logs (drop)        interaction logs
Info logs (drop)         (audit/compliance mandated,
Stdout logs (drop)        cannot be shortened)
kube-system logs (drop)

5.4 GCP Built-In Tooling Landscape (as covered)

                     GCP Account
                          |
        -----------------------------------
        |                                 |
   Active Assist                 Security Command Center
   (FREE)                        (PAID)
        |                                 |
   --------------------             Risk overview dashboard
   |      |      |     |            Inbound/outbound traffic monitoring
 Cost  Security Reliab. Perf.       DoS/threat detection & blocking
   |      |      |     |            --> Candidate to replace manual
 CUD   Password  PDB   Traffic          IP-blocking of scraper bots
 recs  policies  Backup analysis
       Auditing  plans
                 Maint.
                 windows

6. Commands & Configurations

Command / Config	Purpose	Explanation
`gcloud version`	Verify the Google Cloud SDK is installed and check its current version	First sanity check after installing the SDK
`gcloud auth login`	Authenticate the local CLI against your Google account	Opens a browser window; you must log in with the same account that has access to the target GCP project
`gcloud config set project <PROJECT_ID>`	Set the active/default GCP project for subsequent CLI commands	Example given: setting the project ID for the “FintechPlatform” project
`gcloud auth print-access-token` (referred to as “token printing” in the call — exact syntax as commonly used)	Confirm the CLI is correctly authenticated and pointed at the right project	Used as a final verification step after login + project config
`mkdir -p compute gke ops db network storage` (pattern)	Create the categorized local inventory folder structure in one shot	`-p` flag allows creating nested/multiple directories without erroring if some already exist
`gcloud compute instances list --format=csv(name,zone,machineType,status,labels,tags,networkInterfaces,...)` (pattern as described)	Export a full inventory of Compute Engine VM instances to CSV	Fields captured: name, zone, machine type, status, labels, IP, disk size — output to `gcp_instances.csv`
`gcloud container clusters list --format=...` (pattern)	Export GKE cluster inventory (name, location, status, node pool config)	Output to a GKE clusters CSV
`gcloud container node-pools list --cluster=<name> --region/zone=<region>` (pattern)	Export node pool inventory per GKE cluster, including node labels/taints	Called out as important for cost analysis — node pool sizing directly drives compute spend
`gcloud sql instances list --format=...` (pattern)	Export Cloud SQL instance inventory	Fields: name, database version, region, tier, disk size, availability (regional/multi-regional), status
(Redis/Memorystore inventory command — same pattern, exact syntax not fully spoken on screen)	Export Redis instance inventory	Fields: instance name, size/tier, API version, related config
`gcloud logging buckets list --location=global`	List all log buckets and their retention settings	Revealed the two buckets in this account: `_Default` (30-day retention) and `_Required` (400-day retention)
(Log bucket retention update command — referenced as “no retention this” in the transcript, exact syntax not fully captured)	Update a log bucket’s retention period	Used to change `_Default` bucket retention from 30 → 14 days
Python script using pandas + openpyxl	Consolidate all category-specific CSV inventories into a single Excel workbook	Enables unified, spreadsheet-based analysis instead of juggling multiple CSVs/console tabs
Python script (log categorization)	Parse and categorize logs into application/infra/debug/info/stdout buckets	Used to identify exactly which log categories are safe to drop
Python script (log filtering / drop script — planned)	Filter out and stop ingesting debug, info, and stdout log categories, plus `kube-system` namespace logs	To be implemented in the following week’s session
Python script (rollback/safety script — planned)	Provide a mechanism to re-enable a previously excluded log category going forward	Not a true data-restoration mechanism — see Section 3.7 for the important caveat

7. Tools & Technologies

Google Cloud SDK (`gcloud` CLI)

Purpose: Command-line interface for interacting with and managing GCP resources.
When to use it: Any scripted/automatable interaction with GCP — inventory building, resource management, scripting.
Advantages: Mandatory and foundational for any serious GCP work beyond the console UI; enables scripting and automation.
Limitations: Requires careful handling of local Application Default Credentials (ADC) when switching between multiple projects, or you’ll get conflicts/wrong-project data.

GCP Active Assist

Purpose: Free, built-in recommendation engine covering cost, security, reliability, and performance.
When to use it: As a first-pass, no-cost review before engaging paid third-party FinOps or security tools.
Advantages: Free (unlike AWS Trusted Advisor’s paid tier); covers four distinct recommendation categories in one place.
Limitations: Recommendations are general-purpose; still requires human judgment to apply business context (as discussed in the prior session regarding third-party tools like CastAI).

GCP Security Command Center (SCC)

Purpose: Centralized security monitoring, threat detection, and traffic analysis for a GCP environment.
When to use it: For organizations (especially regulated ones like fintech) that need automated threat detection/blocking rather than manual, reactive security responses.
Advantages: Can detect and help block malicious traffic patterns (e.g., DoS attacks, scraping bots) automatically.
Limitations: Unlike Active Assist, this is a paid service — cost needs to be weighed against the value of automating what’s currently a manual process.

Prometheus + Grafana

Purpose: Open-source monitoring (Prometheus) and visualization (Grafana) stack.
When to use it: Already in use by this client alongside GCP’s native Cloud Monitoring — a common pattern where teams run both native cloud monitoring and a self-hosted/open-source stack in parallel.
Note: Not explored in depth in this session; flagged as adjacent context for the client’s monitoring cost analysis (upcoming next-session topic).

pandas + openpyxl (Python)

Purpose: Data manipulation (pandas) and Excel file generation (openpyxl) libraries.
When to use it: Consolidating multiple CSV inventory exports into a single, unified Excel workbook for easier analysis.
Advantages: Avoids manual copy-pasting between files; fully scriptable and repeatable for future inventory refreshes.

GCP Cloud Logging

Purpose: GCP’s centralized log management and storage service.
When to use it: Default logging destination for GCP resources (VMs, GKE, managed services).
Key cost lever: Log bucket retention period and log category/volume filtering — both directly drive storage cost.
Limitations/gotchas: The _Required bucket’s retention is compliance-driven and generally not something you can reduce; only the _Default bucket’s retention and log category inclusion are typically within your control.

8. Real-World Production Usage

SKU-level billing analysis as a starting point: This session is a clean, real-world demonstration of the correct first move in any cloud cost optimization engagement — don’t guess at what’s expensive, filter the actual billing data by SKU/service and let the numbers direct your prioritization. In this case, intuition might have pointed toward compute or database costs, but the data showed logging was the dominant cost by a wide margin.
Building an inventory as legal/contractual protection: The framing of the inventory snapshot as a defensible “here’s what existed when we started” artifact is a genuinely useful real-world consulting practice — it protects a services team from being blamed for cost growth caused by resources the client added independently after the engagement began.
Free-tier-first tooling strategy: Checking GCP’s free Active Assist recommendations before reaching for paid third-party tools (echoing the CastAI vs. Karpenter/cluster-autoscaler discussion from the prior session) is a consistent, sound FinOps practice — always exhaust free, cloud-native options before introducing new vendor spend.
Compliance-aware cost optimization: The strict separation between the _Default (optimizable) and _Required (compliance-fixed) log buckets is a realistic pattern in regulated industries — cost optimization work always has to be checked against compliance/audit requirements before being applied, and some costs are simply non-negotiable.
Managing client expectations around irreversible operations: The rollback-script clarification is a valuable, realistic lesson in client communication — technical teams need to be precise about what a “rollback” or “safety net” actually does, especially for operations (like log deletion) that are fundamentally irreversible once executed.
Working within client-imposed constraints: The client’s refusal to change application-level logging behavior, forcing the entire fix to live at the infrastructure layer, is a common real-world constraint. Production engineers frequently have to solve a problem with a subset of the levers they’d ideally want, and this session shows that negotiation playing out directly.

9. Interview Preparation

Beginner Questions

Q1: What is a GCP SKU, and why would you filter billing data by it? A: SKU (Stock Keeping Unit) is GCP’s granular identifier for individual billable line items (e.g., a specific type of log storage, a specific VM configuration). Filtering billing data by SKU lets you see exactly which specific resource or service is driving cost, rather than looking at an aggregated total that hides where the money is actually going.

Q2: What’s the difference between GCP’s _Default and _Required log buckets? A: The _Default bucket holds general production/application/infrastructure logs and has a configurable retention period. The _Required bucket holds everything in _Default plus additional audit/compliance-relevant data (like third-party tool interactions), and has a fixed, typically non-reducible retention period (400 days in this case) because it’s tied to compliance requirements.

Q3: What is gcloud auth login used for? A: It authenticates your local gcloud CLI session against a Google account, via a browser-based login flow, so subsequent CLI commands run with that account’s permissions against GCP resources.

Intermediate Questions

Q4: You’re told a GCP account’s logging costs are unexpectedly high. Walk through how you’d investigate and address it. A: Start by filtering billing data by SKU to confirm logging is actually the driver and quantify it precisely. Next, list the log buckets (gcloud logging buckets list) to see current retention settings — check which buckets are freely configurable (like _Default) versus compliance-mandated (like _Required). Confirm actual business retention needs directly with stakeholders (don’t assume the current setting reflects a real requirement). Categorize the log volume by type (application, infra, debug, info, stdout) to identify categories that can be safely excluded. Propose a combined plan: reduce retention on the configurable bucket, and filter out non-essential log categories at the source (or via exclusion filters if application-level changes aren’t feasible).

Q5: Why is GCP’s Active Assist described as a meaningful differentiator versus AWS Trusted Advisor? A: Active Assist’s core recommendation set (cost, security, reliability, performance) is free on GCP, whereas AWS Trusted Advisor’s full recommendation coverage requires a paid AWS Support plan. This makes Active Assist a genuinely free first-pass audit tool, which matters for cost-conscious engagements or smaller organizations that haven’t invested in premium cloud support tiers.

Q6: A client asks for a “rollback plan” before you delete/filter historical log data. What do you need to clarify with them before proceeding? A: Clarify that filtering or deleting log data going forward is generally irreversible for data that’s already been removed — a “rollback script” can only change future behavior (e.g., re-enabling a previously excluded log category so new data starts flowing again), not retroactively recover data that no longer exists. Setting this expectation up front avoids a client believing they have a full safety net for a change that, by nature, doesn’t have one for historical data.

Advanced Questions

Q7: Design a systematic inventory-building process for a GCP account with no existing documentation, using the gcloud CLI. What categories would you capture, and why does the organizational structure matter? A: Organize by resource domain — compute (VM metadata: type, zone, status, IPs, disks), GKE (clusters, node pools, node labels/taints — critical because node pool configuration directly drives compute cost and scheduling behavior), databases (Cloud SQL and Redis/Memorystore, including tier, region, availability configuration), networking (external IPs and load balancer rules — often a source of orphaned/wasted spend), storage (GCS bucket configuration), and ops (logging and monitoring configuration). Structuring the inventory this way lets you reason about cost and risk by domain rather than as one undifferentiated resource list, and it maps cleanly onto how cloud billing itself is typically broken down (compute, storage, networking, managed services), making the subsequent cost-optimization analysis far more tractable. Consolidating everything into a single unified spreadsheet (e.g., via a pandas/openpyxl script) at the end makes cross-domain analysis and reporting much easier than working across a dozen separate CSVs.

Q8: A client refuses to let you make application-level logging changes, but wants logging costs reduced. What are your levers, and what are the trade-offs of each? A: With application-level changes off the table, your levers are entirely at the infrastructure/platform layer: (1) reduce log retention period on any non-compliance-mandated buckets — straightforward, proportional cost reduction, no risk to application behavior; (2) apply log exclusion filters at the ingestion layer (e.g., excluding specific namespaces like kube-system, or excluding debug/info/stdout severity levels) — effective but requires careful auditing first to ensure you’re not excluding something operationally or contractually necessary; (3) evaluate whether some “logs” are actually better served by a cheaper storage tier or export destination rather than staying in the primary logging service’s most expensive storage class (not explicitly covered in this transcript, but a standard complementary lever). The main trade-off across all of these: you’re optimizing cost without touching the source of the problem (excessive log emission), so the underlying inefficiency persists in the application even after the infra-layer fix — this is a durable workaround, not a root-cause fix, and should be documented as such for the client.

Q9: How would you reconcile a scenario where the aggregate billing figure for a service (e.g., “30,000 GB over 90 days”) doesn’t cleanly match a per-day calculation presented separately (e.g., “30 GB/day × 30 days”)? What would you do before presenting a cost-savings number to a client? A: Treat any live, back-of-envelope arithmetic as provisional until cross-checked against the actual billing/usage data pulled directly from the cloud provider. In this case, the two figures given in the session (30,000 GB over 90 days, versus an implied ~30 GB/day × 30 days = 900 GB calculation) don’t reconcile cleanly, and a careful engineer would pull the actual daily log ingestion volume from Cloud Logging metrics (or the billing export) rather than relying on a rounded mental-math example used for illustrative purposes in a live call. Before presenting a specific savings figure externally (to a client or in a report), always verify it against the authoritative billing/usage source rather than a live-session approximation.

10. Exam & Certification Notes

(Relevant to GCP Associate Cloud Engineer / Professional Cloud Architect / FinOps-adjacent certifications.)

gcloud auth login vs. gcloud auth activate-service-account: Know the distinction — interactive user login (browser-based) versus non-interactive service account authentication (common for CI/CD pipelines). This session used interactive login since it was manual CLI work by an individual.
GCP log bucket retention: Know that _Default and _Required are the two built-in log buckets in every GCP project, and that _Required has a compliance-oriented, generally fixed retention (this session’s client had it set to 400 days) — a common exam distinction between “configurable” and “system-mandated” GCP defaults.
Committed Use Discounts (CUDs): GCP’s discount mechanism for committing to sustained resource usage — the direct GCP analog to AWS Reserved Instances/Savings Plans. Exams frequently test recognizing this as a cost-optimization lever distinct from sustained-use discounts (which apply automatically without commitment, for certain resource types).
Active Assist vs. Recommender: Active Assist is the umbrella/brand name for GCP’s recommendation capabilities across cost, security, reliability, and performance; “Recommender” (referenced in the prior session in this series) is the underlying API/service that powers these recommendations. Exams may test recognizing these as connected but distinctly named GCP concepts.
Application Default Credentials (ADC): Understand that ADC is the mechanism gcloud/GCP client libraries use to discover credentials automatically, and that switching between multiple GCP projects/accounts on one machine requires managing ADC carefully to avoid targeting the wrong project.
Security Command Center tiers: SCC has different service tiers (not detailed in this transcript) with different levels of threat detection capability — a common exam nuance is knowing SCC is a paid, opt-in service rather than automatically enabled.

11. Cheat Sheet

GCP Inventory-Building Commands (pattern reference):

gcloud version
gcloud auth login
gcloud config set project <PROJECT_ID>
gcloud auth print-access-token          # verify auth

gcloud compute instances list --format=csv(...)     > compute/gcp_instances.csv
gcloud container clusters list --format=...          > gke/gke_clusters.csv
gcloud container node-pools list --cluster=<name>    > gke/node_pools.csv
gcloud sql instances list --format=...                > db/cloud_sql.csv
gcloud logging buckets list --location=global         > ops/log_buckets.csv

Cloud Logging Cost Optimization — Two Independent Levers:

Retention period — shorten non-compliance-mandated buckets (e.g., 30 → 14 days)
Log category filtering — drop debug/info/stdout logs; exclude noisy namespaces (e.g., kube-system)

GCP Free vs. Paid Optimization/Security Tools:

Tool	Cost	Covers
Active Assist	Free	Cost, security, reliability, performance recommendations
Security Command Center	Paid	Threat detection, traffic monitoring, active blocking

Log Bucket Rules of Thumb:

_Default bucket → generally safe to optimize (retention + filtering)
_Required bucket → compliance-mandated, generally do not touch retention
Log buckets are GCS buckets under the hood — no separate storage mechanism

Rollback Script Reality Check:

✅ Can: prevent the same log category from being excluded again in the future
❌ Cannot: restore log data that has already been deleted

Cost Optimization Prioritization Rule: Always start from actual SKU-level billing data, not assumptions — rank cost contributors, then tackle the largest, most tractable item first.

12. Gaps & Assumptions

Client name confirmation: This session names the client as “FintechPlatform” (transcribed variably as “FintechPlatform”/“Polomial”/“Polonomial” across the audio) — treated as the same client referenced anonymously as “the fintech/Web3 client” in the prior shadowing session in this series, based on matching context (Dubai-based, GCP-primary, similar cost figures). Worth confirming the exact spelling directly.
Inconsistent logging volume figures: The session states ~30,000 GB (30 TB) of log storage over a 90-day window in one part of the discussion, but later uses an illustrative “30 × 30 = 900” calculation that implies a per-day volume inconsistent with that figure. This document flags the inconsistency explicitly rather than silently reconciling it (see Section 3.6) — verify actual daily/monthly log ingestion volume directly from GCP billing/metrics before using either figure externally.
Exact gcloud command syntax: Several commands were described verbally and shown on-screen but not always fully dictated/spelled out in the transcript (e.g., the exact log bucket retention-update command, the exact Redis/Memorystore inventory command). This document presents the pattern and intent of each command based on standard gcloud CLI conventions; exact flags should be verified against current GCP CLI documentation before being used in a real environment, since GCP CLI syntax can change between SDK versions.
Networking cost figure: The exact 90-day networking cost was never fully stated in the transcript due to a mid-discussion screen-sharing interruption (a Discord auto-timeout setting). Left unquantified in this document rather than guessed.
”STO log” terminology: Interpreted as stdout logs based on context (a standard, commonly-excluded log category); the transcript’s phonetic rendering (“STO log,” “studio uh logs”) doesn’t perfectly confirm this — flagged as a reasonable but not certain interpretation.
GCP Startup Credits section is general context, not engagement-specific: The instructor presented this as broadly useful knowledge (applicable to any startup, including participants’ own future ventures) rather than a client deliverable — included here for completeness since it was explicitly taught content, but it’s not part of the FintechPlatform cost-optimization scope itself.
”Token printing” command: Referenced informally in the transcript without the instructor stating the exact command syntax on screen; represented in this document as functionally equivalent to gcloud auth print-access-token, GCP’s standard way to verify an active, valid authentication token — worth confirming against what was actually run if you have access to the session recording.
This document consolidates a live, sometimes-interrupted call (multiple screen-sharing dropouts due to a Discord timeout setting) — where the same topic was revisited after an interruption, this document presents the reconciled, final version of that discussion rather than the raw back-and-forth.