SRE Labs (Advanced Track) — Shadowing Session: GCP Cloud Monitoring Cost Optimization & Security Scanning

Structured educational resource covering sre labs (advanced track) — shadowing session: gcp cloud monitoring cost optimization & security scanning.

junior 20 min read 11 sections

#kubernetes#cloud-k8s#aws#gcp

FintechPlatform Client — Metric Exclusion, Prometheus Scrape Tuning, Cloud Credit Sourcing, ScoutSuite/Prowler

2. Table of Contents

Executive Summary
Table of Contents
Detailed Structured Notes
- 3.1 GCP/Cloud Credit Sourcing Strategy (General Knowledge)
- 3.2 Client Observability Stack — Current State
- 3.3 Cloud Monitoring Metric Audit — Methodology
- 3.4 Identified Noisy/Unused Metric Categories
- 3.5 Exclusion Mechanisms — Console UI vs. Scrape-Time Blocking
- 3.6 Prometheus Scrape Interval Tuning
- 3.7 Session Recap (As Given Live)
- 3.8 Security Scanning Tools — ScoutSuite and Prowler
- 3.9 Compute Engine Cost Preview (Next Session’s Scope)
- 3.10 Inventory-as-Audit-Trail — Reinforced
- 3.11 Program Logistics & Outstanding Items
Key Concepts Table
Architecture & Workflow Analysis
Commands & Configurations
Tools & Technologies
Real-World Production Usage
Interview Preparation (Beginner / Intermediate / Advanced)
Exam & Certification Notes
Cheat Sheet
Gaps & Assumptions

3. Detailed Structured Notes

3.1 GCP/Cloud Credit Sourcing Strategy (General Knowledge)

Presented as broadly useful knowledge for any startup, not specific to FintechPlatform:

Standard credit amounts offered by major cloud providers:

Provider	Credit Amount	Validity
GCP	~$300,000 (≈ ₹3.5 Cr)	3 years
AWS	~$100,000	2 years
Azure	~$200,000	2 years

Common Indian startup pattern (described, not necessarily endorsed as best practice): many startups sequentially rotate across cloud providers purely to keep operating on free credits — e.g., 3 years on GCP, then migrate to AWS for 2 years once GCP credits are exhausted (and any extension has been negotiated and also exhausted), then to Azure for another 2 years, and so on — resulting in many startups effectively paying nothing for cloud infrastructure for extended periods. Some startups instead prioritize reliability and split infrastructure across two cloud providers simultaneously for high availability, rather than purely chasing credits.

How to actually obtain/extend credits: credits are typically accessed through a regional cloud partner/vendor, not by contacting the cloud provider directly. Process demonstrated live: search for “[cloud provider] partner” plus your region (e.g., searching for a GCP partner in Dubai) via the cloud provider’s own official partner-directory page, which lists certified regional partners who can facilitate credit applications/extensions. A specific partner example was found and referenced live (a Dubai-based partner) as a live demonstration of this search process.

Client-specific application: FintechPlatform (Dubai/UAE-based) had already used its full $300K GCP credit allocation and was seeking an extension to at least a year further. Because credits are region-specific, Indian cloud vendors/partners were unable to assist — the team had to specifically locate and engage a UAE-based GCP partner to pursue the extension, and would need to repeat this same region-specific partner search separately for AWS and Azure credit inquiries if pursued.

3.2 Client Observability Stack — Current State

The client runs two parallel observability systems: a self-hosted Prometheus + Grafana stack, and GCP’s native Cloud Monitoring.
Reason for the duplication: some developers on the client’s team are more comfortable working with GCP’s native Cloud Monitoring dashboards specifically, and the client explicitly asked to retain both rather than consolidate onto one.
Cloud Monitoring’s cost over the prior ~2 months: approximately $1,000.
Dashboards exist in Cloud Monitoring for a wide range of resource categories: Compute Engine instances, storage, load balancers, GKE clusters, BigQuery, and more.

3.3 Cloud Monitoring Metric Audit — Methodology

Starting point: GCP’s Metrics Management console view, which lists every metric currently being ingested/stored, along with its sample volume and last-read timestamp.

Scale of the problem: approximately 1,800 distinct metrics were being stored.

Audit process, demonstrated live:

Sort/filter metrics by sample volume (highest generators first) to prioritize investigation.
For each high-volume metric, check its “last read” date — the most recent point at which any dashboard or query actually pulled data from that metric.
Extend the analysis window deliberately (the instructor widened it from a default view out to a custom range starting January 1st through the present — roughly a 45-day window at time of the call, later cross-checked against 90-day and even 180-day windows for some metrics) to avoid false positives from a too-narrow default view.
Interpretation rule: if a metric shows zero reads across an extended window (30, 90, even 180 days in some cases examined) despite generating a large sample volume (hundreds of thousands to hundreds of millions of samples), it’s a strong candidate for exclusion — it’s being paid for but never actually consumed by any dashboard.
Clarification on what “used” means (raised directly by a participant): a metric counts as “used” when a dashboard or query actually pulls/reads its stored data — visible in the console as a specific count of samples read out of total samples generated (e.g., “31K read out of 317 million generated” was shown live as an example of a metric that is actively used, with a recent last-read date, contrasted against metrics showing zero reads).

3.4 Identified Noisy/Unused Metric Categories

Categories confirmed, during the live audit, as high-volume but effectively unused:

Category	Notes
`kube-system`-prefixed metrics	Broad category of Kubernetes system-level metrics; large sample volume, not pulled by any dashboard
Pod toleration / status-reason metrics	Specifically flagged; one example generated ~300 million samples over a 180-day window with zero dashboard reads
”Feature enabled” pod metrics	Another specific low-value, high-volume category identified
*All `container_network_`-prefixed metrics**	A broad family of metrics, nearly all unused except a small number of exceptions; handled as a single bulk exclusion category rather than item-by-item
Histogram-derived metrics (`_bucket`, `_sum`, `_count` suffixes)	Confirmed not pulled by any Grafana dashboard in this environment
Istio (`istio`/`sto`-prefixed) metrics	Client explicitly confirmed they are not actively using Istio; these metrics were being scraped and stored anyway, purely as observability noise

Process for building the exclusion list: categories were documented in a shared spreadsheet/doc (linked in the session chat), reviewed, and — given the client’s clear “not currently using this” confirmations for several categories (e.g., Istio) — queued for formal exclusion pending final client sign-off on the full list.

Planned automation: given the scale (1,800+ metrics), the instructor stated an intention to build a regex-based exclusion script rather than manually excluding each metric one at a time through the console UI — the script would check each metric’s last-read date and sample volume against defined thresholds (e.g., only flag metrics generating more than 100,000 samples with zero recent reads) and apply exclusions in bulk. This script was not yet built at the time of the call — described as a planned next step, to be shared once completed.

3.5 Exclusion Mechanisms — Console UI vs. Scrape-Time Blocking

Two genuinely different levels at which “excluding” a metric can happen — an important distinction raised directly by a participant:

Console-level exclusion (“Exclude Metric” → “Create Rule” in Metrics Management): demonstrated live — right-click a metric in the Metrics Management view, select “Exclude metric,” and create a rule. This stops the metric from being stored/billed going forward, but doesn’t necessarily prevent it from being generated/scraped in the first place by the collection agent.
Scrape-time / ingestion-time blocking (the more fundamentally cost-effective fix): achieved by editing the collector configuration itself — demonstrated live via a PodMonitoring Kubernetes resource (GCP’s Prometheus-compatible scrape configuration mechanism). A metricRelabelings block was added with action: drop, a sourceLabels reference to the metric name, and a regex pattern matching the unwanted metric name prefix (e.g., an Istio-related prefix) — this prevents the matching metrics from ever being ingested/scraped at all, rather than merely hiding already-collected data from storage/billing.
A separate, more fundamental option for VM-level metrics specifically: if observability was never enabled/installed on a given Compute Engine VM in the first place (the observability/monitoring agent is opt-in at VM creation), no metrics from that VM are scraped at all — the most complete form of exclusion, applicable when a given resource genuinely doesn’t need any monitoring.

Practical guidance given: for a large-scale, ongoing cost problem like this (1,800+ metrics), scrape-time/ingestion-time blocking is the more durable and cost-effective fix, since it stops the cost at its source rather than requiring ongoing post-hoc exclusion-rule maintenance in the console.

3.6 Prometheus Scrape Interval Tuning

Default scrape interval found in the Prometheus configuration: 15 seconds — meaning Prometheus queries/scrapes the cluster for fresh metric data every 15 seconds.
Change implemented (for non-production only): increased from 15 seconds to 60 seconds — a 4x reduction in scrape frequency, translating to roughly a 75% cost reduction on the scraping-driven portion of the monitoring cost for non-production environments.
Production explicitly left unchanged: the instructor noted production monitoring granularity is “critical for the business” and was not touched by this optimization — this change was scoped specifically and deliberately to non-production.

3.7 Session Recap (As Given Live)

A participant explicitly requested a consolidated recap partway through the highly demo-heavy session — the instructor’s own summary, preserved here:

Analyzed the full set of ~1,800 metrics currently stored in the account.
Identified specific categories of metrics that are high-volume but not pulled by any dashboard (Prometheus/Grafana or Cloud Monitoring-native) in an extended lookback window.
Began building an exclusion list of these categories, to be finalized into an automated regex-based exclusion script.
Increased the Prometheus scrape interval from 15s to 60s for non-production.
Added a scrape-time exclusion (metricRelabelings/drop action) for Istio-related metrics in the PodMonitoring configuration.
Not yet done, planned for follow-up: obtaining Grafana access (not yet granted at time of this call) to directly compare Grafana’s own dashboards against Cloud Monitoring’s dashboards, identify genuine duplicates, and remove redundant dashboards across both systems.

3.8 Security Scanning Tools — ScoutSuite and Prowler

Framed explicitly as a lightweight, overview-level security scan — not a full Phase 2 security audit (which comes later in the engagement) — intended to give the client a general sense of security gaps.

ScoutSuite:

Installation demonstrated: set up a Python virtual environment, pip install ScoutSuite, then ran a scan command against the GCP environment (a different command is used for AWS).
Output: an auto-generated HTML report, categorized by resource type (Compute, Database, Management, Functions, Networking, etc.), with a filterable dashboard view.
Example findings surfaced live: default SSH keys enabled (flagged as a gap), default firewall rules still in use (flagged red/critical), default network not removed (flagged red/critical), serial ports enabled (flagged). Each finding includes an explanation of why it’s a risk and a remediation reference.
Philosophy of the tool, explained directly: ScoutSuite evaluates the environment from an attacker’s perspective — reasoning through what an attacker would attempt (e.g., exploiting a default SSH port) — rather than being organized primarily around a specific compliance framework. It is explicitly not compliance-framework-heavy.
Known limitation: ScoutSuite does not scan two or three specific GCP resource types (the exact resource type name wasn’t recalled live by the instructor — flagged in Gaps & Assumptions).
Client-facing reporting tip: rather than sharing the raw HTML report directly with the client, the instructor described using a custom Python script to convert the HTML report into a more presentable Excel/CSV format for client review.
Provenance/trust: confirmed as an open-source tool maintained by the NCC Group, a well-regarded security organization, and noted as widely used across many organizations for cloud security auditing — offered as reassurance to a participant asking whether it’s safe/appropriate to use in their own current organization.

Prowler:

Positioned as the compliance-heavy alternative to ScoutSuite — supports filtering findings by specific compliance frameworks (ISO, HIPAA, and others were named as examples).
Installation attempted live (Python-based, similar pattern to ScoutSuite) but hit an unresolved error during the session — the instructor did not debug it live within this call.
A separate, web/SaaS-hosted version of Prowler was also shown, described by the instructor as new to them (not previously used) — offering an “attacker path” style dashboard (e.g., flagging publicly-exposed S3 buckets as a critical finding) alongside dedicated compliance-framework filtering views.
Both tools’ installation links were shared in the session chat.

EKS-specific security scanning (kube-hunter, kubescape, kube-bench) was referenced again as already covered in a previous session’s dedicated recorded demo, rather than being repeated live in this call.

3.9 Compute Engine Cost Preview (Next Session’s Scope)

Presented as a preview/teaser for the following session’s focus, not executed live in this call:

Current Compute Engine monthly cost: approximately $3,000.
Instance inventory: 19 total instances — 2 × N2 high-memory instances, 6 × T2D-standard instances (one size tier), and 11 × T2D-standard instances (a smaller size tier) — plus a cluster with additional nodes. 8 of the 19 instances belong to a managed instance group. 23 attached disks and 44 snapshots were also noted.
Identified savings opportunity: a Committed Use Discount (CUD) — GCP’s equivalent of AWS Savings Plans — was identified as available, estimated to save approximately $800/month if adopted.
A 100% cost jump from January was flagged as visible in the billing trend — to be explained to the client using the inventory-backed method described in Section 3.10.
Next session’s planned scope: Compute Engine, GKE, and Redis (Memorystore) cost optimization together in one session, followed eventually by Cloud SQL and remaining networking/artifact-registry items to close out Phase 1 (cost optimization) of the overall three-phase engagement.

Full list of cost-driver categories across the account (as summarized live, in rough order of cost significance): Compute Engine, Cloud Monitoring, Cloud Memorystore (Redis), Cloud SQL, Cloud Logging, networking, Artifact Registry, and Cloud Storage.

3.10 Inventory-as-Audit-Trail — Reinforced

Directly building on the same principle established in an earlier session (building a resource inventory as a defensible “before” snapshot): the instructor explained that when the client questions a cost spike (e.g., the 100% jump from January noted above), the team can use the previously-built, timestamped inventory to show exactly what resources existed at that point in time, and precisely what was added since (e.g., 4 additional compute instances, 2 additional node groups, and 1 additional Cloud SQL instance) — giving a concrete, evidence-based explanation for the cost increase rather than a vague or defensive response.

4. Key Concepts Table

Concept	Explanation	Example	Why It Matters
Metric “last read” audit	Checking whether a stored metric has actually been queried/read by any dashboard within an extended lookback window	A metric generating 300M samples over 180 days with zero reads	Directly identifies metrics being paid for but never consumed — the core lever for Cloud Monitoring cost optimization
Console-level exclusion vs. scrape-time blocking	Excluding a metric from storage/billing after collection (console) vs. preventing it from ever being scraped/ingested (collector config)	`PodMonitoring` `metricRelabelings` with `action: drop` vs. “Exclude Metric” in the console UI	Scrape-time blocking is the more fundamentally cost-effective fix, stopping cost at the source rather than requiring ongoing exclusion-rule maintenance
Duplicate observability stack cost	Running two parallel monitoring systems (e.g., Prometheus/Grafana + Cloud Monitoring) doubles storage/ingestion cost for largely overlapping data	Client kept both because some developers prefer native GCP dashboards	A direct, ongoing cost multiplier that’s easy to overlook once both systems are established and “just work”
Prometheus scrape interval as a cost lever	The frequency at which Prometheus queries the cluster for metrics directly drives ingestion volume and cost	15s → 60s for non-prod = ~75% reduction in scrape-driven cost	A simple, low-risk, high-leverage optimization, especially for non-production environments where granularity matters less
Regional cloud partner requirement for credits	Cloud provider credits are typically accessed/extended through a certified regional partner, not the cloud provider directly, and are region-specific	A Dubai-based client required a UAE-registered GCP partner, not an Indian one	A practical, non-obvious constraint worth knowing when advising any non-domestic client on cost/credit strategy
Attacker-perspective vs. compliance-framework-driven security scanning	ScoutSuite evaluates from an attacker’s likely approach; Prowler is organized around specific compliance framework requirements	ScoutSuite flags a default SSH port as exploitable; Prowler filters findings by HIPAA/ISO	Different tools suit different audit goals — general security hygiene vs. formal compliance verification
Inventory as cost-spike evidence	A timestamped resource snapshot lets a team explain a billing increase with concrete evidence rather than a vague response	Showing the client exactly which 4 instances, 2 node groups, and 1 SQL instance were added since a baseline snapshot	Reinforces a recurring principle across this engagement — inventory work pays for itself repeatedly, not just at project kickoff

5. Architecture & Workflow Analysis

5.1 Cloud Monitoring Metric Audit Workflow

1. Open GCP Metrics Management console
        |
        v
2. Sort metrics by sample volume (highest first)
        |
        v
3. For each high-volume metric, check "last read" date
        |
        v
4. Extend lookback window (30 -> 90 -> 180 days) to confirm
        |
        v
    -----------------------------------
    |                                  |
Zero reads in extended            Recently read, actively
window despite high volume        used by a dashboard
    |                                  |
    v                                  v
CANDIDATE FOR EXCLUSION            KEEP (in use)
    |
    v
5. Categorize (kube-system, container_network_*,
   histogram suffixes, Istio-related, etc.)
    |
    v
6. Client review/sign-off on exclusion list
    |
    v
7. Apply exclusion:
   - Console "Exclude Metric" (post-collection), OR
   - Scrape-time block via PodMonitoring metricRelabelings
     (prevents ingestion entirely -- preferred for durability)
    |
    v
8. [Planned] Automate via regex-based bulk exclusion script

5.2 Two Levels of Metric Exclusion

Metric Generated by Application/Infrastructure
        |
        v
   Scraped by Prometheus/Collector (every N seconds)
        |
   -----------------------------------------------
   |                                               |
SCRAPE-TIME BLOCK                          No scrape-time block
(metricRelabelings, action: drop)                  |
   |                                               v
   v                                        Metric ingested
Metric NEVER ingested                       into storage
(most cost-effective;                              |
 stops cost at the source)                  -----------------------------
                                             |                           |
                                       CONSOLE EXCLUSION            No exclusion
                                       ("Exclude Metric" rule)      (fully stored,
                                             |                       fully billed)
                                             v
                                       Stops future billing,
                                       but ingestion/scrape
                                       cost already incurred
                                       up to this point

5.3 Duplicate Observability Stack (Current State)

Client Infrastructure (GKE + Compute Engine)
        |
   -----------------------------------------
   |                                        |
Prometheus + Grafana                  GCP Cloud Monitoring
(self-hosted)                         (native)
   |                                        |
Scrapes cluster metrics                Also scrapes/ingests
(15s interval, now 60s                 many of the SAME metrics
 for non-prod)                                |
   |                                        |
Grafana dashboards                    Cloud Monitoring dashboards
   |                                        |
   -----------------------------------------
                    |
                    v
        SIGNIFICANT DASHBOARD DUPLICATION
        (not yet resolved -- pending Grafana
         access to compare and de-duplicate)

5.4 Cloud Credit Sourcing Path (General Pattern)

Startup/Organization needs cloud credits
        |
        v
Identify target cloud provider (GCP/AWS/Azure)
        |
        v
Search: "[cloud provider] partner directory"
        |
        v
Filter/search by REGION (credits are region-specific)
        |
        v
Identify certified regional partner
        |
        v
Contact partner -> initiate credit application/extension
        |
        v
[Typical pattern for India-based startups]:
Use credits for ~2-3 years on Provider A
-> exhaust + extend as far as possible
-> migrate to Provider B for its credit cycle
-> repeat

6. Commands & Configurations

Command / Config	Purpose	Explanation
GCP Console → Monitoring → Metrics Management	Central view of all stored metrics, their sample volume, and last-read date	The primary tool used throughout the metric audit
Right-click metric → “Exclude metric” → “Create rule”	Console-level exclusion of a specific metric from future storage/billing	Demonstrated live for several individually-identified noisy metrics
`PodMonitoring` resource — add `metricRelabelings` block with `action: drop`, `sourceLabels: [__name__]` (or similar), and a `regex` pattern	Scrape-time exclusion — prevents matching metrics from ever being ingested	Used specifically to block Istio-related metrics; the more durable, cost-effective form of exclusion versus console-level exclusion alone
Prometheus scrape config — `scrape_interval` field, changed from `15s` to `60s`	Controls how frequently Prometheus queries the cluster for fresh metric data	Applied specifically to non-production; production left at the original interval
`pip install scoutsuite` (within a Python virtual environment), then run the ScoutSuite scan command against GCP	Install and run ScoutSuite for a full-environment security scan	Generates an HTML report categorized by resource type; command differs for AWS vs. GCP
Custom Python script (referenced, not detailed) to convert ScoutSuite’s HTML report to Excel/CSV	Produce a more presentable, client-friendly report format	A practical reporting technique for sharing findings with non-technical stakeholders
Prowler installation (Python-based, similar pattern to ScoutSuite)	Install and run Prowler for compliance-framework-oriented security scanning	Installation was attempted live but hit an unresolved error in this session

7. Tools & Technologies

GCP Cloud Monitoring (Metrics Management)

Purpose: GCP’s native metrics storage, visualization, and management interface.
When to use it: For auditing metric usage/cost, and for GCP-native dashboarding when teams prefer it over a self-hosted alternative.
Cost note: every stored metric — whether actively used or not — contributes to ongoing storage/ingestion cost, making periodic usage audits a meaningful cost-optimization practice.

Prometheus + Grafana

Purpose: Self-hosted metrics collection (Prometheus) and visualization (Grafana) stack.
When to use it: As an open-source alternative or complement to cloud-native monitoring; commonly run alongside cloud-native tools in real organizations (as seen in this exact client), sometimes leading to genuine duplication that needs periodic review.
Cost lever: the scrape interval directly controls ingestion volume/cost — tuning it per-environment (looser for non-prod, tighter for prod) is a low-risk optimization.

ScoutSuite

Purpose: Open-source, multi-cloud security auditing tool, maintained by the NCC Group.
When to use it: For a general, attacker-perspective security posture scan across a cloud environment — well suited to an initial/lightweight security overview rather than a formal compliance audit.
Advantages: Free, trusted (NCC Group-maintained), widely adopted; produces a clear, categorized HTML report with remediation guidance.
Limitations: Not organized primarily around compliance frameworks; doesn’t scan every GCP resource type (a small number of exceptions).

Prowler

Purpose: Open-source, compliance-framework-oriented cloud security scanning tool, also available as a web/SaaS-hosted version.
When to use it: When findings need to be filtered/organized by a specific compliance framework (ISO, HIPAA, etc.) rather than a general attacker-perspective view.
Note: Installation hit an unresolved error live in this session — worth testing independently before relying on it for a real engagement.

8. Real-World Production Usage

The metric-usage-audit technique demonstrated here is directly reusable for any organization running a non-trivial observability stack — checking “last read” against sample volume is a simple, mechanical process that reliably surfaces genuine waste, and is exactly the kind of low-risk, high-confidence cost optimization that should be standard periodic hygiene, not a one-time project.
The distinction between post-collection exclusion and scrape-time blocking reflects a mature understanding of where cost actually accrues in an observability pipeline — many engineers stop at “hide it from the dashboard,” missing that true savings requires stopping ingestion at the source; this session models the more complete, durable fix.
Running duplicate observability stacks because “some developers prefer X” is a genuinely common real-world pattern, and this session shows a pragmatic response to it: rather than forcing consolidation immediately (a larger, more disruptive change requiring broader buy-in), the team is first eliminating clear waste within the existing dual-stack reality, with dashboard de-duplication as a planned follow-up once more information (Grafana access) is available.
The region-specific cloud-partner requirement for credits is a genuinely practical, easy-to-overlook piece of knowledge for anyone advising a non-domestic client (or working at an agency/consultancy serving international clients) — assuming your own region’s vendor relationships transfer automatically to a client in a different region is a real, avoidable mistake.
Using an inventory snapshot to substantiate a cost-spike explanation to a client is a mature, trust-building consulting practice — reinforced here as a recurring theme across this entire engagement, illustrating that inventory work isn’t a one-time kickoff task but an ongoing asset used repeatedly throughout a cost-optimization engagement.
The ScoutSuite-vs-Prowler distinction (attacker-perspective vs. compliance-framework-driven) reflects a real, practical choice security engineers make based on the actual goal of a given scan — a general security hygiene check versus a formal audit against a specific named compliance standard require genuinely different tooling approaches, not just “run a scanner.”

9. Interview Preparation

Beginner Questions

Q1: What does it mean for a metric to be “unused” in a cost-optimization context, and how would you check for this? A: A metric is considered unused if it’s being generated and stored (incurring cost) but never actually queried or displayed by any dashboard. This is checked via the monitoring platform’s “last read” timestamp for each metric, cross-referenced against its sample volume (how much data it’s generating) — a metric with a large sample volume but no reads over an extended window (weeks to months) is a strong candidate for removal.

Q2: Why might an organization end up running two overlapping observability stacks (e.g., Prometheus/Grafana and a cloud-native monitoring tool) at the same time? A: Often due to team preference or historical accumulation — different team members may be more comfortable with different tools, or a cloud-native tool may have been adopted later without fully retiring the original self-hosted stack. This results in genuine duplication of both cost and effort, since largely the same underlying metrics are being collected and stored by two separate systems.

Q3: What’s a quick, low-risk way to reduce Prometheus-related cost without removing any metrics? A: Increase the scrape interval — the frequency at which Prometheus queries targets for fresh data. A longer interval (e.g., 60 seconds instead of 15 seconds) directly reduces the volume of data ingested and stored, at the cost of slightly less granular/real-time data — usually an acceptable trade-off for non-production environments where near-real-time monitoring isn’t critical.

Intermediate Questions

Q4: Explain the difference between excluding a metric via a monitoring console’s UI versus blocking it at scrape time, and why the distinction matters for cost. A: Console-level exclusion (e.g., a GCP Metrics Management “exclude metric” rule) prevents a metric from being billed/stored going forward, but the metric may still be actively scraped/collected by the underlying agent — meaning the exclusion is essentially a downstream filter. Scrape-time blocking (e.g., a drop action in a Prometheus/PodMonitoring relabeling configuration) prevents the metric from ever being collected in the first place. The distinction matters because scrape-time blocking is more fundamentally cost-effective and durable — it stops the cost at its actual source rather than relying on an exclusion rule to catch it after collection, and it also reduces unnecessary load on the collection/scraping pipeline itself.

Q5: A client wants to know why their cloud bill increased significantly compared to last month. How would you approach answering this credibly? A: Rather than offering a general explanation, use a concrete, evidence-based approach: compare a timestamped resource inventory from before the increase against the current state, identifying exactly which resources were added, removed, or changed in the interim. This lets you give the client a precise, itemized explanation (e.g., “4 additional compute instances, 2 additional node groups, and 1 additional database instance were provisioned since [date]”) rather than a vague or speculative answer — building trust and demonstrating that the cost is well-understood and attributable to specific, identifiable changes.

Q6: Why might a security scanning tool like ScoutSuite be described as evaluating an environment “from an attacker’s perspective” rather than being purely compliance-driven, and when would you choose it over a compliance-framework-oriented tool like Prowler? A: ScoutSuite’s findings are organized around what a realistic attacker would actually attempt and exploit (e.g., default open ports, overly permissive default configurations) rather than being structured primarily around satisfying a specific named compliance standard’s checklist. This makes it well-suited for a general security hygiene review or an initial security posture assessment. You’d choose a compliance-framework-oriented tool like Prowler instead when the actual deliverable needs to demonstrate adherence to a specific named standard (HIPAA, ISO, PCI-DSS, etc.) — for example, when a client or regulator specifically requires evidence of compliance against a named framework, not just a general security assessment.

Advanced Questions

Q7: Design a systematic process for auditing and reducing an organization’s Cloud Monitoring/observability costs, using the approach demonstrated in this session as a starting point. A: Start by inventorying every currently-stored metric along with its sample volume and last-read timestamp, using an appropriately extended lookback window (not just the default short window, since infrequently-but-genuinely-used metrics could otherwise be misclassified as unused). Categorize high-volume, zero-read metrics into logical groups (by prefix, resource type, or subsystem) rather than evaluating thousands of individual metrics one at a time. For each category, confirm with relevant stakeholders (application teams, the client) whether the underlying feature/tool (e.g., Istio) is genuinely unused before excluding its metrics, to avoid accidentally removing something that has value not currently reflected in dashboard usage. Apply exclusions at the most durable level available — scrape-time/ingestion blocking where possible, rather than only console-level post-collection exclusion — and build tooling (e.g., a regex-based bulk exclusion script) to make this a repeatable, low-effort process rather than a one-time manual cleanup, since new low-value metrics will continue to accumulate over time as the environment evolves. Separately, tune collection frequency (scrape intervals) per environment tier, applying looser intervals to non-production where real-time granularity isn’t business-critical.

Q8: A client insists on continuing to run both a self-hosted Prometheus/Grafana stack and a cloud-native monitoring tool in parallel, citing team preference. How would you approach cost optimization given this constraint, rather than pushing for consolidation? A: Accept the constraint as a genuine business/organizational reality rather than treating it as a problem to immediately solve by forcing consolidation, which would require broader buy-in and carries its own disruption risk. Within that constraint, focus on eliminating clear, low-risk waste first — unused metrics, overly frequent scrape intervals for non-critical environments, and metrics being collected for features the client isn’t actually using (e.g., unused service mesh components). As a distinct follow-up phase, obtain visibility into both systems’ actual dashboards (not just their underlying metrics) to identify genuinely duplicate dashboards — content that exists in both Grafana and the cloud-native tool — since removing confirmed duplicates reduces cost without removing anything any team member is actually relying on, sidestepping the more contentious question of full stack consolidation while still capturing meaningful savings.

Q9: How would you decide whether a given security finding from a general-purpose scanning tool (like ScoutSuite) warrants immediate remediation versus being logged for a later, more formal audit phase? A: Assess based on the finding’s actual exploitability and blast radius in the specific environment — e.g., default firewall rules or default SSH access being left enabled on internet-facing resources represents an immediately exploitable, high-severity gap warranting prompt remediation, since it doesn’t require any special access or insider knowledge to exploit. Lower-severity findings, or findings on non-production/isolated resources with limited blast radius, can reasonably be logged and addressed as part of a more comprehensive, later compliance-focused audit phase (as this session explicitly scoped this particular scan as a lightweight overview ahead of a fuller Phase 2 security engagement) — the key judgment is distinguishing “this is actively exploitable right now with minimal effort” from “this is a gap that should be closed as part of ongoing security maturity work,” and prioritizing the former for immediate action regardless of which broader engagement phase is officially underway.

10. Exam & Certification Notes

(Relevant to GCP Associate Cloud Engineer / Professional Cloud Architect certifications, and general FinOps/observability-adjacent content.)

GCP Cloud Monitoring metric storage and billing model: Know that Cloud Monitoring bills based on the volume of metric data ingested and stored, independent of whether that data is ever queried — directly relevant to why unused-metric audits are a genuine cost-optimization technique, not just a housekeeping exercise.
Committed Use Discounts (CUDs): GCP’s mechanism for committing to sustained resource usage in exchange for a discount — the direct GCP analog to AWS Savings Plans/Reserved Instances, referenced again in this session regarding Compute Engine.
Prometheus scrape interval (scrape_interval): A standard, frequently relevant Prometheus configuration parameter — know that it directly trades off data granularity against ingestion volume/cost, and that different environments (prod vs. non-prod) commonly warrant different values.
PodMonitoring (GCP Managed Service for Prometheus): GCP’s managed, Prometheus-compatible scrape configuration mechanism for GKE — relevant to understanding how metric relabeling/dropping is configured in a GKE-native context, as opposed to a self-hosted Prometheus scrape_config.
Security scanning tool categories: Understand the general distinction between attacker-perspective/general-hygiene security scanners and compliance-framework-oriented auditing tools — a conceptual distinction that may appear on security-adjacent certification content even without naming these specific tools.

11. Cheat Sheet

Cloud Credit Amounts (approximate, standard offers):

Provider	Amount	Validity
GCP	$300K	3 years
AWS	$100K	2 years
Azure	$200K	2 years

Metric Audit Rule: High sample volume + zero reads over an extended window (30-180 days) = exclusion candidate.

Two Levels of Exclusion (know the difference):

Console-level (“Exclude Metric”) = stops billing going forward, but scraping may still occur
Scrape-time (metricRelabelings, action: drop) = prevents ingestion entirely — more durable/effective

Prometheus Scrape Interval: 15s → 60s for non-prod = ~75% reduction in scrape-driven cost. Leave prod untouched unless explicitly justified.

Noisy Metric Categories to Always Check:

kube-system-prefixed
container_network_*
Histogram suffixes (_bucket, _sum, _count)
Service mesh metrics (e.g., Istio) if the mesh isn’t actually in active use
Pod status/toleration/feature-flag metrics

Security Scanning Tool Choice:

General hygiene / attacker-perspective → ScoutSuite
Compliance-framework-specific (HIPAA, ISO, etc.) → Prowler

Inventory-as-Evidence Rule: Always keep a timestamped resource inventory — it’s the fastest, most credible way to explain a cost spike to a client.

Regional Credit Rule: Cloud credits are region-specific — a client outside your home region requires a cloud partner registered in their region, not yours.

12. Gaps & Assumptions

The exact GCP resource types ScoutSuite cannot scan were referenced live but the instructor could not recall the specific name(s) in the moment — flagged as an unresolved detail; verify directly against ScoutSuite’s own documentation for the current, complete list of supported/unsupported GCP resource types.
Prowler’s installation error was not resolved within this session — the tool was shown partially (installation attempt, plus a separate web/SaaS-hosted version demo) but not fully validated working in this specific environment. Treat Prowler’s live-demo status in this session as incomplete, not as confirmation it’s fully operational in this client’s environment.
The regex-based bulk metric-exclusion script was described as planned but not yet built at the time of this call — this document describes its intended purpose/logic as stated, not a finished, verified artifact.
The exact resolution to the bastion→production SSH mystery (from earlier war-room sessions) is not included in this transcript — the instructor confirmed it was solved by participants and posted in Discord, but the actual technical solution/RCA content itself is outside this transcript’s scope. Refer to the referenced Discord thread/RCA for the actual resolution.
Specific instance counts/types for Compute Engine (2x N2 high-memory, 6x and 11x T2D-standard at two different size tiers) were given somewhat loosely in the live narration — presented here as stated, but worth double-checking against the actual inventory sheet/spreadsheet referenced in the session for precise figures before using them externally.
The “100% cost jump from January” figure was mentioned as an observed trend in the billing dashboard but the underlying full explanation/breakdown (beyond the general “4 instances + 2 node groups + 1 SQL instance” example given) wasn’t fully detailed in this transcript — treat this as a preview/flagged item for the next session’s deeper Compute Engine/GKE optimization work, not a fully closed-out finding.
This document consolidates a demo-heavy session with substantial live UI navigation and some genuine tool errors (e.g., the Prowler installation issue) — content has been reorganized topically for clarity rather than presented in strict chronological order, consistent with the approach used for prior packages in this series.