Masterclass: Designing Scalable, Low-Latency, Production-Grade AI Agent Systems
Structured educational resource covering masterclass: designing scalable, low-latency, production-grade ai agent systems.
2. Table of Contents
- Executive Summary
- Table of Contents
- Detailed Structured Notes
- 3.1 What Is an “Agent”? (Working Definition)
- 3.2 Why Naive Agents Fail — The Four Core Problems
- 3.3 User Latency Tolerance Thresholds
- 3.4 Worked Example: The Naive Grafana DevOps Chatbot
- 3.5 Security Considerations for Agentic Systems
- 3.6 The Production-Grade Solution — Three Pillars
- 3.7 Pillar 1: Orchestrator/Brain — From ReAct Loop to DAG
- 3.8 Pillar 2: Three-Tier Memory Hierarchy
- 3.9 Pillar 3: Small Language Model (SLM) Intent Classification & Routing
- 3.10 Full Worked Example: E-Commerce Customer Support Agent
- 3.11 Advanced Optimization: Speculative Execution
- 3.12 Advanced Optimization: Reducing Time-to-First-Token (TTFT)
- 3.13 Closing Q&A: DevOps → MLOps Career Path (Lighter-Weight Capture)
- Key Concepts Table
- Architecture & Workflow Analysis
- Commands & Configurations
- Tools & Technologies
- Real-World Production Usage
- Interview Preparation
- Exam & Certification Notes
- Cheat Sheet
- Gaps & Assumptions
3. Detailed Structured Notes
3.1 What Is an “Agent”? (Working Definition)
The instructor deliberately avoids a rigid, universal definition, since “agent” is used loosely across the industry. Working definition given:
- At its simplest, an agent can be just a basic API call to an LLM that returns a contextual answer.
- At its more complex end, an agent is a system where: a task is given to an LLM → the LLM constructs its own prompt → that prompt is sent to another LLM (or the same one) → that LLM decides what actions to take (API calls, DB queries, etc.) → results are aggregated → a further LLM call summarizes/synthesizes the aggregated results into a final response.
- Instructor’s framing for the session: don’t over-anchor on a precise definition — for this session’s purposes, an agent can be thought of as, at minimum, “a basic LLM query” that the rest of the session will build structure and optimization around.
3.2 Why Naive Agents Fail — The Four Core Problems
The instructor identifies four core reasons naive/tutorial-style agentic frameworks fail once they hit real production traffic:
- The Latency Trap — Handing an entire task to a single LLM call chain (think → act → observe → think again) routinely takes 10–20+ seconds. No production user will tolerate this.
- State/Memory Stored in RAM — Naive frameworks store conversation history and session state in a local in-process variable (RAM). This works fine for a single instance but breaks horizontal scaling: with a load balancer routing requests across multiple pods, a user’s second request may land on a different pod that has no memory of the first request, silently losing state and breaking the reasoning chain.
- Tool Cascading — When an agent’s workflow requires calling API A, then (based on A’s result) API B, then API C, the total latency becomes the sum of every step (
time(A) + time(B) + time(C) + LLM reasoning time between each), and any failure triggers retries that further increase cost and latency unpredictably. This creates unpredictability across cost, latency, and behavior simultaneously. - Observability Gap — When an agent fails, there’s typically no way to tell where it failed: was it an LLM hallucination, a tool/API failure, a cache-layer outage, or something else? Without structured logging, this makes production debugging extremely difficult.
Solution sketch given for problem #4 (observability) — since this audience already has DevOps observability background, the instructor gives only a brief solution outline rather than a deep dive:
- Structured logging at the prompt level and the token level (e.g., logging number of tokens consumed, number of tools called).
- Per-step latency logging for every stage of the pipeline.
- Distributed tracing — assigning a trace/request ID to every request and every LLM call within it.
- Metrics tracking, specifically P50 and P95 latency: P50 = the latency experienced by the median (50th percentile) of users; P95 = the latency experienced by the 95th percentile of users (i.e., a near-worst-case view, useful for catching tail-latency problems that averages hide).
3.3 User Latency Tolerance Thresholds
A concrete, memorable latency budget was given, based on general user-behavior research, to justify why the above problems matter:
| Response Delay | User Perception |
|---|---|
| < 100 ms | Feels instant |
| < 1 second | Feels smooth; user doesn’t notice |
| 2–3 seconds | Delay becomes noticeable |
| > 5 seconds | User becomes frustrated |
| > 10 seconds | User abandons the tool/product and does not return |
Implication: any production agentic system that routinely takes the naive-design’s 10–20 seconds per response will lose users and cannot be adopted by serious production companies — this threshold table is the quantitative justification for everything that follows in the redesign.
3.4 Worked Example: The Naive Grafana DevOps Chatbot
To make the abstract problems concrete, the instructor walks through how most engineers would first design an AI chatbot layered over Grafana to help debug production errors (a realistic DevOps use case):
- User asks the chatbot: “Why is this error rate so high?”
- This user prompt is combined with a system prompt (e.g., “You are a DevOps agent. Your job is to read the code and find the error in the system and answer the user.”).
- The combined prompt is sent to an LLM.
- The LLM decides it needs to read the codebase → makes a separate API call (e.g., a GitLab call) to fetch the relevant file.
- Once the file is found, the entire file plus the original prompt is sent back into the LLM.
- The LLM analyzes the code and error, and returns a diagnosis.
- A further LLM-driven tool call may then attempt to fix the code file.
Why this is the “worst” design, in the instructor’s words: every one of these steps is sequential and blocking — nothing can proceed until the previous LLM/API call finishes — meaning total latency stacks up linearly across every step, easily exceeding the 10–20 second abandonment threshold from Section 3.3. This is described as “what most people design the first time,” including the instructor himself, absent deliberate system-design thinking.
Trainee discussion point (why this matters for DevOps): a trainee explicitly asked how this connects to DevOps work. The consensus reached in discussion: DevOps engineers are the ones who get pulled in when an agentic system exhibits high latency, high error rates, or high cost — diagnosing why the system’s internal design produces those symptoms is itself a DevOps-adjacent (or MLOps) responsibility, which is the throughline justifying the whole session.
3.5 Security Considerations for Agentic Systems
Raised as a trainee question (concern: an LLM reading an entire dashboard/logs exposes a lot of internal data) and given a brief, layered answer (explicitly flagged by the instructor as a big topic deserving its own separate session, so treated at overview level here):
- Guardrails — prevent the system from reading/accessing more information than it should.
- Proxy layer — since every call is ultimately a network call, an intermediary proxy sits in front of any call to an LLM or agent. The proxy inspects network parameters/data and can strip or redact anything potentially sensitive/vulnerable before it reaches the model. A trainee connected this to ingress controllers acting in a similar proxy capacity for filtering/sanitizing data before it leaves an organization’s premises/endpoint.
- Trusted/enterprise-managed AI platforms — rather than integrating directly with arbitrary third-party AI APIs, enterprises commonly use a trusted managed offering they already have a data-processing relationship with (example given: AWS Bedrock, since many companies already store their data on AWS and can extend that trust to Bedrock-hosted agents).
- Prompt sharding / isolation between layers — user input is not sent raw into the LLM layer that has control over sensitive data. Instead, there’s a first layer that only constructs the prompt (combining user input with a system prompt) and has no control over actual data access; a second layer’s LLM is the one with actual data/tool access. This separation is explicitly described as a defense against prompt injection — since the layer touching sensitive systems is not the same layer directly exposed to unsanitized user input.
- Restricting inference surface area — many production companies don’t let users freely converse with an LLM; instead they expose a constrained set of pre-selected forms/questions, limiting what can be inferred/extracted.
Instructor’s caveat: even with all these layers, “security lapses can still happen” — this is presented as risk mitigation, not a guarantee.
3.6 The Production-Grade Solution — Three Pillars
Once the four naive-agent problems are established, the instructor introduces the solution as three interlocking pillars (paraphrasing the on-screen summary):
- Orchestrator (“the brain”) — move decision-making away from letting a single large LLM freely decide the entire workflow; instead use a structured graph-based orchestrator (see 3.7).
- Memory hierarchy — replace RAM-based state storage with a tiered system across cache, vector DB, and relational/data-warehouse storage (see 3.8).
- Routing: small model for intent classification, large model reserved for reasoning — use a lightweight small language model (SLM) to quickly classify what the user wants and route the request, reserving the expensive, slower large language model (LLM) purely for final reasoning/synthesis (see 3.9).
3.7 Pillar 1: Orchestrator/Brain — From ReAct Loop to DAG
The problem with the “ReAct” pattern (naive baseline):
- The naive pattern is: the model thinks, takes an action, observes the result, then thinks again — described in the transcript as the “ReAct” pattern (the instructor did not recall/state the acronym’s full expansion during the session).
- This pattern is:
- Sequential — each step depends on completing the previous one.
- Blocking — you cannot proceed until the LLM finishes reasoning about what to do next.
- Unstructured — there’s no fixed shape to the workflow; thought → action → thought can continue indefinitely.
- Hard to parallelize, since steps are inherently sequential.
- Latency-unpredictable, since you don’t know in advance whether “thinking” or “acting” will be the slow step.
- Hard to debug, for the same unstructured reasons.
The solution: structure the workflow as a graph.
- Rather than giving the LLM free rein to decide the entire workflow (the “brain” being the LLM itself), the redesigned system uses an explicit orchestration graph — a control layer that decides the shape of the workflow ahead of time, rather than letting the LLM improvise step-by-step.
- The instructor refers to this repeatedly in the transcript as a “directed cyclic graph.” Structurally and by his own explanation (“it can never circle back to itself… if a graph becomes cyclic, it will be stuck in a loop”), the property being described is actually acyclic, not cyclic — the standard term for this pattern is a Directed Acyclic Graph (DAG), which the instructor also explicitly names (“DAG”) elsewhere in the same explanation. Assumption: this package treats “DAG” as the intended term throughout (Directed Acyclic Graph), and flags the “cyclic” wording as a likely verbal slip inconsistent with the instructor’s own stated reasoning (see Gaps & Assumptions, Section 12).
- In a DAG-based design: distinguishable tasks are nodes (e.g., intent classification, a DB query, a vector search), and dependencies between them are edges. Tasks that do not depend on each other’s output (e.g., a policy-document lookup and a delivery-address lookup) sit on the same level of the graph and can be executed in parallel, which is the core mechanism that eliminates the “tool cascading” latency problem from Section 3.2.
3.8 Pillar 2: Three-Tier Memory Hierarchy
Rather than storing all conversational/session state in RAM (Section 3.2, problem #2), the redesigned system defines three storage tiers, chosen by how fast the data needs to be retrieved:
| Tier | Technology (example) | Target Latency | What’s Stored |
|---|---|---|---|
| Tier 1 — Cache | Redis (referred to as “radius” in the transcript — see Gaps & Assumptions) | Sub-second (under ~1 second) | Anything tied to the current conversation: immediate tool outputs, immediate reasoning results, anything that must be available in milliseconds |
| Tier 2 — Vector DB | Any vector database | ~200 ms – 1 second | Large but searchable content: embeddings for semantic search (e.g., policy documents, past conversation summaries) |
| Tier 3 — Data Warehouse / SQL | e.g., PostgreSQL | No strict sub-second requirement | Structured metadata: request IDs, session IDs, conversation IDs, and similar durable records |
Practical example given (customer conversation history):
- For a short conversation (illustrative example: ~10 turns/messages), the exact/full history is stored directly in the cache tier (Redis).
- For a long conversation (illustrative example: ~100 turns/messages), the entire conversation is summarized, and that summary (not the raw full history) is used as context and stored in the cache tier; the same summary is also stored as an embedding in the vector DB tier, so that similar future queries (from the same or even a different user) can retrieve relevant context via semantic search without needing the LLM to re-reason over a long raw history.
Why this matters: this tiered design is what allows the system to only send the necessary amount of context to the LLM on each call (rather than an ever-growing full conversation transcript), which directly reduces both latency and cost, and is what makes horizontal scaling and pod restarts safe (state no longer lives in a single pod’s RAM).
3.9 Pillar 3: Small Language Model (SLM) Intent Classification & Routing
- Deciding what kind of request this is (e.g., “this needs an API call,” “this needs a DB query,” “this needs a vector/semantic search,” “this needs a cache lookup”) is described as a simple classification task — and simple classification tasks do not require a large, expensive, reasoning-optimized LLM.
- Instead, a small language model (SLM) — which the instructor notes could even be a classical ML classifier, or a lightweight foundation model (example given: Amazon Titan) — is used purely to classify the user’s query into a predefined, finite set of intents (illustrative count given: “7, 8, 10, whatever intents”) and determine the routing (i.e., which backend/tier — API, DB, vector DB, cache — the request should go to).
- Benefit: this removes the large/slow LLM entirely from the initial decision-making step, which is described as saving substantial latency and cost, since intent classification with an SLM can complete in milliseconds rather than the multi-second reasoning time a large LLM would take for the same decision.
- Illustrative pseudocode style given in the session (paraphrased, not verbatim code from the transcript):
intent, route = slm_classifier(model="amazon-titan-or-similar", query=user_query) # intent examples: "policy_lookup", "delivery_address_lookup", "refund_check", etc. # route examples: "vector_db", "sql_db", "cache", "external_api" - The large LLM is reserved exclusively for the final reasoning/synthesis step, after all the necessary data has already been retrieved via the fast, parallel, SLM-routed calls.
3.10 Full Worked Example: E-Commerce Customer Support Agent
This is the session’s main end-to-end design walkthrough, built collaboratively with trainees.
Scenario: An AI customer support agent for a large e-commerce platform.
Example complex user query (deliberately multi-intent):
“I ordered shoes last week. Haven’t received them. I want to change the delivery address also. Can I get a refund if it’s delayed?”
Why this query is hard:
- It is multi-intent — the user wants to (a) change their delivery address and (b) inquire about a refund — not a single, atomic request.
- It requires a DB lookup (to find the order/delivery date for shoes ordered “last week”).
- It requires a policy lookup (to check whether address changes and refunds are allowed under company policy for this scenario).
- It requires reasoning and live conversational state to combine all of the above into a coherent answer.
Naive design (what most people build first) — walked through explicitly as the anti-pattern:
User query
-> combined with system prompt -> sent to LLM (decision-making step, ~5-10 sec)
-> DB call to check delivery address state (sequential)
-> Policy document lookup for refund/address-change rules (sequential)
-> Refund state check via another API/policy document (sequential)
-> All results combined with user query -> sent to reasoning LLM
-> Final response
This design is sequential end-to-end and stacks up significant latency (each step adds its own multi-second delay), matching the anti-pattern already established in Section 3.4.
Optimized (DAG-based) design — built step-by-step with trainee input:
- User query → SLM Intent Classifier. The SLM identifies this as a multi-intent query and splits it into separate, independently routable sub-intents: (a) policy check and (b) delivery address lookup.
- Routing decision, reasoned through with trainees:
- Where is the policy document stored? → It’s large but searchable text → routed to the Vector DB (semantic search).
- Where is the user’s delivery address stored? → It’s simple structured per-user data (user ID, phone number, address) → routed to the relational DB / data warehouse.
- Follow-up scenario discussed: if in a second message the user asks a related follow-up (e.g., “actually I don’t want to change my delivery address, I want to change my pin code”), the delivery-related lookup for that second turn is served from the cache (Redis) tier instead of hitting the DB again, since the relevant context is already “hot” from the current conversation.
- Both lookups run in parallel (not sequentially), using asynchronous execution. Illustrative pseudocode given in the session:
task1 = async(vector_db_search, query=policy_query) task2 = async(db_lookup, query=delivery_address_query) result1, result2 = await asyncio.gather(task1, task2) - Final reasoning step: once both parallel results (policy details + delivery address) are available, they are combined with the original user query and sent to the large LLM for final reasoning, and the synthesized response is returned to the user.
Result: by introducing (a) a memory hierarchy, (b) an orchestrator/routing layer, and (c) parallel execution instead of sequential cascading, the “bad” naive design becomes a substantially faster, more debuggable, and horizontally scalable design — described in-session as turning “our bad memory… [into] a good memory at least.”
3.11 Advanced Optimization: Speculative Execution
Even after the above redesign, one bottleneck remains unoptimized: the final LLM reasoning call itself still takes real time (the instructor notes this step “still going to take time” even after the earlier optimizations).
Speculative execution technique:
- In a normal flow, the system waits until the user finishes typing and submits their query before beginning intent classification and retrieval.
- With speculative execution, the system instead consumes the user’s input live, as they type (technically implemented via WebSockets, per the instructor, though implementation detail was not elaborated further).
- Based on early words/partial sentences alone, the SLM intent classifier begins predicting intent before the user finishes typing.
- Once an intent is predicted, the corresponding vector search / DB lookup begins immediately — so that by the time the user finishes typing and hits submit, the retrieval step has often already completed.
- Net effect: the only remaining wait the user experiences is the final LLM synthesis step (which the instructor notes is still roughly the same ~5 seconds as before) — but because the retrieval work was already done in the background, the perceived end-to-end latency drops significantly. The instructor is explicit that this technique does not actually speed up the LLM reasoning call itself — it only hides/overlaps the retrieval latency with the user’s typing time.
- Amplifying effect at scale: in systems with many concurrent users (e.g., customer support chatbots), many queries are similar or repeatedly reference the same documents — meaning a large fraction of the “predicted” retrievals are often already present in the Redis cache or vector embeddings from prior similar queries, making speculative execution even more effective in aggregate than the single-user theoretical case would suggest.
Illustrative example walked through: for the partial query “What’s the refund policy if my flight is…”, the SLM can recognize the “refund policy” and “flight” signals from the live partial input alone, determine the intent and route (a specific flight-refund-policy document), and begin the vector-embedding semantic search before the user finishes typing out the rest of the sentence (e.g., specific date, airline, and route details).
3.12 Advanced Optimization: Reducing Time-to-First-Token (TTFT)
Beyond speculative execution, three further techniques target reducing Time-to-First-Token (TTFT) — the delay before the user sees the first word of the LLM’s response, which the instructor identifies as the key perceptual latency metric to optimize (as opposed to total end-to-end latency):
-
Caching the system prompt. Every request sent to the final reasoning LLM must be accompanied by a system prompt (e.g., instructions describing what kind of data is being sent — “this is vector DB data,” “this is cache/Redis data,” etc.). Since these system prompts are predictable and repeat by intent/route, they can be pre-cached per intent-and-route combination, so the system doesn’t have to reconstruct/re-send this framing text from scratch on every call. The instructor is explicit that this does not speed up raw LLM inference — it only reduces the time to assemble and dispatch the final prompt.
-
KV (Key-Value) Caching. This targets making the LLM inference itself faster, not just prompt assembly.
- Rooted in how the transformer/attention mechanism underlying LLMs works: every token processed generates Query (Q), Key (K), and Value (V) vectors internally.
- By caching the Key and Value vectors (KV cache) from prior computation, the model can avoid recomputing them from scratch on subsequent tokens/calls, substantially speeding up inference.
- The instructor deliberately did not go into the underlying attention-mechanism mathematics in this session, treating it as a known optimization technique to be aware of rather than derived from first principles here.
-
Properly managed conversation-history storage (extending the strategy from Section 3.8):
- Short conversations (illustrative: ~10 turns): store the exact/full history in the cache tier (Redis).
- Long conversations (illustrative: ~100 turns): summarize the conversation; store the summary (not the full raw history) in the cache tier as context, and additionally store that summary as an embedding in the vector DB.
- Cross-user benefit: if a different user asks a similar question, the system can retrieve a relevant prior summary from the vector DB, eliminating the need for the LLM to perform complex reasoning from scratch, since relevant context/precedent is already available. This reduces both latency and the reasoning burden placed on the final LLM call.
3.13 Closing Q&A: DevOps → MLOps Career Path (Lighter-Weight Capture)
The session ran long and closed with an open Q&A that shifted from system-design content into career and study-path guidance for DevOps engineers interested in moving toward MLOps/agentic-AI work. Captured here at a lighter weight (per standard handling for career-coaching-style content), but retaining the concrete technical study pointers given:
- Is this MLOps? The instructor’s view: designing agentic systems can be considered a form of MLOps if viewed from a systems perspective, but he predicts that software engineering broadly is trending toward becoming “MLOps” in general, since code-writing and execution itself is increasingly being automated by agentic AI — meaning the skill of designing good systems for agentic AI is becoming a more central, durable skill than any single current job title.
- What’s the minimum needed to move toward MLOps from DevOps?
- You do not need deep neural-network mathematics (e.g., the instructor explicitly says you don’t need to know exactly how backpropagation or continuous-time modeling works).
- You do benefit from understanding deep learning fundamentals at a basic level, and specifically the instructor recommends understanding the attention mechanism and how transformers/GPT-style models work, since this underlies concepts like KV caching discussed earlier in the session.
- Beyond that, knowing how to design a good agentic system (i.e., everything covered in Sections 3.6–3.12) is described as sufficient to begin working in an MLOps-adjacent role.
- What’s needed to become good (not just get started) in MLOps?
- Deeper observability skills specific to ML systems: identifying where and why a model is failing — e.g., is it hallucinating, or is intent classification going wrong.
- Understanding of overfitting and the need for validation checkpoints (a trainee raised RMSE — root mean square error — as an example metric for checking whether a model has been trained correctly).
- What is explicitly not MLOps (per the instructor) — i.e., where the boundary sits:
- The core model training workflow — preparing train/test/validation splits, preprocessing and purifying data, selecting a loss function and optimizer, and monitoring the training process itself for overfitting/bias/proper loss convergence — is characterized as core deep learning engineering / data science / research work, and explicitly not MLOps in the instructor’s framing. He does not think MLOps practitioners need to master this deeply, though basic awareness is implied to be useful.
- Resources offered (not detailed in-session): the instructor offered to share implementation repositories, blog posts, his hand-drawn whiteboard diagrams, and mentioned there being “one or two” comprehensive MLOps-focused books/syllabi he would follow up with separately — none of these were named specifically in the transcript (see Gaps & Assumptions).
4. Key Concepts Table
| Concept | Explanation | Example | Why It Matters |
|---|---|---|---|
| Latency Trap | Sequential LLM/tool calls stack up delay, often exceeding 10-20 seconds | Naive Grafana chatbot: prompt → LLM → tool → LLM → response | Users abandon systems that take >10 seconds to respond |
| State-in-RAM Anti-Pattern | Storing conversation/session state in local process memory breaks horizontal scaling | Pod A holds conversation history; a later request routed to Pod B has no memory of it | Prevents safe autoscaling and pod restarts (a core Kubernetes concern) |
| Tool Cascading | Sequential, dependent API calls where each waits on the previous one’s result | API A → API B → API C, latency = sum of all three | Creates unpredictable cost, latency, and behavior |
| Observability Gap | No visibility into which layer (LLM, tool, cache) caused a failure | Agent errors out with no indication of root cause | Structured logging/tracing/metrics (P50/P95) are required to debug production agents |
| ReAct Loop | Think → act → observe → think again pattern; sequential, blocking, unstructured | Naive agent deciding step-by-step what to do next | The baseline anti-pattern that DAG-based orchestration replaces |
| DAG (Directed Acyclic Graph) Orchestration | Tasks as nodes, dependencies as edges; independent tasks run in parallel | Policy lookup and delivery-address lookup run simultaneously | Enables parallelization, eliminating cascading latency |
| Three-Tier Memory Hierarchy | Cache (sub-second) / Vector DB (200ms-1s) / Data warehouse (durable metadata) | Redis for live convo state; vector DB for policy docs; SQL for IDs | Replaces unsafe RAM storage with scalable, purpose-fit storage |
| SLM (Small Language Model) Intent Classification | A lightweight model classifies user intent/routes requests, instead of using a large LLM for this simple task | Classify query into “policy_lookup” vs “delivery_address_lookup” | Removes the large, slow LLM from simple decision-making, cutting latency dramatically |
| Speculative Execution | Predicting intent and beginning retrieval from live/partial user input, before they finish typing | Detecting “refund policy” + “flight” mid-sentence and starting vector search early | Overlaps retrieval latency with typing time, reducing perceived response time |
| TTFT (Time to First Token) | The delay before a user sees the first word of the LLM’s response | Optimized via prompt caching, KV caching, and conversation summarization | The key perceptual latency metric users actually notice |
| System Prompt Caching | Pre-caching predictable system-prompt text per intent/route combination | Caching “this is vector DB data” framing text | Reduces prompt-assembly time (not raw inference time) |
| KV (Key-Value) Caching | Caching Query/Key/Value vectors from the transformer attention mechanism | Reuses cached K/V instead of recomputing on each token | Speeds up actual LLM inference, not just prompt assembly |
| Conversation Summarization Strategy | Full history stored for short conversations; long conversations are summarized and embedded | 10-turn convo stored raw; 100-turn convo stored as summary + vector embedding | Reduces context size sent to LLM, cutting cost/latency and enabling cross-user reuse |
| P50 / P95 Latency Metrics | P50 = median user’s latency; P95 = latency experienced by the 95th percentile of users | Used in structured observability for agent systems | P95 catches tail-latency problems that averages/medians hide |
| Prompt Injection Defense (Layered Prompting) | Separating the layer that constructs a prompt from the layer with actual data/tool access | User input doesn’t directly reach the data-access LLM layer unsanitized | Reduces the attack surface for prompt injection |
5. Architecture & Workflow Analysis
5.1 Naive (Anti-Pattern) Agent Design
User Query
|
v
Combine with System Prompt
|
v
LLM Reasoning Call #1 (~3-8 sec) <-- decides what tool/API to call
|
v
Tool / API Call #1 (~2-5 sec)
|
v
LLM Reasoning Call #2 (~3-8 sec) <-- interprets tool result, decides next step
|
v
[ ... repeats sequentially for each additional tool/API needed ... ]
|
v
Final Response to User
TOTAL: often 15-20+ seconds (exceeds user abandonment threshold)
Conversation/session state --> stored in local pod RAM (breaks horizontal scaling)
5.2 Production-Grade (DAG-Based) Agent Design
User Query
|
v
Small Language Model (SLM) -- Intent Classification & Routing (milliseconds)
|
+-------------------------+
| |
v v
[Parallel / Async Execution]
| |
Vector DB Search Relational DB / Cache Lookup
(semantic search, (structured per-user data,
~200ms-1s) or Redis if already "hot")
| |
+-------------------------+
|
v
Merge results + predefined
cached system prompt for this
intent/route combination
|
v
Final LLM Reasoning Call
(single call, KV-cached
inference where possible)
|
v
Response to User
Conversation/session state --> tiered memory hierarchy (Redis / Vector DB / SQL),
NOT local pod RAM
5.3 Three-Tier Memory Hierarchy
+-------------------------------+
Tier 1 (Cache) | Redis ("radius") | Target: < ~1 second
| - current conversation state |
| - immediate tool outputs |
| - short (~10-turn) full convo history
+-------------------------------+
|
v
+-------------------------------+
Tier 2 (Vector) | Vector Database | Target: ~200ms - 1 second
| - embeddings for semantic search
| - policy documents
| - summarized long (~100-turn) conversation history
+-------------------------------+
|
v
+-------------------------------+
Tier 3 (SQL/DW) | Data Warehouse / RDBMS | Durable, non-latency-critical
| - request IDs, session IDs
| - conversation IDs, metadata
+-------------------------------+
5.4 Speculative Execution Timeline
Normal flow:
User types entire query --> [submit] --> Intent classification --> Retrieval --> LLM synthesis --> Response
|------------------- all sequential after submit -----------------|
Speculative execution flow:
User starts typing -----> (live partial input via WebSocket)
|
v
SLM predicts intent from partial input (happens WHILE user is still typing)
|
v
Vector search / DB retrieval begins EARLY (overlapped with remaining typing time)
|
v
User finishes typing --> [submit] --> (retrieval likely already complete)
|
v
LLM synthesis (still ~5 sec, but now the
ONLY visible wait) --> Response
5.5 E-Commerce Customer Support Agent — End-to-End Flow
User: "I ordered shoes last week. Haven't received them. I want to change the
delivery address also. Can I get a refund if it's delayed?"
|
v
SLM Intent Classifier --> splits into multi-intent:
| (a) Policy check (refund/address-change rules)
| (b) Delivery address lookup
|
+----------------------------+----------------------------+
| |
v v
Vector DB Search Relational DB Lookup
(policy document, (user's delivery address,
semantic search) order/shipping status)
| |
+----------------------------+----------------------------+
|
v
Merge results + cached system prompt
|
v
Final LLM Reasoning Call
|
v
Response to User
Follow-up turn ("change my pin code instead"):
delivery-related context now served from Redis (already "hot"),
not re-queried from the relational DB.
5.6 Component Roles
- SLM (Small Language Model) Classifier/Router: the fast “front door” of the system — decides intent and destination in milliseconds, replacing a slow LLM decision step.
- Orchestrator (DAG engine): the structural “brain” that determines which tasks can run in parallel vs. which have dependencies, replacing free-form LLM-driven ReAct looping.
- Redis (cache tier): serves anything needed in under ~1 second — live conversation state, hot lookups, short conversation history.
- Vector DB: serves large, semantically searchable content — policy documents, long-conversation summaries, embeddings.
- Relational DB / Data Warehouse: serves structured, durable metadata — IDs, per-user records, non-latency-critical lookups.
- Final LLM: used exclusively for the last-mile reasoning/synthesis step, after all necessary context has already been gathered in parallel.
- Proxy / Guardrail layer: sits in front of any outbound call to an LLM or agent, inspecting and redacting sensitive data before it can leave the environment.
6. Commands & Configurations
Note: This was a whiteboard/conceptual system-design session, not a hands-on coding or infrastructure walkthrough. No real, runnable code, YAML, or CLI commands were dictated or shown on screen. The instructor did narrate illustrative code shapes in spoken pseudocode form to communicate architectural intent; these are captured below as illustrative pseudocode, not verbatim source.
| Illustrative Pseudocode / Concept | Purpose | Explanation |
|---|---|---|
intent, route = slm_classifier(model="amazon-titan-or-similar", query=user_query) | Intent classification & routing | A single lightweight call to an SLM (or even a classical ML model) that returns both the classified intent and which backend to route to |
task1 = async(vector_db_search, query=policy_query) <br> task2 = async(db_lookup, query=delivery_address_query) <br> result1, result2 = await asyncio.gather(task1, task2) | Parallel execution of independent lookups | Demonstrates the shift from sequential tool cascading to concurrent, non-blocking retrieval using async/await-style execution |
| P50 / P95 latency tracking | Observability metric collection | Referenced as the standard metrics to track per-request/per-tool latency in structured logging, to catch both typical and tail-latency issues |
| Trace/Request ID per LLM call | Distributed tracing | Referenced as necessary for correlating logs across the multiple LLM/tool calls in a single user request |
| WebSocket-based live input streaming | Enables speculative execution | Referenced as the mechanism for consuming a user’s partial/live typed input before they submit, to enable predictive intent classification |
7. Tools & Technologies
-
LLM (Large Language Model) — e.g., general-purpose reasoning models.
- Purpose: final-stage reasoning and response synthesis after all context has been gathered.
- When to use: reserved for the single, final reasoning step — not for cheap classification/routing decisions.
- Advantages: strong reasoning and language generation capability.
- Limitations: slow (multi-second latency per call) and expensive relative to smaller models; should not be used for simple, high-frequency decisions.
-
SLM (Small Language Model) — example given: Amazon Titan (or a classical ML classifier).
- Purpose: fast intent classification and request routing.
- When to use: any decision that can be reduced to a finite, predefined set of intents/categories.
- Advantages: millisecond-scale latency, far lower cost than a large LLM for the same decision.
- Limitations: not suited for open-ended reasoning tasks — only classification-style decisions.
-
Redis (referred to as “radius” in the transcript)
- Purpose: sub-second cache layer for live conversation state, immediate tool outputs, and short conversation history.
- When to use: any data needed in under roughly 1 second.
- Advantages: extremely fast key-based lookups.
- Limitations: not designed for semantic/similarity search or large durable datasets.
-
Vector Database
- Purpose: stores embeddings for semantic/similarity search — e.g., policy documents, long-conversation summaries.
- When to use: content that is large but needs to be searched by meaning rather than exact match, with a ~200ms-1s latency budget.
- Advantages: enables fast semantic search across large unstructured/text corpora.
- Limitations: not intended for simple structured/relational lookups (e.g., per-user address records).
-
Relational Database / Data Warehouse (e.g., PostgreSQL-style)
- Purpose: durable storage of structured metadata — request IDs, session IDs, conversation IDs, per-user structured records like delivery addresses.
- When to use: data with no strict sub-second latency requirement, or naturally relational/structured data.
- Advantages: strong consistency and structured query capability.
- Limitations: not optimized for either sub-second cache-style access or semantic search.
-
AWS Bedrock (mentioned as an example trusted enterprise AI platform)
- Purpose: managed, enterprise-trusted way to deploy/access AI agents without directly integrating arbitrary third-party AI APIs.
- When to use: enterprises that already store data on AWS and want to extend an existing trust relationship to AI agent access.
- Advantages: reduces the security exposure of directly wiring internal data into an untrusted external AI provider.
- Limitations: not deeply explored in this session beyond being cited as an illustrative example.
-
WebSockets
- Purpose: enables consuming a user’s live, partial input (as they type) rather than waiting for full message submission.
- When to use: implementing speculative execution / predictive intent classification.
- Advantages: allows overlapping retrieval latency with the user’s typing time.
- Limitations: implementation details were not elaborated in this session.
-
LangChain (referenced by name)
- Purpose: mentioned as the kind of library/framework used to wire up an SLM call for intent classification (e.g., “it will be something like a LangChain call…”).
- When to use: building the glue code between a chosen small/large model and the application logic.
- Advantages/Limitations: not elaborated in depth in this session; referenced only as an example of the implementation layer.
8. Real-World Production Usage
-
Enterprise use case demonstrated: an AI-powered customer support agent for a large e-commerce platform, handling multi-intent queries (delivery-address changes + refund policy questions) — a realistic production scenario requiring parallel data retrieval and policy-aware reasoning.
-
Enterprise use case referenced (DevOps-specific): an AI chatbot layered over an observability tool (Grafana) to help diagnose production errors — explicitly used throughout the session as the running example connecting agentic-system design to DevOps day-to-day work.
-
Production implementation patterns highlighted:
- Never rely on in-process RAM for conversational/session state in a system expected to scale horizontally or run under an orchestrator like Kubernetes (where pods can be restarted or requests load-balanced across replicas at any time).
- Use a tiered memory strategy matched to latency requirements, rather than a single storage backend for everything.
- Reserve expensive, high-latency LLM reasoning calls for the final synthesis step only; push all classification/routing decisions to cheaper, faster models.
- Parallelize independent retrieval steps using async execution rather than allowing sequential tool cascading.
-
Security considerations: guardrails, proxy-based data inspection/redaction (paralleling ingress-controller patterns familiar to a DevOps audience), trusted managed AI platforms (e.g., AWS Bedrock) for enterprises with existing cloud trust relationships, layered prompt construction to reduce prompt-injection risk, and restricting the inference surface exposed to end users.
-
Cost optimization considerations: routing simple classification decisions to small/cheap models instead of large LLMs is presented as both a latency and cost optimization — every avoided large-LLM call directly reduces per-request cost. Caching (system prompts, KV cache, conversation summaries) similarly reduces redundant compute/token spend.
-
Scalability considerations: the entire redesign — DAG-based parallel execution, externalized tiered memory (instead of RAM), and stateless request routing — is explicitly framed as what enables horizontal scaling and safe pod restarts in a Kubernetes-style production environment, directly addressing the state-bottleneck failure mode identified in Section 3.2.
9. Interview Preparation
Beginner Questions
Q1: What are the four core reasons naive AI agents fail in production? A: (1) The latency trap — sequential LLM/tool calls stacking up to 10-20+ seconds; (2) storing conversation/session state in local RAM, which breaks horizontal scaling; (3) tool cascading — sequential, dependent API calls that create unpredictable cost/latency/behavior; (4) an observability gap — no visibility into which layer (LLM, tool, cache) caused a failure.
Q2: Why is storing conversation history in a pod’s local RAM a problem in a Kubernetes environment? A: If a load balancer routes a user’s next request to a different pod than the one that handled their first request, that new pod has no access to the RAM-stored history, silently losing context. It also means the system cannot safely autoscale or allow Kubernetes to restart pods without losing active sessions.
Q3: What is the general rule of thumb for how response latency affects user experience? A: Under 100ms feels instant, under 1 second feels smooth, 2-3 seconds becomes noticeable, over 5 seconds causes frustration, and over 10 seconds causes users to abandon the product.
Intermediate Questions
Q1: Why use a small language model (SLM) for intent classification instead of the same large LLM used for reasoning? A: Intent classification is a simple, bounded classification task (mapping a query to one of a predefined set of intents), which doesn’t require the reasoning capability of a large LLM. Using an SLM (or even a classical ML classifier) for this step reduces latency from multi-second LLM reasoning time down to milliseconds, and significantly reduces cost, since the large LLM is reserved only for the final synthesis step.
Q2: Describe the three-tier memory hierarchy used in a production agentic system and what belongs in each tier. A: Tier 1 is a cache (e.g., Redis) for anything needed in under ~1 second — current conversation state, immediate tool outputs. Tier 2 is a vector database for large, semantically-searchable content needing ~200ms-1s latency — embeddings, policy documents, conversation summaries. Tier 3 is a relational database/data warehouse for structured, durable metadata like request/session/conversation IDs, without strict sub-second requirements.
Q3: What is “tool cascading” and why is it a problem? A: Tool cascading is when an agent’s workflow requires calling one API, then based on that result calling a second API, then a third — each step sequentially dependent on the previous one’s output. Total latency becomes the sum of every step’s latency plus reasoning time in between, and any single failure in the chain can trigger retries that compound both latency and cost unpredictably.
Q4: How does converting a sequential agent workflow into a DAG (Directed Acyclic Graph) improve performance? A: In a DAG, independent tasks (nodes with no dependency relationship between them) can be identified and executed in parallel using asynchronous execution, rather than being forced through a single sequential chain. This directly addresses the tool-cascading latency problem by overlapping work that doesn’t need to happen in order.
Advanced Questions
Q1: Walk through the full optimized architecture for a multi-intent customer support query, from user input to final response. A: (See Section 3.10 and diagram 5.5 in full.) In short: user query → SLM classifies it into parallel sub-intents → independent retrieval tasks (e.g., vector DB semantic search for policy, relational DB lookup for user data) execute concurrently via async/await → results are merged with a pre-cached, intent-specific system prompt → a single final LLM call performs reasoning/synthesis over the combined context → response returned to the user. Follow-up turns in the same conversation can be served from the cache tier instead of re-querying the database.
Q2: What is speculative execution in the context of AI agents, and what latency does it actually reduce? A: Speculative execution consumes a user’s live, partial input (e.g., via WebSockets) before they finish typing, using an SLM to predict intent early and begin retrieval (vector search/DB lookup) in the background while the user continues typing. It is important to note this does not speed up the final LLM reasoning/inference call itself — that step still takes roughly the same time. What it eliminates is the perceived wait, because retrieval work that would normally happen after submission has already completed by the time the user hits submit, overlapping it with typing time instead of adding it afterward.
Q3: Explain the difference between reducing TTFT via system-prompt caching versus KV caching. A: System-prompt caching reduces the time to assemble and dispatch the final prompt to the LLM by pre-caching the predictable framing/instruction text associated with a given intent/route combination — it does not make the LLM’s own inference any faster. KV (Key-Value) caching, by contrast, targets the transformer/attention mechanism itself: by caching the Query/Key/Value vectors generated during attention computation, the model avoids recomputing them from scratch, which does directly speed up the actual inference process.
Q4: In the layered security model described for agentic systems, why is separating the “prompt construction” layer from the “data access” layer an effective mitigation against prompt injection? A: If a single LLM layer both directly receives raw, unsanitized user input and has control over sensitive data/tool access, a malicious or malformed user input can potentially manipulate that layer into misusing its data access. By splitting this into two layers — one that only constructs/combines the prompt with a system prompt (and has no data-access privileges), and a second layer that performs the actual data access/reasoning (and is shielded from directly ingesting raw unsanitized user text) — the attack surface for prompt injection affecting privileged operations is reduced.
10. Exam & Certification Notes
While this is not tied to a specific vendor certification, the concepts are increasingly relevant to MLOps-adjacent interview and system-design assessments, and to general AI/ML system-design rounds in DevOps/SRE/Platform Engineering interviews:
- Frequently tested concept: The distinction between what belongs in a cache, a vector database, and a relational/data warehouse layer, driven by latency requirements and data shape (structured vs. semantically-searchable vs. hot/live).
- Frequently tested concept: P50 vs. P95 latency — a common trick question is confusing “average latency” with these percentile metrics; P95 specifically surfaces tail-latency issues that an average or median can mask.
- Potential trick question: “Does speculative execution make the LLM respond faster?” — No; it does not reduce the LLM’s own inference time. It overlaps retrieval latency with the user’s typing time, reducing perceived end-to-end latency only.
- Potential trick question: “Is KV caching the same as caching the system prompt?” — No; system-prompt caching only speeds up prompt assembly/dispatch, while KV caching speeds up the actual transformer inference computation via cached attention Key/Value vectors.
- Memorization-worthy point: The four naive-agent failure modes: latency trap, RAM-based state (horizontal scaling breakage), tool cascading, observability gap.
- Memorization-worthy point: User latency tolerance thresholds: <100ms instant, <1s smooth, 2-3s noticeable, >5s frustrated, >10s abandonment.
- Frequently tested concept: Why storing session state in local process memory is fundamentally incompatible with horizontally-scaled, container-orchestrated deployments (a common systems-design interview theme that bridges classical distributed-systems knowledge with AI agent design).
11. Cheat Sheet
Four Reasons Naive Agents Fail:
- Latency trap (sequential calls, 10-20+ sec)
- State stored in RAM (breaks horizontal scaling)
- Tool cascading (sequential dependent API calls)
- Observability gap (can’t tell where failure occurred)
User Latency Tolerance:
- < 100ms = instant
- < 1s = smooth
- 2-3s = noticeable
-
5s = frustrated
-
10s = abandonment
Three Pillars of a Production-Grade Agent:
- Orchestrator/Brain → DAG-based (not free-form ReAct loop)
- Tiered Memory → Cache (Redis) / Vector DB / Data Warehouse
- SLM for intent+routing, LLM reserved for final reasoning only
Memory Tier Quick Reference:
| Tier | Latency Target | Data |
|---|---|---|
| Cache (Redis) | < ~1s | live conversation state, short history |
| Vector DB | ~200ms-1s | embeddings, policy docs, long-convo summaries |
| SQL/Data Warehouse | not latency-critical | IDs, structured metadata |
Latency Optimization Techniques:
- Speculative execution → predicts intent from live/partial typed input, starts retrieval early (reduces perceived latency only)
- System prompt caching → speeds up prompt assembly (not inference)
- KV caching → speeds up actual LLM inference via cached attention Key/Value vectors
- Conversation summarization → short convos stored raw; long convos summarized + embedded
Security Layers for Agentic Systems:
- Guardrails → limit data access scope
- Proxy layer → inspects/redacts sensitive data on outbound calls
- Trusted managed AI platforms (e.g., AWS Bedrock) → leverage existing enterprise trust
- Prompt construction/data-access layer separation → mitigates prompt injection
- Restricted inference surface → limit users to pre-selected forms/questions
Illustrative Parallel Execution Pattern:
task1 = async(call_1)
task2 = async(call_2)
result1, result2 = await asyncio.gather(task1, task2)
12. Gaps & Assumptions
- ”DAG” terminology inconsistency: The instructor repeatedly says “directed cyclic graph” verbally, while also explicitly naming it “DAG” and describing behavior that is unambiguously acyclic (“it can never circle back to itself… if a graph becomes cyclic, it will be stuck in a loop”). Standard terminology for this pattern is Directed Acyclic Graph. Assumption: this is a verbal slip in the source recording, not an intentional distinct concept, and this package uses “Directed Acyclic Graph (DAG)” throughout as the corrected, intended term. This is flagged rather than silently corrected, per the source-fidelity instructions.
- ”Radius” likely means “Redis”: Throughout the transcript, the speaker consistently says what was transcribed as “radius” when describing a fast, sub-second cache layer alongside references to conversation state, TTL-like sub-second latency, and general caching use cases. Assumption: this is a mishearing/mistranscription of “Redis”, the well-known in-memory cache/data store, and this package uses “Redis” throughout while noting the transcript’s literal wording here for transparency. This was not explicitly confirmed verbatim in the source (e.g., no spelling was given on screen).
- ”ReAct” full expansion not given: The instructor explicitly states he does not recall/know the full expansion of the “ReAct” acronym during the session. This package uses “ReAct” as commonly understood in the agentic-AI literature (Reasoning + Acting) for context in the Key Concepts table, but this expansion was not stated in the transcript itself and should be treated as background knowledge added for clarity, not something the instructor confirmed.
- Eviction/third-rule-type gap not applicable here — not relevant to this session (carried over caution from a prior session’s format; no equivalent unresolved technical detail was identified in this transcript beyond the two items above).
- No real, verbatim code/config was provided: This was a conceptual whiteboard session. All pseudocode in Sections 3.9, 3.10, and 6 (the SLM classifier call, the
asyncio.gatherpattern) is reconstructed from the instructor’s spoken description of what the code would look like, not copied from an actual screen-shared file or repository. It should be treated as illustrative of the pattern, not as production-ready syntax. - Resource references not identified: Near the end of the session, the instructor offers to share implementation repositories, blog posts, whiteboard diagrams, and “one or two” MLOps-focused books/syllabi, but does not name any of them specifically in the transcript (he states he doesn’t remember the book name and will follow up separately). These resources are therefore not included in this package, since no verifiable title/link was given in the source material.
- Speaker attribution: As with prior sessions, individual trainee questions are attributed by first name/nickname as they appear in the transcript (e.g., “Sai,” “Kishor,” “Nish,” “Nra/Surendra”) where reasonably identifiable from dialogue context; the primary instructor delivering the masterclass content is not explicitly named in this transcript and has therefore been left unspecified in this package rather than guessed.
- Career-guidance section captured at lighter weight: Consistent with how career-coaching content is generally handled in this workflow, Section 3.13 (DevOps → MLOps career path) is captured as a condensed advisory summary rather than a full technical breakdown, since its primary content is study-path guidance rather than a system-design walkthrough — though the specific technical study pointers given (attention mechanism, transformers, overfitting, train/test/validation) were preserved since they are concrete and reusable.