Production-Grade AI Agent Systems Design

Structured educational resource covering production-grade ai agent systems design.

senior 45 min read 13 sections

#kubernetes#aws#cost-optimization#debugging

Complete Learning Package — Scalable, Low-Latency, Streamable Agents for Production

Guest Session by Lead Architect | SRE Labs (Advanced Track) — AI in DevOps Track

Source: 2026-03-01-11-07-12.md — Morning session (11 AM IST, 1 March). Guest presenter: Lead Architect (AI/ML engineer; MLOps practitioner). Billed as the “AI in DevOps” additional call announced at the end of the previous session. This is a whiteboard-first session — no code was shared live, though the presenter promised to share example repositories and blog links afterward.

Context within the program: This is a standalone guest session, not part of the main Titan Grid project arc. It connects to DevOps via the premise that agentic AI systems will increasingly automate DevOps tasks (incident diagnosis, log analysis, automated remediation). The content is primarily AI systems design with DevOps as the motivating use case.

What this is and is not: This session covers system architecture for production AI agents — intent routing, memory hierarchy, parallelisation, speculative execution, TTFT optimisation. It does NOT cover training models, neural network mathematics, or MLOps tooling (Kubeflow, MLflow). The presenter explicitly scoped those out. Think of it as “distributed systems design, but the distributed component is an LLM.”

2. Table of Contents

Executive Summary
Table of Contents
The Four Failure Modes of Naive Agents
- 3.1 Failure Mode 1 — The Latency Trap
- 3.2 Failure Mode 2 — The State Bottleneck (RAM Storage)
- 3.3 Failure Mode 3 — Tool Cascading
- 3.4 Failure Mode 4 — Observability Gap
The Production Architecture — Four Solutions
- 4.1 Solution 1 — Replace ReAct with a DAG Orchestrator
- 4.2 Solution 2 — Three-Tier Memory Hierarchy
- 4.3 Solution 3 — SLM Intent Classifier + Parallel Routing
- 4.4 Solution 4 — Speculative Execution
TTFT Optimisation (Reducing LLM Synthesis Latency)
- 5.1 System Prompt Caching
- 5.2 KV Caching
- 5.3 Conversation History Summarisation
Security Considerations for Agentic Systems
The DevOps Grafana AI Chatbot — Worked Example
The E-commerce Customer Support Agent — Worked Example
Architecture & Workflow Analysis
Key Concepts Table
DevOps-to-MLOps Career Path
Interview Preparation
Cheat Sheet
Gaps, Assumptions & What Was Promised

3. The Four Failure Modes of Naive Agents

A “naive agent” is what any competent engineer builds on their first attempt — sequential, stateful, LLM-driven. Here’s exactly why it fails in production.

3.1 Failure Mode 1 — The Latency Trap

The naive flow:

User submits query
  → Query + system prompt sent to LLM (3–8 seconds reasoning)
  → LLM decides what tool to call
  → Tool API called (2–5 seconds)
  → Tool result returned to LLM
  → LLM synthesises final answer (3–8 seconds)
Total: 8–21 seconds minimum

User attention thresholds (empirically established UX research):

Delay	User perception
< 100ms	Instant — feels immediate
< 1 second	Smooth — user doesn’t notice
2–3 seconds	Noticeable delay
> 5 seconds	Frustrated — user actively waiting
> 10 seconds	User abandons the session entirely

At 20+ seconds, no production system is viable. This is why naive AI agents that “work” in tutorials cannot be used in real products.

Example (Grafana AI Chatbot): User asks: “Why is the error rate so high right now?”

LLM combines question + system prompt → decides to read the codebase
API call to GitLab/GitHub: fetches relevant files
LLM reads file + identifies error → another LLM call to write the fix
Total: 15–25 seconds for the first token

Even Cursor and GitHub Copilot — if they used this naive design — would be unusable. They don’t; they use optimised architectures.

3.2 Failure Mode 2 — The State Bottleneck (RAM Storage)

What naive agents do:

conversation_history = []   # defined at startup, lives in RAM

def chat(user_message):
    conversation_history.append(user_message)
    response = llm.call(conversation_history)
    conversation_history.append(response)
    return response

Why this works for one user, breaks for many:

Single pod, single user: works fine
                         ┌─────────┐
User → Load Balancer →   │  Pod A  │ (has conversation_history in RAM)
                         └─────────┘

Multiple pods (after horizontal scaling):
                         ┌─────────┐
Request 1 → Pod A        │  Pod A  │ ← conversation_history LIVES HERE
Request 2 → Pod B        └─────────┘
                         ┌─────────┐
                         │  Pod B  │ ← NO conversation_history here
                         └─────────┘
                         Pod B sees Request 2 with no prior context.
                         Agent's reasoning is completely broken.

Consequences:

Cannot autoscale (adding pods breaks sessions)
Kubernetes pod restart = all session state lost
Load balancer cannot route sessions intelligently because state is invisible to it
Any data stored in RAM is at risk: conversation history, embeddings, system prompts, tool outputs, database query results

This is a fundamental horizontal scaling failure. Any agent that stores state in memory variables is architecturally incompatible with Kubernetes auto-scaling.

3.3 Failure Mode 3 — Tool Cascading

What cascading looks like:

User query → LLM decides to:
  Step 1: Call API A (fetch order status) → 2–5 seconds
  Step 2: Based on result of Step 1, call API B (check refund policy) → 2–5 seconds
  Step 3: Based on result of Step 2, call API C (check delivery address) → 2–5 seconds
  Step 4: Combine all results → send to LLM for reasoning → 3–8 seconds
Total: 9–23 seconds just for tool calls

Three compounding problems:

Unpredictable latency — each LLM call might decide to add another tool call; you don’t know when it will stop
Exponential cost — each retry (on LLM failure) repeats the entire cascade, multiplying API call costs
Unpredictable behaviour — LLM might cascade differently on each call for the same query, making the system non-deterministic

The root cause: Giving the LLM full autonomy to decide the order and number of tool calls. The LLM is designed for reasoning, not for planning efficient call graphs.

3.4 Failure Mode 4 — Observability Gap

What happens when a naive agent fails:

User sees: “There was an error processing your request.”
Developer sees: nothing useful

Questions that cannot be answered without proper observability:

Did the LLM hallucinate the answer?
Did the API call fail (network error, timeout, rate limit)?
Did the cache layer return stale data?
Was the intent classification wrong (user asked about X, agent searched for Y)?
How many tokens did this call consume? What did it cost?
At which step did latency spike?

Solution (because DevOps engineers already know observability):

Structured logging at every layer:
  - Log at prompt level: what was the full prompt sent to LLM?
  - Log token count: how many input/output tokens?
  - Log tool calls: which tools were called, in what order?
  - Log latency per step: intent classification, each API call, LLM synthesis

Tracing:
  - Assign a unique request ID to every user query
  - Every LLM call, tool call, and DB query carries that request ID
  - End-to-end trace: user query → final response, step by step

Metrics:
  - P50 latency: latency experienced by 50% of users
  - P95 latency: latency experienced by 95% of users (the tail matters)
  - Error rate per step (LLM errors, API errors, cache misses)
  - Token cost per query (for budget management)

This is identical to how you’d instrument a microservice — the same Prometheus/Grafana/tracing stack applies. The difference is the additional LLM-specific metrics (token count, hallucination rate, intent classification accuracy).

4. The Production Architecture — Four Solutions

4.1 Solution 1 — Replace ReAct with a DAG Orchestrator

What ReAct is (the naive approach): ReAct = Reason + Act (the standard naive agent loop)

Model THINKS about what to do
  → Model ACTS (calls a tool)
  → Model OBSERVES the result
  → Model THINKS again
  → Model ACTS again
  → [loop continues until model decides it's done]

Problems: sequential (cannot parallelise), blocking (waits for each step), unstructured (no defined end state), unpredictable (model decides how many loops to run).

What a DAG orchestrator is:

A Directed Acyclic Graph (DAG) is a graph where:

Nodes = individual tasks (intent classification, DB query, vector search, LLM reasoning)
Edges = dependencies between tasks (DB query result feeds into LLM reasoning)
Directed = dependencies flow in one direction
Acyclic = no loops — the graph can never circle back to a previous state

Why ACYCLIC is critical:
  If the graph had a cycle:
    Node A → Node B → Node C → Node A (cycle)
  The agent would loop forever: think → act → think → act → ...
  This is exactly what ReAct does. A DAG prevents this.

The DAG replaces the LLM as the orchestration brain:

User query
  ↓
Intent Classifier (SLM — fast, cheap)
  ↓ (parallel branches for each intent)
  ├── Branch 1: Vector DB search (policy documents)
  └── Branch 2: SQL DB query (user order data)
         ↓ (both complete asynchronously)
  Results aggregated
         ↓
  LLM Reasoning (only ONCE, with pre-assembled context)
         ↓
  Final response to user

The key insight: The LLM is no longer the orchestrator. A rule-based, code-defined DAG is the orchestrator. The LLM only does what it’s uniquely suited for: final reasoning and response generation. Everything else is handled by faster, cheaper, deterministic components.

Python implementation (three lines of parallelism):

import asyncio

async def handle_query(user_query):
    # Intent classification (SLM — milliseconds)
    intents = slm.classify(user_query)   # returns: ['vector_db', 'sql_db']

    # Parallel execution of all detected intents
    task1 = asyncio.create_task(vector_db.search(query))      # policy docs
    task2 = asyncio.create_task(sql_db.fetch(user_id=...))    # order data

    result1, result2 = await asyncio.gather(task1, task2)

    # Final LLM reasoning with pre-assembled context
    response = await llm.reason(query, context=[result1, result2])
    return response

asyncio.gather(task1, task2) executes both tasks in parallel. If each takes 200ms, the total is ~200ms — not 400ms (sequential). At 10 parallel tasks, the saving is 9× the per-task latency.

4.2 Solution 2 — Three-Tier Memory Hierarchy

Never store any state in application RAM. Use an external memory system with three tiers:

TIER 1: Redis (Hot Memory — <10ms access)
  What to store:
    - Current conversation messages (last N turns)
    - Tool call outputs from this session
    - Intermediate reasoning steps
    - Any data needed immediately in this request's context
  When to use:
    - Data needed within the same conversation session
    - Data required in < 1 second
    - Data that changes frequently (live state)

TIER 2: Vector Database — Pinecone, Weaviate, pgvector (Warm Memory — 100–500ms)
  What to store:
    - Document embeddings (policy PDFs, knowledge base articles)
    - Historical conversation summaries (see §5.3)
    - Any large data that needs semantic search
  When to use:
    - "Find me the policy document about refunds" (semantic search, not exact match)
    - Cross-user pattern matching ("has this error happened before?")
    - Data needed within 200ms–1 second

TIER 3: SQL / Data Warehouse — PostgreSQL, BigQuery (Cold/Structured — 10–100ms for indexed queries)
  What to store:
    - User metadata (user_id, address, order history, preferences)
    - Session metadata (request_id, session_id, conversation_id)
    - Transaction records, audit logs
  When to use:
    - Structured lookups with known keys ("get order #12345 for user ABC")
    - Anything that needs to be queryable with SQL predicates

Routing example (e-commerce bot):

User: "Where is my order from last week?"

Intent: order_status_lookup
Route: SQL DB
Query: SELECT status, estimated_delivery FROM orders WHERE user_id = :uid ORDER BY created_at DESC LIMIT 1

Second turn: "What about the PIN code for that delivery?"
Route: Redis (the order data from Turn 1 is already cached here)
No new DB query needed — answer from cache in <10ms

The routing decision — where does data live?

Data type	Storage	Rationale
Policy document (refund, shipping)	Vector DB	Large, searchable, not user-specific
User order details	SQL DB (first access) → Redis (subsequent)	Structured, user-specific
Conversation history (short)	Redis	Needs to be instant; changes every turn
Conversation summary (long history)	Redis + Vector DB	Summary in Redis; embedding in Vector DB for cross-user matching
Error logs, metrics	SQL / Data Warehouse	Structured, queryable, large volume

4.3 Solution 3 — SLM Intent Classifier + Parallel Routing

The two-model pattern:

SLM (Small Language Model): Classifies the user’s query into predefined intents and routes to the correct memory tier. Cheap, fast (milliseconds), deterministic.
LLM (Large Language Model): Performs the final reasoning and generates the natural language response. Expensive, slow (seconds), probabilistic.

What “intent” means: A predefined category of action the agent can take. Examples:

INTENTS = [
    "api_call",          # call an external service
    "vector_db_search",  # semantic search in documents
    "sql_db_query",      # structured database lookup
    "redis_cache_read",  # read from conversation cache
    "direct_response",   # LLM can answer without any tool call
]

How intent classification works:

from langchain import AmazonTitan   # or any small/cheap model

def classify_intent(user_query):
    # SLM classifies into predefined intents + returns route
    result = slm.invoke(
        prompt=f"Classify this query into one or more intents: {user_query}\n"
               f"Available intents: {INTENTS}\n"
               "Return: {{'intents': [...], 'routes': [...]}}"
    )
    return result   # e.g., {'intents': ['vector_db_search', 'sql_db_query'],
                    #         'routes': ['policy_docs', 'orders_db']}

Multi-intent handling: A single user query can have multiple intents that are processed in parallel:

"I haven't received my order and want to know the refund policy"
    ↓
Intent classifier (SLM — ~50ms):
    Intent 1: sql_db_query → route: orders table → fetch order status
    Intent 2: vector_db_search → route: refund_policy.pdf → semantic search
    ↓
asyncio.gather([sql_query, vector_search])  # run in parallel (~200ms)
    ↓
Merge results → LLM reasoning → Response
Total: ~300ms (vs 20+ seconds naive)

Why SLM, not LLM, for classification: The LLM “knows” about your intents but it takes 3–8 seconds to respond. A small model (fine-tuned on your intent list, or even a classical ML classifier) returns in < 100ms. The task is classification — not reasoning. Matching a query to a predefined list of 10 categories does not require GPT-4-level reasoning. Use the cheapest model that can do the job.

4.4 Solution 4 — Speculative Execution

The idea: Start processing before the user finishes typing.

Standard flow (non-speculative):

User types full query → hits Enter → query sent → intent classification → data fetching → LLM → response
[User typing: 5–15 seconds] + [processing: 300ms] = user waits 5–15 seconds minimum

Speculative execution flow:

User STARTS typing "What's the refund policy if my flight—"
  → WebSocket sends live keystrokes to intent classifier
  → After a few words, SLM predicts: intent = refund_policy_lookup, route = policy_docs
  → Vector DB search for refund policy STARTS IMMEDIATELY
  → By the time user finishes typing "—from Delhi to Mumbai is cancelled?"
    → Policy document is ALREADY in context
  → Query goes straight to LLM with pre-fetched context
  → Response in < 2 seconds (most of the work was done during typing)

Implementation:

Uses WebSockets (streaming input, not request-response)
SLM classifies intent on partial sentence (first 3–5 words often sufficient)
Prefetch only if confidence > threshold (to avoid wasted API calls)
If speculative fetch is wrong (rare), fall back to standard flow on final query

Why it works especially well in practice:

Many user queries are similar → speculated data is often already in Redis cache from a previous user’s query
Vector DB data (policy documents) is stable → cached result from the speculative fetch is almost always correct
Even if the speculative fetch takes the “wrong” route, the second (correct) route completes in < 200ms — the user barely notices

5. TTFT Optimisation (Reducing LLM Synthesis Latency)

TTFT = Time To First Token — the time from when the full context reaches the LLM to when the user sees the first word of the response. This is the one component that cannot be eliminated (you must use the LLM), but it can be reduced.

5.1 System Prompt Caching

The problem: Every LLM call needs a system prompt — instructions telling the LLM what role to play and how to interpret the context. Generating this prompt from scratch every time wastes tokens and time.

The solution: Pre-define system prompts for each intent+route combination and cache them:

SYSTEM_PROMPTS = {
    "refund_policy": "You are a customer support agent. The following is a company refund policy document. Use it to answer the user's question accurately and briefly.",
    "order_status": "You are a customer support agent. The following is the user's order data from the database. Answer the user's question about their order status.",
    "combined_refund_order": "You are a customer support agent. You have both the user's order data and the refund policy. Answer the user's question comprehensively.",
}

# Intent + route → instantly fetch cached system prompt
system_prompt = SYSTEM_PROMPTS[f"{intent}_{route}"]

No LLM call needed to generate the system prompt. The correct one is selected in microseconds based on the detected intent.

5.2 KV Caching

Background: Internally, LLMs represent text as mathematical vectors called Query (Q), Key (K), and Value (V) — from the Transformer attention mechanism. Every token in the input is converted to K and V vectors before reasoning begins.

The problem: For repeated system prompts or frequently-sent context, the LLM recalculates K and V vectors every time — wasted computation.

KV caching: Store the pre-computed K and V vectors for stable context (especially system prompts) so the LLM doesn’t recalculate them on every call.

Without KV cache:
  Full prompt (1000 tokens) → LLM computes K,V for ALL 1000 tokens → response

With KV cache:
  System prompt (200 tokens) → K,V computed ONCE → cached
  Next call: K,V for system prompt REUSED → LLM only computes K,V for new 800 tokens
  Inference is significantly faster + cheaper (fewer token computations)

KV caching is typically a configuration option in LLM serving infrastructure (vLLM, TGI, AWS Bedrock). For DevOps engineers: if you’re deploying self-hosted LLMs, enabling KV cache is one of the highest-ROI optimisations.

5.3 Conversation History Summarisation

The problem: As a conversation grows, sending the full history to the LLM becomes expensive (more tokens = more cost + slower inference) and eventually exceeds the context window.

The solution — sliding window + summarisation:

Short conversation (≤ 10 turns):
  Store exact history in Redis
  Send exact history to LLM as context
  Fast, precise

Long conversation (> 10 turns):
  Store: [last 10 messages (exact)] + [summary of all prior messages]
  The summary is generated by a cheap LLM call when conversation reaches the threshold
  Store summary in Redis (immediate access) and as embedding in Vector DB (cross-user matching)
  Send: last 10 exact messages + summary to LLM
  LLM processes fewer tokens → faster TTFT + lower cost

Cross-user benefit: When user B asks a question similar to a past conversation of user A, the Vector DB semantic search finds user A’s conversation summary and uses it as additional context — without user A’s personal data (summary is de-identified before storing in Vector DB).

6. Security Considerations for Agentic Systems

Three layers of protection (from the Q&A):

Layer 1 — Guardrails

Rules that restrict what data the LLM can access and what it can say:

Input guardrails: strip PII from user queries before they reach the LLM
Output guardrails: scan LLM responses for sensitive data, hallucinations, or harmful content before returning to user
Tool guardrails: limit which tools/APIs the LLM is allowed to call

Layer 2 — Proxy Layer

All LLM API calls pass through an enterprise proxy (similar to an ingress controller):

User query → [Proxy] → LLM API
Proxy checks:
  - Rate limits (prevent abuse)
  - Data classification (block PII from leaving premises)
  - Prompt injection detection (user trying to override system prompt)
  - Logging/audit trail of all LLM calls

This is directly analogous to an API Gateway (Kong) or ingress controller — it’s the network choke point for all LLM traffic.

Layer 3 — Trusted Cloud Provider + Enterprise LLM Deployment

For regulated industries (fintech, healthcare):

Use AWS Bedrock, Azure OpenAI, or Google Vertex AI — data does not leave the cloud provider your organisation already trusts
Self-hosted LLMs (on-prem or private VPC) for maximum data control
Separate the intent classifier (handles raw user queries) from the reasoning LLM (handles pre-processed context) — the reasoning LLM never sees raw user input

Prompt Injection Defence

Two-level LLM architecture:

Level 1 (no data access):
  Role: parse + structure the user query
  Has no access to your data systems
  User cannot "jailbreak" this level to steal data (it has none)

Level 2 (has data access):
  Role: reasoning over assembled context
  Never receives raw user input — only processed, structured context from Level 1
  Even if user includes malicious prompt in their query, Level 1 strips/normalises it

7. The DevOps Grafana AI Chatbot — Worked Example

The use case: An AI chatbot embedded in Grafana where DevOps engineers can ask natural language questions about system health.

Naive implementation (what you’d build first):

DevOps engineer: "Why is the error rate so high?"
  → Full query + massive system prompt ("you are a DevOps agent, read code, find errors...")
  → Sent to LLM
  → LLM decides: need to read the codebase
  → LLM calls GitHub API to fetch relevant files
  → Files + query sent back to LLM for analysis
  → LLM identifies error → writes suggested fix
  → LLM calls API to modify the file
Total: 15–25 seconds

Production implementation (optimised):

DevOps engineer: "Why is the error rate so high?"
  ↓
SLM Intent Classifier (< 50ms):
  Intent 1: metrics_query → route: Prometheus API (fetch current error rate data)
  Intent 2: log_search → route: Loki (fetch recent error logs)
  Intent 3: code_search → route: Vector DB of codebase embeddings (find relevant files)
  ↓
asyncio.gather([prometheus_query, loki_query, vector_search])  # parallel, ~200ms
  ↓
Context assembled: current metrics + recent logs + relevant code sections
  ↓
LLM Reasoning (with pre-cached DevOps system prompt): synthesise explanation (~3–5 seconds)
  ↓
Streaming response to DevOps engineer (user sees first tokens in ~1 second via TTFT optimisation)

What the Grafana AI bot can automate:

Surface root cause hypotheses from metrics + logs + code
Check if a similar error occurred before (vector search over historical incidents)
Suggest remediation steps based on past RCAs
Draft a Jira ticket or PagerDuty incident
Raise a PR with a suggested code fix (but human reviews — not fully automated)

What it cannot/should not automate:

Autonomously deploy code changes (too high risk — PR review is the gate)
Autonomously scale or restart production systems without human confirmation
Make decisions based on incomplete data without surfacing uncertainty

8. The E-Commerce Customer Support Agent — Worked Example

User query (multi-intent):

“I ordered shoes last week and haven’t received them. I also want to change my delivery address. Can I get a refund if they’re delayed?”

Intent breakdown by SLM:

Intent	Type	Route	Retrieval method
Order status	Structured lookup	SQL DB (orders table)	`SELECT * FROM orders WHERE user_id = :uid ORDER BY created_at DESC LIMIT 1`
Delivery address change	Write operation + policy check	SQL DB + Vector DB	DB update + policy document semantic search
Refund eligibility	Policy lookup	Vector DB	Semantic search: “refund policy delayed delivery”

Parallel execution:

task1 = asyncio.create_task(sql_db.query("SELECT status, est_delivery FROM orders WHERE user_id=..."))
task2 = asyncio.create_task(vector_db.search("refund policy if order delayed"))
task3 = asyncio.create_task(vector_db.search("delivery address change policy"))
results = await asyncio.gather(task1, task2, task3)  # all 3 run simultaneously

Memory tier decision — second turn:

“What about changing my PIN code?”

Order data already in Redis from Turn 1 → no new DB query
Redis read < 10ms → negligible latency
LLM receives cached order data + new query → responds almost instantly

Full architecture:

User Query → WebSocket (live input for speculative execution)
  ↓
[Speculative] SLM starts predicting intents at "I ordered shoes..."
  → Prefetch: order table query already running
  ↓
[On Enter] Full query → SLM finalises: 3 intents → 3 parallel tasks
  ↓
asyncio.gather(sql_query, vector_search_refund, vector_search_address)
  ↓
Results cached in Redis (for follow-up questions in this session)
  ↓
Context assembled → System prompt fetched from cache (intent-specific)
  ↓
LLM Reasoning → Streaming response
  ↓
User sees first word in < 1 second; full response in < 3 seconds

9. Architecture & Workflow Analysis

9.1 Naive Agent vs. Production Agent Comparison

NAIVE AGENT:
User Query
  ↓ (blocked until LLM decides)
LLM Orchestrator (3–8 seconds)
  ↓ (LLM decides tool calls)
Tool A call (2–5 seconds)
  ↓ (waits for Tool A)
Tool B call (2–5 seconds, based on Tool A result)
  ↓ (waits for Tool B)
LLM Synthesis (3–8 seconds)
  ↓
User Response
Total: 10–26 seconds | Sequential | Non-scalable | Unpredictable

PRODUCTION AGENT:
User Query (WebSocket streaming)
  ↓ (concurrent with typing)
SLM Intent + Route Classification (<100ms)
  ↓ (parallel branches)
  ├─ Redis Read (hot data from prior turns, <10ms)
  ├─ Vector DB Search (policy/knowledge docs, 100–300ms)
  └─ SQL DB Query (structured user data, 10–100ms)
         ↓ (all complete asynchronously)
  Context assembly + system prompt cache retrieval (<10ms)
         ↓
LLM Reasoning (3–8 seconds, but running on pre-assembled context)
         ↓ (streaming response — user sees first token early via TTFT opt.)
User sees response beginning within ~1 second
Full response in 3–5 seconds
Total perceived latency: 1–3 seconds | Parallel | Horizontally scalable

9.2 Memory Tier Decision Tree

When to use which tier:

Data needed NOW in this request (< 10ms):
  └─ Redis
      e.g., last 10 conversation turns, current session tool outputs

Data needed for search/similarity (100–500ms):
  └─ Vector DB
      e.g., policy documents, knowledge base, past incident summaries

Data with known structure/key (10–100ms):
  └─ SQL DB
      e.g., user profiles, order records, metadata

Data from prior sessions of SAME user (but not this session):
  └─ Redis (recent) + Vector DB (summary/embeddings)

Data that was already fetched this session (follow-up question):
  └─ Redis (already cached from first fetch)

10. Key Concepts Table

Concept	Definition	Why it matters
Naive agent (ReAct)	Sequential think→act→observe loop; LLM controls orchestration	Produces 20+ second responses; incompatible with production
DAG (Directed Acyclic Graph)	Task graph with no cycles; nodes = tasks, edges = dependencies	Enables parallelism; prevents infinite loops; deterministic execution
Intent classification	Categorising a user query into predefined action types	Replaces LLM routing (3–8s) with SLM routing (<100ms)
SLM (Small Language Model)	Lightweight model used for classification/routing	Millisecond latency; cheap; deterministic; appropriate for classification tasks
Three-tier memory hierarchy	Redis (hot) → Vector DB (searchable) → SQL (structured)	Enables horizontal scaling; appropriate latency per data type
Horizontal scaling	Adding more pods to handle more users	Breaks with RAM-stored state; requires external memory
Parallel execution	Running independent tasks simultaneously (`asyncio.gather`)	Reduces total latency to the slowest single task, not the sum
Speculative execution	Pre-fetching data while user is still typing	Hides fetch latency behind typing time; user perceives near-instant response
TTFT (Time To First Token)	Time from query submission to first response token reaching user	Primary UX metric for AI systems; target < 1 second perceived
System prompt caching	Pre-defining and caching the LLM instruction prompt per intent	Eliminates prompt construction latency; reduces token count
KV caching	Caching the Key-Value vectors computed during transformer attention	Reduces LLM inference time for repeated context (system prompts)
Conversation summarisation	Compressing long conversation history into a summary for LLM context	Reduces token count for long sessions; stays within context window
Vector DB	Database storing text as numerical embeddings; enables semantic search	”Find documents similar in meaning to this query” — faster and more accurate than keyword search
Embedding	Numerical representation of text as a vector	Enables similarity search (e.g., “refund policy” matches “money back guarantee”)
Prompt injection	User crafts input to override or bypass system prompt	Security risk for data-connected agents; mitigated by two-level LLM architecture
Guardrails	Rules constraining LLM input/output	Prevents PII leakage, hallucination propagation, harmful content
Proxy layer	Network intermediary for all LLM API calls	Logging, rate limiting, PII filtering, audit trail

11. DevOps-to-MLOps Career Path

From the session Q&A:

What DevOps engineers need to learn to enter MLOps:

What to learn	Why
Agentic system design (this session’s content)	Core MLOps engineering skill — how to build systems that use LLMs reliably
Transformer/GPT fundamentals (attention mechanism at a conceptual level)	Needed to understand KV caching, TTFT, context windows
Vector databases (Pinecone, Weaviate, pgvector)	New data tier in AI systems; analogous to knowing PostgreSQL
LLM serving infrastructure (vLLM, TGI, AWS Bedrock, Vertex AI)	The “Kubernetes” of LLMs — deploying and scaling model inference
MLOps-specific observability	Same tools (Prometheus, Grafana, tracing) but with LLM-specific metrics (token count, hallucination rate, intent accuracy)

What DevOps engineers do NOT need for MLOps:

Neural network mathematics (backpropagation, loss functions, optimisers) — that is the data scientist’s domain
Model training workflows (train/test/validation splits, overfitting detection) — that is the ML engineer’s domain
Deep learning research — that changes weekly; engineers follow it but don’t drive it

The distinction (presenter’s framing):

Data Scientist/ML Researcher: builds and trains models; understands the mathematics
ML Engineer: takes trained models and optimises them for inference
MLOps Engineer: deploys, monitors, and maintains ML systems in production; ensures reliability, latency, cost control
DevOps Engineer moving to MLOps: brings production reliability skills to the ML layer; adds the data fetching, orchestration, memory, observability, and scaling layer around the LLM

Suggested learning path (from presenter’s commitment to share):

Understand the Transformer attention mechanism (not the maths — the concept)
Build a simple agent with LangChain/LlamaIndex — then rebuild it with the DAG pattern
Deploy a model with vLLM or AWS Bedrock — observe how KV caching affects latency
Add the three-tier memory hierarchy to an existing application
Add structured logging and P50/P95 tracking

12. Interview Preparation

Q1. What is wrong with a naive LLM agent and why can’t it be used in production? Four failure modes: (1) Latency trap — sequential LLM→tool→LLM flow takes 10–20+ seconds; users abandon at >10 seconds. (2) State bottleneck — storing conversation history in application RAM breaks horizontal scaling; Kubernetes pod restarts lose all session state. (3) Tool cascading — each API call waits for the previous, creating exponential latency and unpredictable cost. (4) Observability gap — when the agent fails, you can’t tell whether the LLM hallucinated, the API failed, or the cache was stale.

Q2. Explain the three-tier memory hierarchy for production AI agents. Redis (hot memory, < 10ms): stores current session state — last N conversation turns, tool outputs from this request. Vector DB (warm memory, 100–500ms): stores document embeddings and past conversation summaries — enables semantic search over large corpora. SQL/Data Warehouse (structured, 10–100ms): stores user metadata, order records, session IDs, audit logs. The key rule: anything accessed in the same request twice goes into Redis on first access. Anything requiring semantic search goes in the Vector DB. Anything with a known key/ID goes in SQL.

Q3. How does a DAG orchestrator solve the tool cascading problem? A DAG (Directed Acyclic Graph) pre-defines which tasks are independent and can run in parallel, and which depend on each other. The LLM no longer decides the order of tool calls — the code-defined graph does. Independent intents (e.g., fetch order status AND fetch refund policy) are executed concurrently with asyncio.gather(). Total latency becomes the latency of the slowest single task rather than the sum of all tasks. The “acyclic” property prevents the agent from looping — the graph has a defined end state.

Q4. What is speculative execution in the context of AI agents? Starting data retrieval before the user has finished typing their query. A WebSocket streams the user’s live keystrokes to the SLM intent classifier. After the first few words, the classifier predicts the likely intent and begins fetching the relevant data (vector search, DB query) speculatively. By the time the user hits Enter, the data is already available. The LLM receives pre-assembled context and produces a response almost immediately. Works especially well when many users ask similar questions (speculative fetch hits the Redis cache).

Q5. What is TTFT and how do you optimise it? TTFT = Time To First Token — the delay between a user submitting a query and seeing the first word of the response. Three optimisations: (1) System prompt caching — pre-define system prompts for each intent+route combination; select the cached prompt in microseconds rather than constructing it dynamically. (2) KV caching — cache the key-value vectors computed during transformer attention for stable parts of the context (system prompts); reduces LLM compute on repeated calls. (3) Conversation summarisation — compress long conversation history into a summary; the LLM processes fewer tokens, reducing inference time and cost.

13. Cheat Sheet

Naive agent failure modes: Latency trap (20s+) | RAM state (breaks horizontal scaling) | Tool cascading (sequential) | Observability gap

User attention thresholds: <100ms instant | <1s smooth | 2–3s noticeable | >5s frustrated | >10s user leaves

Production agent architecture:

WebSocket (streaming input)
  ↓
SLM Intent Classifier (<100ms) → [intent + route]
  ↓ parallel asyncio.gather()
  ├─ Redis (< 10ms): hot session state
  ├─ Vector DB (100–300ms): semantic search over docs/embeddings
  └─ SQL DB (10–100ms): structured user data
  ↓
System prompt cache lookup (<1ms)
  ↓
LLM Reasoning (3–8s, but TTFT optimised)
  ↓
Streaming response to user (first token < 1s perceived)

Memory tier rules: Current session data → Redis | Large searchable docs → Vector DB | Structured user data → SQL

TTFT optimisations: System prompt caching | KV caching (in serving infra) | Conversation summarisation

Speculative execution: Start fetching data when user begins typing (not when they hit Enter)

Security layers: Guardrails (input/output) | Proxy (network-level) | Trusted cloud provider | Two-level LLM (Level 1 has no data access)

DevOps-to-MLOps path: Agentic system design + Transformer concepts + Vector DBs + LLM serving (vLLM/Bedrock) + MLOps observability

14. Gaps, Assumptions & What Was Promised

Session was conceptual-only: No code was shown (whiteboard session). The presenter promised to share:

Repositories with implementation examples
Blog posts/articles on the topics covered
Contact details (shared via WhatsApp at end)
A list of MLOps learning resources / books

Content the presenter explicitly did not cover (marked as “save for a later session”):

Specific tooling: LangChain/LlamaIndex/CrewAI in depth
LLM serving infrastructure (vLLM, TGI, AWS Bedrock, Vertex AI configuration)
Fine-tuning vs. RAG (Retrieval-Augmented Generation) trade-offs
Evaluation frameworks (how to measure hallucination rate, intent accuracy)
The second half of his slide deck (only the first half was covered due to time)

Assumptions and interpretation flags:

“React” in context of agents refers to the ReAct framework (Reason + Act), not React.js the frontend library
”TTFT” — presenter initially forgot and said “TDFT” before correcting to TTFT (Time To First Token)
The presenter improvised the whiteboard; the slide deck covered additional material not discussed (promised to share)
“SLM” (Small Language Model) — the presenter was clear this can be a classical ML classifier, not necessarily a miniature LLM; e.g., a fine-tuned BERT for intent classification, or even a simple rule-based system for well-defined intents
KV caching was described at a conceptual level (Q, K, V vectors in transformers); the implementation is typically a server-side setting in LLM inference frameworks, not something the application developer implements directly

Relationship to SRE Labs (Advanced Track) curriculum: This is an additional session, not part of the main project arc. It was announced at the end of the previous day’s session as an 11 AM IST call for those interested in AI. The main program continues with Helm (Phase 6) and Observability (Phase 7) of the Titan Grid project on the regular Sunday schedule.