How is an LLM system design interview different from classical system design?

Classical system design tests deterministic bottlenecks (network throughput, disk I/O, database sharding, caching). LLM system design tests non-deterministic probabilistic constraints: token economics, GPU memory bounds (KV cache), latency metrics like Time-to-First-Token (TTFT), hallucination evaluation guardrails, and dynamic model routing.

What are the most common metrics tested in LLM system design?

The key metrics interviewers look for are: Time-to-First-Token (TTFT < 800ms for conversational UI), Inter-Token Latency (ITL < 40ms for human reading speed), tokens per second per GPU (throughput), retrieval Recall@K for RAG, cache hit ratio for semantic caches, and cost per 1,000 successful queries.

How does semantic caching work in production?

A semantic cache converts incoming prompts into vector embeddings and checks a low-latency vector store (like Redis or Qdrant) with an exact cosine similarity threshold (typically 0.92-0.96). If a match is found, the pre-computed LLM completion is returned in under 20ms, bypassing the model inference layer and slashing API costs by 30-65%.

Why is hybrid search (Dense + BM25) mandatory in enterprise RAG designs?

Vector-only embeddings struggle with exact identifier queries (part numbers, error codes, legal statute numbers, drug names). BM25 handles exact keyword lexical matching, while vector embeddings capture semantic intent. Reciprocal Rank Fusion (RRF) and a cross-encoder reranker combine the two to deliver 95%+ precision.

LLM System Design Interview Cheatsheet

Q: How does semantic caching work in production?

A semantic cache converts incoming prompts into vector embeddings and checks a low-latency vector store (like Redis or Qdrant) with an exact cosine similarity threshold (typically 0.92-0.96). If a match is found, the pre-computed LLM completion is returned in under 20ms, bypassing the model inference layer and slashing API costs by 30-65%.

Q: Why is hybrid search (Dense + BM25) mandatory in enterprise RAG designs?

Vector-only embeddings struggle with exact identifier queries (part numbers, error codes, legal statute numbers, drug names). BM25 handles exact keyword lexical matching, while vector embeddings capture semantic intent. Reciprocal Rank Fusion (RRF) and a cross-encoder reranker combine the two to deliver 95%+ precision.

Quick answer: In 2026, the traditional System Design interview ("Design Twitter" or "Design a URL Shortener") has been superseded by LLM System Design at Meta, OpenAI, Anthropic, Google, and top AI startups. Rather than testing relational sharding and microservice queues, interviewers evaluate seven core building blocks: high-throughput gateway orchestration, semantic caching, hybrid RAG retrieval pipelines, TTFT streaming optimization, guardrail and injection defense, multi-agent loop termination, and continuous offline/online evals.

When interviewing for Senior AI Engineer, Staff ML Engineer, or AI Solution Architect positions in 2026, you will inevitably face an end-to-end LLM System Design round. Candidates who treat LLMs like standard stateless microservices fail quickly.

LLM systems have unique constraints that break classical assumptions:

Extreme Compute Asymmetry: A traditional API handler takes 5ms of CPU; an LLM generation takes 800ms–8,000ms of high-cost H100/A100 GPU compute.
Stateful KV Caches: High concurrency is bounded by GPU High Bandwidth Memory (VRAM), not network bandwidth.
Probabilistic Failure Modes: Systems do not simply return 200 or 500; they return syntactically valid yet factually hallucinated, toxic, or non-deterministic outputs.

Below is your complete whiteboard blueprint: the master gateway architecture and 7 battle-tested system designs you can deploy directly in your interview loops.

The Canonical Enterprise LLM Gateway Architecture

Every production LLM interview question requires a variation of this end-to-end gateway:

flowchart TD Client["Client / SDK"] --> Gateway["API Gateway
(Rate Limiting, Auth, Org Quotas)"] Gateway --> Cache{"Semantic Cache
(Redis / Vector Store)"} Cache -->|Cache Hit < 25ms| Client Cache -->|Cache Miss| Guard["Prompt Guardrails
(Llama-Guard / Injection Check)"] Guard --> Router["Dynamic Model Router
(Complexity Classifier)"] Router -->|Simple Query| Cheap["Fast Model
(Gemini Flash / Llama-3.1-8B)"] Router -->|Complex Reasoning| Frontier["Frontier Model
(GPT-4o / Claude 3.5 Sonnet)"] Cheap & Frontier --> Stream["Streaming Evaluator
(Chunked PII & Hallucination Filter)"] Stream --> Client Stream -.-> AsyncLog["Async Data Lake
(Traces, Evals, E2E Latency)"]

< 800ms Target TTFT (Time to First Token)

< 35ms Target ITL (Inter-Token Latency)

35% – 60% Typical Semantic Cache Hit Rate

99.9% Gateway Availability SLA

7 Production LLM System Design Questions

Architecture 1

Design a High-Throughput LLM Gateway Serving 10,000 QPS

The Prompt: "Design an internal LLM Gateway for a 50,000-person enterprise that routes requests across multiple vendors (OpenAI, Anthropic, Bedrock, self-hosted vLLM) with rate limiting, org chargebacks, and semantic caching."

Key Components to Draw:

Token-Bucket Rate Limiter per Tenant: Track both Requests Per Minute (RPM) and Tokens Per Minute (TPM) in Redis using sliding window logs.
Semantic Caching Layer: Generate embeddings using a lightweight local model (e.g. all-MiniLM-L6-v2, ~15ms). Query a vector index with a strict cosine threshold (0.95). Return the cached answer if matched, saving 90% latency and 100% downstream API cost.
Circuit Breaker & Fallback Chain: If Claude 3.5 Sonnet returns 429 or 503, automatically fallback to GPT-4o, then to self-hosted vLLM Llama-3.1-70B on internal Kubernetes clusters.
Asynchronous Tracing & Telemetry: Log full prompts, completions, token usage, and latency to an Apache Kafka / OpenTelemetry queue. Never block the user-facing response on database persistence.

Architecture 2

Design an Enterprise Hybrid RAG System Over 50 Million Documents

The Prompt: "Design a search-and-answer assistant over millions of corporate PDFs, Confluence pages, and Jira tickets. The system must never quote stale permissions or hallucinate internal policies."

Key Components to Draw:

Structure-Aware Document Chunking: Do not use naive 500-character fixed splits. Parse PDF hierarchies with layout-aware tools (e.g. Unstructured / PDFMiner). Preserve markdown tables as single self-contained chunks with parent document metadata.
Hybrid Search (Dense + Lexical): Run parallel retrieval: BM25 for exact part numbers, acronyms, and employee IDs; dense vector search (e.g. BGE-large) for conceptual questions.
Reciprocal Rank Fusion (RRF) & Cross-Encoder Reranker: Merge candidate sets using RRF. Pass the top 50 candidates through a cross-encoder reranker (e.g. Cohere Rerank or BGE-Reranker-large) to select the final top 5 context chunks.
Document-Level Access Control (ACL) Filtering: Filter vector retrieval using metadata pre-filtering (user_groups IN document_read_permissions) so users never see context from unauthorized departments.

Architecture 3

Design a Low-Latency AI Code Completion Engine (Copilot)

The Prompt: "Design a code completion system for an IDE extension with latency strictly under 250ms end-to-end, serving 100,000 concurrent developers."

Key Engineering Signals:

Speculative Decoding & Prefix Caching: Pre-cache the current file's AST (Abstract Syntax Tree) and import tree in GPU KV cache using RadixAttention (vLLM / SGLang).
Fill-in-the-Middle (FIM) Architecture: Format prompts with explicit <PRE>, <SUF>, and <MID> tokens to allow the model to condition on both preceding and succeeding code lines.
Client-Side Debounce & Cancellation: If the user types another character while an inference request is in transit, immediately send an HTTP/2 RST_STREAM cancellation frame to the server to free GPU compute.

Architecture 4

Design an Autonomous Multi-Agent Customer Support Escalator

The Prompt: "Design an agentic support system that resolves tier-1 tickets by calling internal database APIs, reading documentation, and initiating refunds up to $100."

Key Engineering Signals:

Deterministic State Machine over Free-Form ReAct: Use graph-based orchestration (LangGraph / StateGraph) where agent nodes have bounded execution limits rather than open-ended recursive loops.
Human-in-the-Loop (HITL) Guardrail Checkpoints: For financial actions > $100 or account cancellations, freeze agent state and emit a webhook to an internal support dashboard. Resume execution only upon human OAuth approval.
Loop Termination Safeguards: Hard limit tool executions to a maximum of 6 steps per ticket; if uncompleted, gracefully downgrade to human triage with full execution trace logs.

Architecture 5

Design a Continuous Offline and Online LLM Evaluation Pipeline

The Prompt: "How do you know if your new prompt or fine-tuned model is better before deploying it to production? Design the automated testing and drift monitoring harness."

Key Engineering Signals:

Golden Evaluation Dataset: Version 1,000 curated prompt-response pairs with ground truth assertions and edge cases stored in Git-LFS.
LLM-as-a-Judge with Dual-Model Cross-Scoring: Use two independent judge models (e.g., GPT-4o and Claude 3.5 Sonnet) with randomized presentation order to eliminate position bias.
Shadow Deployments & RAGAS Metrics: Shadow 5% of production traffic through the challenger pipeline and compute Faithfulness, Answer Relevance, and Context Precision in real-time.

Architectures 6 & 7: Knowledge Graphs & Edge Serving

Architecture 6

Real-Time Unstructured Document to Knowledge Graph Extraction

The Core Challenge: Extracting entities, relationships, and temporal events from 10,000 clinical trials or SEC filings daily into a Neo4j / Amazon Neptune graph with schema constraints, entity deduplication, and relationship verification.

Architecture 7

Air-Gapped Multi-Tenant Private VPC Model Serving with vLLM

The Core Challenge: Serving quantized models (AWQ/FP8) on Kubernetes across AWS GovCloud / on-premise H100 clusters. Implementing dynamic batching, PagedAttention, and automated GPU node scaling while guaranteeing zero internet telemetry egress.

System Design Trade-Off Matrix for the Whiteboard

Design Dimension	Option A	Option B	When to Choose Which
Inference Engine	Managed API (OpenAI/Anthropic)	Self-Hosted vLLM / SGLang	Choose API for fast time-to-market; choose vLLM for data sovereignty, strict PII, and > 50M tokens/month where GPU amortizes cheaper.
RAG Retrieval	Dense Vector Only	Hybrid (BM25 + Dense + Rerank)	Always choose Hybrid in enterprise production. Dense alone drops recall on serial numbers, SKU codes, and legal citations.
Agent Orchestration	Autonomous ReAct Loop	Deterministic State Machine	Choose State Machines for enterprise workflows with compliance/financial liability; choose ReAct only for exploratory open-ended research tools.
Evaluation Strategy	Pure Human Review	LLM-as-a-Judge + Golden Testbed	Human review fails to scale beyond 200 samples. Use LLM-as-a-judge for continuous CI/CD gating, calibrated by human audits on 2% sample sets.

Neil Dave

AI Solution Architect & Generative AI Engineer · Bangalore, India

LinkedIn · Substack · Book a Strategy Call

Get Advanced AI System Design in Your Inbox

No hype. Real architectural breakdowns, latency formulas, and production code.

Subscribe to Newsletter →

Article

The LLM System Design Interview Cheatsheet: 7 Production Architectures (2026)

The Canonical Enterprise LLM Gateway Architecture

7 Production LLM System Design Questions

Architectures 6 & 7: Knowledge Graphs & Edge Serving

System Design Trade-Off Matrix for the Whiteboard