Requirements, Scoping & Feasibility Analysis
The single most expensive mistake in AI system design is applying a probabilistic engine to a deterministic problem — this phase exists to catch that before any architecture work starts.
Before drawing a single box, decide honestly whether this problem even needs an LLM — and write down the cost and latency numbers that will constrain every choice after this one.
Most production AI failures trace back to skipping this phase — teams reach for an LLM because it's exciting, not because the problem demands probabilistic reasoning. Everything downstream — cost, latency, reliability — gets harder to fix later than to prevent here.
phase objectives
Deterministic vs. probabilistic classification
Does this task have one correct, computable answer given the inputs? "What's this customer's account balance?" is a database query — deterministic. "Summarize this contract's risk clauses" has no single correct string — that's where LLMs earn their cost. Get it wrong either direction and the system either becomes an unreliable, expensive lookup table, or breaks the moment inputs vary from what was anticipated.
Token economics, ROI modeling & NFRs
What does one interaction cost — input plus output tokens (output runs 3–5x pricier) times expected volume? Does the system beat the alternative on cost? TTFT, throughput and latency budget need to be derived from real user expectations before architecture decisions get made — a 500ms budget eliminates most multi-hop agentic RAG designs before a box gets drawn.
system components & topologies
The latency budget waterfall
Instead of one number ("under 3 seconds"), the budget is allocated across every hop. Breaking it down at design time means that when the SLA is violated in production, the team knows exactly where to look — without it, debugging latency becomes archaeology.
tooling & tech stack
Cost modeling in a spreadsheet seeded with real provider pricing; validated post-launch.
Requirements tracking plus a short markdown Architecture Decision Record — the answer, six months later, to "why didn't we just fine-tune?"
Back-of-envelope throughput math on paper, validated later with real load tests.
trade-offs & challenges
The "let's use an LLM" trap
99% correct isn't acceptable for arithmetic that used to be a formula. Mitigation: require written justification any time probabilistic logic touches a compliance-sensitive computation.
TTFT vs. throughput tension
Streaming holds a connection open for the whole generation, straining the pool under concurrency. Mitigation: multiplexed protocols like SSE over raw long-lived sockets.
Cost model blind spots
Agentic/retry flows have a fatter cost tail — budget on P90, not average. Output tokens, priced 3–5x higher, make prompt-only estimates look artificially cheap.
Architectural Decision Frameworks & Model Selection
Phase 1 told you whether to use an LLM. Phase 2 commits you to which technique and which model — and to a decision-making process you can defend later, not "we picked GPT-4 because everyone does."
Climb the technique ladder one rung at a time — pick the minimum viable technique that meets the NFRs, and force a justification before reaching for the next, more expensive rung.
phase objectives
Build vs. Buy, and technique selection
"Train from scratch" is almost never right for application teams — the real decision is API vs. self-hosted OSS, hinging on data sensitivity, cost at scale, and latency control. Teams reach for fine-tuning because it feels like "real ML work," when prompt engineering or RAG would solve the actual problem more cheaply.
Multi-model routing strategy
Not every query deserves the most expensive, most capable model. A routing policy decides, per request, which model handles it — based on task complexity, context window, and cost target. This is a deliverable in its own right, even if implemented as a simple classifier.
system components & topologies
The decision matrix
Score the problem 1–5 across five axes. Static facts + tone shift → Prompt Engineering. External/frequently-changing knowledge → RAG. Multi-hop reasoning over interconnected entities → GraphRAG. Durably changing behavior or format → Fine-tuning/LoRA.
tooling & tech stack
Proprietary APIs — frontier capability, managed reliability, zero infra overhead.
Self-hosted OSS, served via vLLM, TGI, or Ray Serve — full data residency, own the ops burden.
Parameter-efficient fine-tuning, dramatically cheaper than full fine-tuning.
Model-agnostic routing gateways behind a unified API.
Graph databases and entity/relationship extraction for GraphRAG.
trade-offs & challenges
RAG vs. fine-tuning confusion
The rule of thumb worth memorizing: RAG for facts, fine-tuning for format/voice/task-specialization. Fine-tuning to "teach the model the product catalog" is almost always a RAG problem in disguise.
Vendor lock-in risk
Hardcoding to one provider's quirks means switching later means rewriting application logic. Mitigation: abstract behind a gateway with provider-agnostic schemas.
Router misclassification
A router sending a complex query to a cheap model doesn't error — it silently returns a worse answer. Mitigation: default to the larger model when confidence is low, and validate with shadow-routing.
Data Engineering, Vectorization & Governance
Building the retrieval substrate that grounds the system in truth — and enforcing access control at the retrieval layer, not just the UI.
Most "the AI gave a wrong answer" bugs don't come from the model — they come from bad retrieval feeding it bad context. Filter access before scoring, never after.
phase objectives
Ingestion — idempotent, incremental, lineage-tracked
Idempotent: re-running the pipeline shouldn't duplicate vectors. Incremental: the whole corpus shouldn't need reprocessing on every change. Lineage-tracked: every chunk traces back to its source, version, and ingestion timestamp.
Chunking + embedding tuned to domain structure
A legal contract, a Slack export, and an API reference have completely different natural units of meaning — getting this wrong is one of the most common causes of vague, fragmented RAG answers.
Hybrid retrieval, tuned for recall and precision
Vector search alone misses exact-match cases (a SKU, an error code); keyword search alone misses semantic paraphrases. Combine both, plus graph traversal when Phase 2 calls for it.
system components & topologies
Governance enforcement point — pre-filter, not post-filter
ACL/metadata filters apply inside the vector DB query itself, restricting which vectors are even scored — never as a step run after retrieval to strip out unauthorized results. This is the single most consequential point in the whole phase.
Hybrid search topology
Query → [BM25/keyword] + [dense vector] + [optional graph traversal] → fusion (RRF or cross-encoder reranker) → top-K to the model. A cross-encoder is more accurate but too slow for the whole corpus, so it only reranks the cheap first pass's top candidates.
tooling & tech stack
Vector databases — pgvector when Postgres is already running and fewer moving parts is worth it.
Ingestion orchestration and layout-aware parsing of PDFs, Word docs, HTML.
Managed or self-hosted embedding models.
PII masking at ingestion, and policy-as-code for ABAC rules evaluated at query time.
trade-offs & challenges
Chunking is a tuning knob, not a one-time decision
Too-small loses context; too-large dilutes the embedding. Mitigation: run chunk-size sweeps against the eval set, don't guess a number that "seemed reasonable."
RBAC-at-retrieval is the #1 RAG security gap
Many teams enforce access control only at the app layer, while the vector DB has no ACL awareness — a crafted prompt can surface chunks the user shouldn't see.
PII leakage through retrieval
Masking a customer's name before embedding can make that chunk unfindable later. Mitigation: mask for indexing, but gate the unmasked original behind the same ABAC policy.
Agentic Workflows, Memory & Orchestration
Choosing an agent topology proportional to task complexity — and making tool-calling deterministic and safe.
Default to the simplest agent topology that could work. Every agent-to-agent handoff is a place context gets lost, latency accumulates, and cost multiplies.
phase objectives
Topology proportional to task complexity
"Multi-agent" sounds sophisticated and is tempting, but every hand-off is a place things degrade. Start with the simplest topology; only add agents when the task genuinely decomposes into distinct roles.
Deterministic, safe tool-calling
The decision to call a tool can be probabilistic, but its execution should be as deterministic and safe as any other backend operation — validated schemas, idempotent side effects.
Short-term vs. long-term memory, kept distinct
Short-term: coherence within a session. Long-term: preferences and recurring facts across sessions. Conflating them produces a memory system that's neither a good buffer nor a good store.
tooling & tech stack
Orchestration — state-machine or role-based conversation, or a bare Temporal/Step Functions state machine.
Short-term session memory vs. dedicated long-term semantic memory stores.
Schema validation enforcing strict I/O contracts on every tool definition.
trade-offs & challenges
Multi-agent overhead
N agents ≈ N× the latency and cost of a single well-designed loop. Default single-agent; escalate only for genuine role separation or parallelism.
Runaway loops
Infinite retries or a planner/critic ping-pong. Mitigation: hard max-iteration count and a token/cost ceiling, with a graceful hand-off to a human.
Memory pollution
Long-term memory accumulates stale, contradictory facts. Mitigation: only persist explicitly confirmed facts, and define a conflict-resolution rule on read.
Guardrails, Security & Evaluation
Making the system safe to expose to real users, and measurable enough to improve with confidence — in CI/CD, not ad hoc.
The model's context window is a new attack surface — it can be persuaded by anything in it, including a retrieved document it never should have trusted as an instruction.
phase objectives
Defend against the OWASP Top 10 for LLMs
In a traditional web app the backend logic is trusted; in an LLM system the model itself is a decision-maker persuadable by anything in its context — including a retrieved document or a tool's output.
Real-time input/output guardrails
A layer inspecting both what goes in and what comes out, in real time, in the request path — fast enough not to blow the Phase 1 latency budget.
A repeatable evaluation harness in CI/CD
Turning "does this feel better?" into a number that runs automatically on every change — without it, every prompt tweak is a gamble evaluated by vibes.
tooling & tech stack
Programmable rails and purpose-built safety classifiers.
RAG-triad eval metrics and CI-friendly prompt regression testing.
An LLM vulnerability scanner, analogous to a security scanner for a web app.
trade-offs & challenges
Guardrail latency tax
Stacking classifier calls adds to TTFT. Mitigation: cheap deterministic checks inline, heavier LLM-based guardrails only for cases the cheap pass flags as ambiguous.
LLM-as-Judge is not ground truth
A judge model favors longer, more confident-sounding answers regardless of correctness. Mitigation: periodically calibrate judge scores against human-labeled sets.
Eval sets go stale
A launch-time golden set stops representing real traffic as the corpus and users shift. Mitigation: mine production logs for hard cases and fold them in on a cadence.
LLMOps, Infrastructure & Resilience
Operationalizing for production-scale reliability and cost control — treating prompts, models, and indexes as first-class deployable artifacts.
This is the phase that separates a working prototype from something that stays up at 3am when a provider has an incident.
phase objectives
A model gateway
Abstracts providers, handles failover, enforces rate limits. Without it, provider-specific logic gets scattered across the codebase, and switching providers means touching every call site.
Caching layers
Prompt caching reuses an exact repeated prefix — cheap, precise. Semantic caching reuses an answer to a similar-but-not-identical question — more powerful, riskier.
CI/CD for prompts, models & indexes
A prompt change is a deployment, with the same rigor as a code deployment — versioned, tested against regressions, rollback-capable.
tooling & tech stack
Gateway products, or AWS Bedrock's built-in cross-region failover for an AWS-native stack.
Exact and semantic caching, alongside native provider prompt-caching APIs.
Eval suites as a merge gate turning Phase 5's harness into an enforced deployment gate.
Self-hosted serving with GPU autoscaling — or managed endpoints trading control for less ops burden.
trade-offs & challenges
Failover correctness
Failing over mid-conversation can silently change tone or safety posture. Mitigation: only failover between models validated as equivalent-tier, and log every event.
Cache staleness vs. hit rate
Semantic caching can serve an outdated answer. Mitigation: bind TTL to the data-freshness SLA, and wire invalidation to source-update events.
Rate-limit retry storms
Uniform retry-with-backoff can synchronize a burst right as a provider recovers. Mitigation: circuit breakers, jittered backoff, load-shedding.
Monitoring, Observability & Feedback Loops
Closing the loop — knowing what the system is doing in production, catching degradation before users complain, feeding real signal back into evaluation.
User complaints are a lagging indicator. By the time enough people notice, a meaningful volume of degraded responses has already shipped.
phase objectives
Full traceability, request by request
Not "is it up" — but "is this individual request behaving correctly, and can I reconstruct exactly what happened when something goes wrong?"
Detect degradation before users complain
This is what separates reactive operations from proactive ones — catching the signal earlier than a support ticket.
Feed real signal back into evaluation
Without a return path, observability is just a read-only window; with it, the system actually improves over time from real usage.
tooling & tech stack
Trace/span-level observability purpose-built for LLM call chains.
Vendor-neutral instrumentation — no lock-in to one platform's SDK.
Embedding and data drift monitoring.
Feedback pipelines joined to trace IDs, feeding a human-labeling queue.
trade-offs & challenges
Drift alerts without action paths
An alert with no owner and no response trains people to ignore alerts. Mitigation: every drift type maps to a documented runbook with a named owner.
Feedback loop bias
Explicit feedback overrepresents power users and the highly frustrated. Mitigation: weight implicit signals more, and run periodic blind quality sampling.
Attribution in multi-agent systems
Pinpointing which component caused a regression needs span-level granularity — flat request logging doesn't have the resolution.
Enterprise Knowledge & Action Assistant
Where everything from Phases 1–7 gets applied to one concrete system, so the decisions can be seen interlocking rather than existing as seven separate frameworks.
One assistant, three execution paths: RBAC-scoped document Q&A (RAG), live data lookups via SQL tool-calling, and real workflow triggers — forcing every phase's decisions to interact with each other.
applying the seven phases
Employee (UI) ── HTTPS ──▶ API Gateway (AuthN/AuthZ) ──▶ Intent Router
│
┌───────────────────────────────┼───────────────────────────────┐
▼ ▼ ▼
RAG Retriever Node SQL Tool-Calling Node Workflow Trigger Node
hybrid search + RBAC filter read-only role + schema idempotent + confirmation
│ │ │
▼ ▼ ▼
Vector DB (ACL-tagged) Live DB (read replica) Workflow Engine (Temporal)
└───────────────────────────────┴───────────────────────────────┘
▼
Model Gateway
failover · prompt + semantic cache
▼
LLM (routed by complexity tier)
▼
Output Guardrails
faithfulness · PII/toxicity · format
▼
Response → Employee
▼
Observability & Feedback Loop ──▶ back into Phase 5 eval set
why this holds up under interview scrutiny
A staff-level interviewer isn't scoring whether a vector DB box and an LLM box got drawn. They're checking whether every arrow can be explained — why the SQL path hits a replica, why RBAC filtering happens inside the vector DB query, why workflow triggers pause for confirmation while RAG answers don't. Each is a specific failure mode from one of the seven phases; the diagram is a compressed record of seven phases' worth of "here's what goes wrong, and here's the decision that prevents it."
Open to senior/lead remote roles
Especially where enterprise .NET meets AI agent engineering — reach out, or see the Microsoft Agent Framework build this playbook draws on.