Every serious decision in an AI architecture is a decision about where the boundary sits between what the model decides and what the system decides.
Watch the walkthrough More resources
The worked example, assembled. Purple boxes are system-controlled components while magenta is where judgement is delegated to a model. Note that learned models appear on both sides; for example an embedding or a reranker is a model inside a controlled pipeline. The question is not whether a component contains ML, but whether we are relying on model judgement for something we need to guarantee. Everything along the bottom runs across every path.
The same architecture, viewed by what it stores and what it depends on.
There are three bands pictured below: the synchronous request path across the top, which every customer message travels through under a latency budget; the data ingestion lane and the external dependencies below it, neither of which runs inside a request; and the four stores along the bottom.
Each store has a different consistency guarantee, a different lifetime and different behaviour when it fails. Treating them as one undifferentiated database is how a stale index gets mistaken for a source of truth, and how a cache quietly starts serving something it shouldn't.
| Establish | Why it changes the design | In the bank example |
|---|---|---|
| Which kinds of request exist | Knowledge, private data and state-changing actions need entirely different machinery. Sorting them is the first architectural act, and everything downstream follows from it. | Policy questions, balance lookups, and freezing a card |
| What is authoritative for each | Retrieval is approximate by design. It finds things that are similar, which is the wrong property for anything with one correct answer. | Documents for fees, the core banking system for balances, never the index for both |
| Freshness tolerance per source | Different sources tolerate different staleness, and that decides ingestion strategy and caching. | Policy within minutes, balances in real time |
| Request amplification | User traffic and model traffic are not equal. One message can become a routing call, embeddings, a rerank, several tool calls and a generation. Your architecture sets the ratio. | 10k concurrent users is nowhere near 10k model calls |
One message does not mean one model call. Choose a request and a set of design decisions, then step through what actually happens and what it costs.
Latencies are an illustrative budget, not measurements.
Each sits on the side of the boundary that owns its decision. System-controlled components enforce what the application has to guarantee. Model-judgement components interpret or generate, where some uncertainty is tolerable. The question is not whether a model is involved but whether a model is being asked to decide something that needs a guarantee.
Select a component below for details on what it decides, how it fails, and what it looks like in the worked example.
Capacity planning follows the execution graph, not user concurrency because user traffic and model traffic are different numbers. Every tier below can be the one that gives way first, and each has a different correct response. Scale what you own; protect what you don’t.
| Limit point | Yours to scale? | What to do when it binds |
|---|---|---|
| Edge rate limits | Yours | First line of admission control. Cheapest place to say no, and the only one that protects everything behind it. |
| Application concurrency | Yours | Stateless services scale horizontally. Cap in-flight requests per instance so queueing is explicit rather than emergent. |
| Queue and admission control | Yours | Absorb bursts, shed load deliberately when the queue grows past what the latency budget allows. A fast honest refusal beats a thirty-second hang. |
| Model provider quotas | Theirs | Not scalable on demand. Route cheaper work to smaller models, degrade optional calls first; reranking before generation, never authorisation. |
| Tool and API concurrency | Shared | Connection pools and per-tool concurrency caps, so one busy tool can’t starve the rest. |
| Downstream banking systems | Theirs | The one you most need to protect. Circuit breakers and backpressure upstream, so your traffic spike doesn’t become their incident. |
The order matters. Shedding at the edge costs one refused request; shedding at the banking API costs everyone already mid-conversation.
Naming components is the easy half. These are the forks where a design is actually judged.
| Decision | Options | What tips it |
|---|---|---|
| How to route | Rules, a trained classifier, embedding similarity, or a small model | Rules add negligible cost and latency and are fully deterministic, but they get brittle as language varies. A classifier is cheap and scales. A model is flexible and costs you latency on every request. Most real systems are hybrid: deterministic for obvious intents, model for ambiguous free text. |
| One agent or several | Single orchestrator with many tools, or separate domain agents | Splitting buys focused prompts, smaller tool sets and separate ownership. It costs model calls, latency, orchestration and much harder debugging. Split when domain separation gives real operational benefit. |
| Keeping knowledge current | Scheduled reprocessing, or event-driven on source change | Scheduled is trivial to build and your staleness window is your timer. Event-driven reacts to change and costs you a pipeline. Anything with a tight SLA needs the second. |
| Retrieval strategy | Vector only, keyword only, or hybrid with a reranker | Embeddings handle meaning and are weak on exact strings like product names, codes, policy numbers. Where exact terminology matters, hybrid plus reranking is worth the extra hop. |
| Model allocation | One strong model everywhere, or tiered by task | One model is simpler to operate and reason about. Tiering cuts cost and latency substantially but multiplies the surface you have to evaluate. Classification almost never needs a frontier model. |
| Provider failure | Evaluated fallback, degrade, or fail closed | A fallback you have never evaluated is an untested deployment that fires on your worst day. For high-stakes paths, refusing is often safer than silently switching to different behaviour. |
| What to remember | Session only, or persistent per-customer memory | Persistence buys personalisation and takes on consent, retention, deletion and residency. In regulated settings the default should be small, explainable and deletable. |
| Caching | None, exact-match, or semantic | Cache by data sensitivity rather than by regeneration cost. Semantic caching needs real care wherever two similar-looking questions have different correct answers. |
Each of these should have an answer before launch, not during an incident.
| Failure | Response | What the customer sees |
|---|---|---|
| Model provider is down | Fail over to an evaluated alternative; if none exists for that task, degrade | Slower, or a narrower set of things it can help with |
| Vector index unavailable | Serve static FAQ content; suppress anything that needs retrieval | Common answers still work, specific policy questions don't |
| Banking API unavailable | Refuse to answer from memory or cache | Live account information is temporarily unavailable - never a guessed number |
| Tool times out | Return an explicit failure to the model rather than silence | A clear "couldn't complete that", not a hallucinated confirmation |
| A dependency fails repeatedly | Circuit breaker opens, fail fast, recover deliberately | Fast honest failure instead of a thirty-second hang |
| Knowledge base is stale | Query the source directly for time-sensitive fields; alert on the staleness SLA | Nothing |
| Retrieved document contains instructions | Treat retrieved content as untrusted; authorisation and tool limits hold regardless | Possibly a worse answer. Never a wider permission |
| Latency jumps | Read the distributed trace, find the stage, fix that stage | Depends how fast you read the trace |
| Traffic spikes | Scale stateless services, protect downstream dependencies, shed load before the banking API feels it | Queuing, then graceful degradation, not a cascade |
Reading a balance and moving money are not the same risk and shouldn't carry the same controls. Before any tool execution, it's useful to ask what has to be true before it's allowed to start.
| Tier | Examples | Controls before it runs |
|---|---|---|
| Read | Balance, transactions, card status, loan details | Authenticated identity, permission check, access logged |
| Low-risk write | Update contact preference, flag a transaction for review | All of the above, plus argument validation and an idempotency key |
| High-risk write | Freeze a card, dispute a charge, move money | All of the above, plus a trusted confirmation the model cannot satisfy on the customer’s behalf. Step-up authentication, transaction signing or a human operator all qualify. In this design it is an application control outside the conversation, so the model can neither generate it nor read a reply as consent |
A “two-second response” is two numbers, not one. Perceived latency is how long until something useful appears on screen; completion latency is how long until the task is finished. Streaming reduces percieved latency but does nothing to completion latency.
The budgets below are illustrative not measured. Capturing latency budgets makes the architectural cost visible early. Six sequential model calls, for example, loses either budget before any code is written.
| Stage | Budget | Note |
|---|---|---|
| Gateway and identity | ~60 ms | Fixed cost on every request |
| Routing | ~40 ms | Negligible on rules, real on a small model |
| Retrieval | ~80 ms | Can run in parallel with a data lookup |
| Reranking | ~50 ms | Skippable for high-confidence single-hit queries |
| Banking API | ~150 ms | Not yours to optimise |
| Generation - first token | ~300–600 ms | Everything above plus this is what the customer waits on |
| Generation - remainder | ~0.8–1.4 s | Streams while they read. Counts against completion, not perception |
On these figures, perceived latency lands near 600 ms to a second, and completion nearer 1.5 to 3 seconds.
For actions, assistant latency and business-operation latency can also be viewed differently: a dispute or a payment can legitimately take far longer than the answer describing it, and the assistant should say so rather than hold a connection open.
A service can return a 200, in under a second, with a fluent answer that's wrong. Every monitor green and the system broken. So evaluation runs per layer, not just at the end.
| Layer | Measures | The question it answers |
|---|---|---|
| Retrieval | Recall@K, MRR, NDCG | Did the right evidence come back, and was it near the top? |
| Generation | Groundedness, faithfulness, relevance, citation accuracy | Is the answer actually supported by what we retrieved? |
| Tool use | Selection accuracy, argument accuracy, task completion, unnecessary calls | Right tool, right arguments, and nothing it didn't need to touch? |
| Routing | Intent accuracy, false positives onto privileged routes, escalation recall | Did it send this to the right place, and does it over-reach? |
| Safety | Injection resistance, unauthorised access attempts, PII leakage, policy compliance | Does the boundary hold when someone pushes on it? |
| End to end | Task success, containment, escalation rate, recontact rate | Did the customer get what they came for? |
| Operations | Latency, availability, error rate, throughput, cost | The ordinary questions, which still apply |
| Outcome | Cost per successfully resolved task | A cheap model that makes people ask three times isn't cheap |
Every metric above needs slicing, because there might be an imbalance. The cases a system handles worst are almost never the cases it sees most.
| Slice | Tool selection accuracy | Why it diverges |
|---|---|---|
| Overall | 98.7% | Dominated by high-volume, low-stakes requests |
| Card freeze | 99.9% | Narrow phrasing, heavily represented in the suite |
| Disputed transaction | 94.1% | Ambiguous language, overlapping tools, far rarer |
| Vulnerable customer flows | 90.8% | Least represented, highest consequence, hardest to get right |
The headline figure is fine but the system is not. Slice by request type, by risk tier, by language, by channel, and by the customer circumstances you have a duty of care toward, then set a threshold per slice rather than one number for everything.
This resource is shared to do three things: pass on what I’ve learned, invite scrutiny from others doing the same work, and contribute to the wider conversation about what production-grade actually means for these systems.
It takes positions rather than listing options. Where the boundary sits between system and model. What to degrade first under load. When to fail closed. If you’d draw one of those lines differently, or you’ve run a system where one of them broke, that’s the feedback I most want. Reach me through the contact form or on LinkedIn.
Free to use and adapt with attribution — CC BY-SA 4.0.