inside your personal firm and virtually any failure is reasonable: you retry, fall again, or doubtlessly even ignore it. Put that very same workflow behind a buyer’s API or MCP server and the grace is gone. Now just one factor issues: did the client get an accurate, usable end result? Their course of is determined by yours delivering one. They, not you, now resolve what counts as delivered. At Databook we course of billions of tokens for the world’s largest enterprises; this text is predicated on actual information from manufacturing flows at scale. I hope it gives you some helpful insights.
Delivering that result’s more durable than it appears to be like, as a result of LLMs are notoriously unreliable. They fail steadily, in 4 flavors: an invalid reply (empty, unparseable, or just improper), a tough error, no reply in any respect, or no reply in time. And the entire run solely succeeds if each step does, so the extra you chain collectively, the extra possibilities there are for certainly one of them to fail. A workflow of individually wonderful steps can nonetheless come out a coin flip.
Inside your personal firm you may take in each certainly one of these, as a result of you might have slack on each axis: retry the failed step, wait out the sluggish one, spend somewhat extra, calm down the bar in case you should. Put the identical workflow behind a buyer’s API and the slack vanishes, as a result of the run now has to clear three useful resource budgets on the identical time, none of which you set:
- Time — a window that closes whether or not or not you’re finished: a tough gateway timeout (one to a few minutes, generally 5) that severs the connection mid-run, or one thing softer: an SLA, a caller blocked on the end result, a course of that may solely wait so lengthy. And it doesn’t resume: when the window closes, the client simply retries, beginning the entire run over from zero.
- Price — now a margin, not a pool. Each run carries a value the client already paid, so it has to return again worthwhile, not merely inexpensive. And the client, not you, decides how usually it runs.
- Tokens and price — a per-minute token funds (TPM) you share throughout each buyer without delay, they usually are likely to name in the identical bursts. You hit the ceiling precisely when load is heaviest, which is strictly when latency is worst.
Beneath all three sits a tough flooring you by no means commerce beneath: high quality. The reply must be proper to rely in any respect. A quick, low cost, on-time reply that’s improper remains to be a failure. High quality isn’t a funds you spend down.
Any certainly one of these you might handle by itself. The bind is that they apply collectively and pull in opposition to one another, so the apparent repair for one spends one other. Wait out a sluggish step and also you blow the time window. Race a second copy to beat the clock and also you burn value and quota. Attain for a stronger mannequin to clear the standard flooring and also you get slower. Not one of the budgets are yours to loosen, so the one transfer left is to commerce intentionally throughout all of them without delay — with out ever dropping under the ground.
That’s what makes a customer-facing workflow a genuinely completely different factor to construct, and it generally forces a playbook that, from the within, appears to be like completely backwards:
- Kill a name that hasn’t failed
- Fireplace a reproduction of a name you’re already paying for
- Drop to a weaker mannequin on function
Inside your personal partitions you’d by no means hassle. You’d simply let the sluggish step end. And the funds that punishes you most quietly is time: miss it and nothing appears to be like damaged in your aspect. An ideal reply that lands a number of seconds late nonetheless reads as a hit in your dashboards and as a failure to the client, and it’s the one restrict nothing within the stack enforces for you.
Right here’s the thesis, up entrance, as a result of every part else serves it: as soon as high quality clears the bar, dependable supply is a query of variance, not pace. A predictable completion time beats a quick one with an extended tail, as a result of your clients can’t run their infrastructure in your finest case; they need to construct on your worst.
What that is — and isn’t: workflows, not free reasoning brokers
One distinction up entrance, as a result of it modifications every part. That is about an agentic workflow: a identified course of move with LLM-powered steps inside it, run by a deterministic orchestrator. It’s not a reasoning agent that decides its personal subsequent transfer at runtime. For a similar activity, a workflow is just quicker: it already is aware of the plan, skips the deliberation, and runs each unbiased step in parallel, so it reaches the identical reply in a fraction of the time and price a reasoning agent would take. Each have their place (reasoning brokers are far more versatile), however they fail otherwise and also you repair them otherwise. A reasoning agent’s downside is deciding what to do; a workflow’s downside (the one clients really feel) is delivering what it already is aware of tips on how to do, with high quality, and in time. This text is in regards to the latter.
How our system is constructed
The findings under come from our structure, and they need to generalize. These are abnormal, direct API calls. Nonetheless, it helps to know the setup so you may examine it to yours.
We run a customized orchestrator over managed third-party APIs (no self-hosted fashions on this dataset), and we run flagship fashions each instantly via their suppliers (OpenAI, Anthropic, …) and thru managed platforms (Bedrock, Databricks, …), so high fashions have greater than 1 supplier. That lets us examine serving paths and transfer work between them.
Our workloads are a mixture: easy agent calls, deep reasoning, extractions, JSON and free textual content outputs. For a big fraction of calls we synthesize a big truth base into a solution, so giant enter and small to medium outputs. The analytics on this article maintain enter and output dimension fixed inside buckets (see appendix).
The sluggish tails we encounter are largely transient. Word that in case your structure is self-hosted or on devoted capability the tail might behave otherwise, and can warrant one other strategy. Secondly, operating a number of suppliers is what makes routing a hedge to a separate funds sensible. With a single supplier, fewer of those strikes can be found.
The declare, and the receipts
So right here’s the transfer that sounds backwards: we lower a step off at 20-30 seconds even once we understand it might need answered completely somewhat later — and that makes the system extra dependable, not much less.
That isn’t a hunch. It’s true on paper — the mathematics of heavy-tailed retries is unambiguous — and it’s true within the information: a scan of effectively over 1,000,000 latest manufacturing LLM calls throughout our enterprise workloads — actual buyer visitors. The very first thing that visitors tells you is how unusual a single name’s timing actually is. A typical longer-output name comes again in a few dozen seconds. However one in 100 takes thirty seconds, generally a full minute or extra — for no motive related to how a lot work it was doing.
That hole between the standard name and the sluggish one underlies a lot of this text. The remainder of the article evaluations what to do about it.
Why the clock is unforgiving
A workflow isn’t judged on its common. It’s judged in opposition to a deadline. On common our flows end comfortably; nonetheless outlier runs in lengthy tails don’t. These tail runs aren’t damaged. They’d return an ideal reply a bit later, and on an inside run they’d rely as successes. On the client’s aspect, each certainly one of them is a failure. The complete tail of your latency distribution, nonetheless appropriate, turns into an addition to your failure price.
That’s why the quantity that issues right here isn’t common latency, it’s variance. A quick median buys you nothing in case your tail is lengthy.
The second squeeze is sunk value. The deeper you might be right into a workflow, the extra you’ve already spent: time, {dollars}, and your TPM quota. A failure on step 9 is much dearer than the identical failure on step two. You throw away every part the workflow constructed and you might have much less of the clock left to shift gears. We by no means restart the entire workflow ourselves, however the buyer will. If we fail, they may virtually actually retry, beginning the complete move once more from the start. That compounds the issue on our aspect. It burns extra value, extra token funds, and the error funds on the SLA. And since the situations that made the run fail normally haven’t modified, the retry has an identical likelihood of failing. Worse, it tends to occur throughout a high-TPM window. The worst doable time to pile additional load onto an already-strained system, and precisely when the percentages of failing once more are highest.
There’s a second multiplier, and it’s straightforward to overlook. The primary is the one from the opening: reliability compounds, so a series of individually wonderful steps can nonetheless come out a coin flip1. However that failure is at all times advised as a narrative about correctness: getting a improper reply.
Right here’s what you virtually by no means hear about: the very same compounding occurs on the clock. Each step provides its personal small likelihood of touchdown within the sluggish tail, and people possibilities stack. So the extra steps you chain, the extra probably it’s that no less than one of them blows the deadline, even when each step is individually quick. That’s the multiplier this text is about, and it’s the one the literature leaves out. So let’s take a look at the numbers.
What an LLM reply time really appears to be like like
The standard occasions within the chart above sit in a reasonably tight band: each mannequin finishes a typical name someplace between eight and twenty seconds. The tails usually are not tight in any respect. One mannequin’s 99th-percentile name is available in round 30 seconds, one other’s previous 80. Related median, wildly completely different worst case. Promise a buyer your median and also you’re mendacity to the 1-in-20 and 1-in-100 calls within the tail, and a multi-step workflow hits these continuously. A quick typical time will not be a predictable one.
The plain objection is that the sluggish calls are simply doing extra work: greater prompts, longer solutions. They aren’t. Pin each the immediate dimension and the response size and the tail barely strikes: inside a single dimension bucket (work held fastened), p99 nonetheless runs two to seven occasions the median (Determine 4). The slowness isn’t about how a lot the decision has to do — in our visitors it’s largely transient (queueing, scheduling, mid-stream rivalry, a supplier hiccup), which is strictly what makes it value interrupting.
One sluggish step sinks the entire run
You’d assume a workflow misses its deadline as a result of many steps have been every somewhat sluggish. It virtually by no means occurs that method. When a series blows its funds, it’s normally one step that wandered into its tail whereas every part else behaved advantageous. Mathematically, a series’s overrun is dominated by its single worst step, not by the buildup of mildly sluggish ones. The entire behaves like its most, not its sum.2
That’s excellent news. You don’t want each step quick. You want to cease any single step from operating away. Which is the cutoff.
Sidebar — The maths, briefly (skip except you want math)
Three outcomes sit beneath the argument:
- Compounding. Simply the arithmetic of unbiased steps: n steps every succeeding with likelihood p provides pⁿ end-to-end. At p = 0.95, ten steps ≈ 60% and twenty ≈ 36% — multiplication, no modeling. The identical compounding hits the clock: every added step is one other unbiased draw in opposition to the latency tail (the 2-7× p99/p50 we measure per name), so the percentages that no less than one step blows its funds solely rise with size. Independence is the simplification — shared capability correlates actual steps — nevertheless it’s the conservative, illustrative case.
- The one large soar. LLM latency is heavy-tailed (lognormal-ish), and the lognormal is subexponential. For unbiased subexponential steps the tail of the sum is simply the sum of the tails — `P(ΣX_i > t) ≈ Σ P(X_i > t) ≈ P(maxᵢ X_i > t)` as t grows. In phrases: a series overruns as a result of one step hit its tail, not as a result of many have been mildly sluggish.2
- Hedging, and why it really works for any failure. Fireplace n unbiased makes an attempt and take the primary good one: if a single try fails with likelihood q, all n fail with likelihood qⁿ. That arithmetic doesn’t care what “fail” means — a blown deadline, a tough error, or a improper reply all purchase down the identical method, which is why the identical retry/race/fallback transfer serves each taste. For the timing taste particularly it additionally shrinks unfold: for the reason that variances of unbiased steps add, `Var(ΣX_i) = Σ Var(X_i)`, capping every step’s tail shrinks the entire chain’s. All of it rests on the makes an attempt being unbiased (recent attracts, recent queue) — which is strictly why a parallel re-draw collapses a transient tail (or an unfortunate dangerous reply) and does nothing for a deterministic one.3
The transfer: lower early, then race
If a step has wandered into its tail, ready is the worst factor you are able to do — you’re spending your scarcest useful resource in your least probably payoff. So that you quit early and check out once more in parallel: fireplace a recent try and take whichever returns first. A recent try hardly ever lands in the identical pothole, so two of them match contained in the time one caught name would have eaten — and the percentages of each being sluggish are tiny (if one is sluggish with likelihood q, two are each sluggish with likelihood q²).3
The median barely strikes: about 10 seconds as a substitute of 12. The tail does the alternative: the 99th percentile drops from roughly 60 seconds to 25, and the run-to-run unfold is greater than lower in half. You purchase predictability for the value of some additional tokens.
That value is actual, and it pushes again. Racing doubles the token invoice on that step, and tokens are a shared, capped funds. So value is a real downward drive on how freely you retry and race. However run the arithmetic and it’s lopsided. Doubling one step prices you that step’s tokens, as soon as. Blowing the deadline throws away every part you’ve already paid for, and the client virtually at all times retries, re-running all N steps of the workflow, no less than as soon as, generally extra. The deeper into the move you might be, the extra one-sided the commerce: a redundant try on step 9 is reasonable subsequent to discarding steps one via 9 and watching them run once more. So that you hedge anyway. You simply don’t hedge indiscriminately, as a result of that shared token funds bites again hardest precisely while you most need to spend it (extra on that strain shortly).
One nuance that decides which fallback to achieve for: the path has to match why the step is failing.
- Gradual for transient causes → re-draw, ideally in parallel. A recent try escapes the stall. (A plain serial retry is weaker right here on an extended step — you’d pay the lengthy technology time twice.)
- Gradual as a result of the work is genuinely large → don’t re-run the identical name. Fall down to a quicker mannequin, or to an alternate path that reaches the identical end result extra cheaply.
- Fallacious, not sluggish → fall up to a extra succesful mannequin. Velocity received’t repair a foul reply; functionality may. (That is the standard flooring from earlier, enforced at runtime.)
Lower on the best sign
A solution time is de facto two phases.4 The await the first token is usually queueing and scheduling; the technology that follows, token by token, is the remainder. Which part carries the tail decides what you set the cutoff on. And that is determined by how a lot the step writes.
For the longer steps this text is about (those that press in opposition to a deadline), the tail lives in technology, not the first-token wait. A sluggish queue is a small slice of a forty-second name; the unfold that blows the funds is within the tokens. So lower these on complete elapsed time, or on tokens emitted thus far in opposition to the time you might have left, not on time-to-first-token. (For brief steps the stability flips: with little to generate, the first-token wait is many of the name, and time-to-first-token turns into the cleaner lower. Measure your personal steps to see which aspect you’re on.)
Two alerts are value wiring in regardless:
- No first token in any respect, previous the cutoff? That’s caught, not sluggish. Quit and hedge. A recent parallel try will get newly scheduled and virtually at all times wins.
- Tokens flowing nevertheless it’ll blow the funds? Don’t re-run it. You’d simply regenerate the identical size on the identical pace. Fall to a quicker mannequin.
And one failure no clock can catch: a step that returns on time however returns junk (e.g. it’s empty, truncated, or unparseable). A latency cutoff sails proper previous it; solely a high quality examine downstream will. For any step that’s imagined to return a selected form, the most cost effective such examine is a strict validation proper after the decision. Parse the end result in opposition to the anticipated schema or object, and deal with a validation failure precisely like every other: lower and fall again (re-draw, or fall up to a extra succesful mannequin). It catches a significant slice of dangerous solutions earlier than they attain the subsequent step. Reducing early buys you predictability, not correctness. Hold these two jobs separate.
The catch: hedging spends the funds you’re shortest on
Racing has an ungainly property. The tail is worst when the system is busy. And “busy” is strictly when your tokens-per-minute funds has the least room left. So the one transfer that fixes the tail needs to spend tokens on the exact second they’re hardest to return by. Do it blindly and also you get a pile-on: sluggish calls set off hedges, hedges add load, load makes every part slower, extra calls cross their cutoff. A latency downside turns into a rate-limit downside.
Two information make this much less forgiving than it first appears to be like. The associated fee is dedicated the moment you fireplace the second name. Cancelling the loser frees your connection, however the supplier retains producing, and billing, the deserted try. There’s no clawback, so all of the management has to dwell on the determination to hedge, not after. And also you normally can’t see how a lot funds is left. Estimating it’s doable however concerned, so any scheme that “eases off because the quota fills” is tough to run in apply.
What works in apply is cruder and extra structural:
- Ship the hedge someplace with its personal funds. Token limits are per-model and per-provider, and most of us run multiple (as famous in How our system is constructed). Routing the retry to a completely different mannequin or supplier will get a separate quota and an unbiased draw. The identical transfer that escapes the stall additionally avoids spending the scarce funds twice.
- Hold hedges uncommon by building. That is what the precomputed cutoffs already purchase you: with the edge set at every step’s measured p95, a hedge fires solely on the sluggish minority, so the additional spend stays small with no runtime accounting in any respect. (Similar cutoffs as the subsequent part, no new equipment.)
- React to the alerts you really get. You most likely can’t learn headroom, however you may learn 429s and climbing latency. Deal with these because the cue to hedge much less and lower later, no more.
- At actual saturation, cease hedging. As soon as the supplier is already returning rate-limit errors, extra makes an attempt solely deepen the opening. Downshift to a smaller, cheaper mannequin or shed the work as a substitute.
One lever we haven’t constructed, and provide solely as a path: an express international cap that holds hedged calls to a small fraction of complete visitors, unbiased of the per-step selections. It’s the principled backstop the tail-at-scale work factors to;3 we set conservative cutoffs as a substitute and haven’t wanted it, however at greater hedge charges that’s the place we’d go subsequent.
Sidebar — A budget strikes you make first
Cutoffs and hedging are insurance coverage. You purchase much less of it if the workflow is constructed effectively to start with. The defaults that fireplace on each request, earlier than any reactive trick:
- Parallelism by design. Lay the move out as a dependency graph and run each step the second its inputs exist. Then go additional — design the dependencies out. Fewer dependencies means extra steps are leaves, and a leaf can fail cheaply with out taking the remainder of the graph down.
- Don’t name the mannequin in any respect while you don’t need to. Essentially the most dependable name is the one you don’t make — use code, lookups, and validators wherever the work doesn’t really want a mannequin.
- Combine fashions per step, not per workflow. Quick and low cost the place it’s sufficient; succesful the place it isn’t.
- Cache the deterministic elements. Don’t pay an LLM twice for a solution that may’t change.
The purpose right here: spend your reliability funds on construction first, so the clock work has much less to repair.
When do you really pull the set off?
The cutoff is a knob, not a relentless. How onerous you flip it comes down to a few plain questions on every step:
- How a lot does the reply want this step? Good-to-have: let it go. Should-have: shield it.
- How a lot is ready on it? If nothing is determined by it, let it run to the deadline. If half the workflow is queued behind it, end it sooner, and ensure it’s proper, as a result of a improper reply right here poisons every part downstream.
- How a lot time is left? Lots: retry calmly. Virtually out: lower quick and fall again.
The extra a step is must-have, load-bearing, and quick on time, the sooner you fireplace the backup and the extra you’ll spend to hedge it. An optionally available, terminal, early step will get none of that. (“Early or late within the move” was by no means the true axis. It was a proxy for the way a lot nonetheless is determined by this step.)
And also you don’t guess the quantity. You run the workflow many occasions, measure every step’s latency curve (P95), and set the cutoff from that curve. Under the step’s worst case, weighted by the three questions. A step that normally solutions in 20 seconds will get lower at 30, although it might need succeeded at 60.
Why virtually no person does this
This isn’t onerous. It’s nuanced, and most groups don’t have the engine for it.
The favored workflow instruments, the Airflows and Temporals, have been constructed to make pipelines sturdy: retry, resume, don’t lose state, they usually’re excellent at it. Their timeout recommendation follows from that purpose: set a per-step timeout longer than the slowest run and retry till it succeeds.5 That’s the best intuition when the job is to sturdy completion, and it’s precisely the improper recommendation when the job is to complete in time. Your workflow engine will fortunately retry a step many occasions; it has no notion of a step’s measured typical time and downstream implications, so it may possibly’t lower early and swap fashions. That isn’t a flaw. It’s by design.
The distributed-systems fundamentals are already on our aspect: work from a deadline funds, match every timeout to measured latency.6 We’re not contradicting that. We’re making use of it to a case these instruments don’t assume: a brief, non-resumable funds the place the best transfer on the cutoff is a quicker various, not the identical name once more. Similar precept, inverted path.
Takeaway
One factor, in case you preserve nothing else: a predictable completion time beats a quick one with an extended tail. Low variance beats low latency. You’ll be able to’t promise a buyer a median, solely a sure. Every little thing right here serves that sure. Reducing early, hedging, racing, designing out dependencies: every trades somewhat common pace for lots much less variance. You quit the best tail to purchase the left.
In a customer-facing agentic workflow, reliability is the product. The craft isn’t proudly owning a bag of retries and fallbacks, these are desk stakes. It’s deciding, per step, whether or not to hedge and when to surrender, from the constraints and the measured conduct of your personal system.
APPENDIX
In regards to the creator
Frank Wittkampf is Head of Utilized AI Engineering at Databook. His crew architects, builds, and operates a completely customized AI stack together with deep reasoning, an agentic workflow engine, AI asset technology, agentic harnesses, information base & context graph, AI pre-processing, multi-tenant AI configuration administration, and so forth. This AI infrastructure powers the GTM groups of high Enterprise corporations like Microsoft, Salesforce, Amazon, Databricks, and plenty of others.
A be aware on the information
The latency figures right here come from latest (June 2026), anonymized manufacturing visitors throughout enterprise buyer workloads — roughly 1.2 million LLM calls over a 30-day window, not artificial benchmarks or a public hint. As described in How our system is constructed, these are direct calls to managed third-party APIs, which is a part of why the sluggish tail is essentially transient. The numbers within the textual content describe the longer calls (output ≥ 600 tokens), since these are those that truly press in opposition to a deadline; shorter calls are quicker and fewer variable. All through, a “tail ratio” (p99/p50) holds name dimension fastened inside a bucket except acknowledged in any other case. Fashions are labeled by household and serving path solely; predictability is determined by the serving path (e.g. a managed API vs. a direct one), not simply the mannequin, so these are intentionally not a mannequin rating. Durations have been bucketed in one-second bins; a tough 90-second ceiling truncates solely the final ~0.2% of longer calls, so the tail you see is actual, not an artifact of the cap.
Isn’t the tail simply the larger calls?
The truthful objection to Determine 4: every row is a token bucket, not a set token rely, so possibly the sluggish calls inside a cell are merely the bigger ones — extra to prefill, extra to generate — and the tail is simply dimension, not something transient.
It isn’t, and the information’s personal form reveals why. If dimension drove the within-cell tail, two issues would observe: the tail ratio would develop with the quantity of labor, and essentially the most tightly bounded cells would have virtually no tail. Neither holds.
Two issues to learn off it. First, the tail ratio is flat at roughly 2–4× throughout each output-size column — it doesn’t climb because the work grows, so the tail doesn’t scale with the work. Second, and decisively, take a look at the leftmost column: these calls emit at most 50 output tokens, so technology time bodily can’t range by greater than a few second — but the tail there’s nonetheless ~3.5×. There is no such thing as a dimension variable giant sufficient to provide that. The residual unfold is transient (queueing, scheduling, a momentary supplier hiccup), which is strictly what a recent try escapes.
Why these numbers look smaller than the two–7× quoted earlier: the column figures listed below are volume-weighted averages throughout many cells, which clean out the unfold, whereas the two–7× within the physique is the per-call envelope — the vary particular person cells really span. Similar information, two completely different cuts: the averages present the tail doesn’t scale with work; the envelope reveals how huge it will get on any given name.
Notes & Footnotes
Word: All pictures created by the creator.
1: Ten steps at 95% every ≈ 60% end-to-end; twenty ≈ 36% (assuming independence).
2: The lognormal lies within the subexponential class, the place the tail of a sum of unbiased phrases is asymptotically the sum of the person tails: `P(S_n > t) ∼ Σ_i P(X_i > t) ∼ P(max_i X_i > t)` as t → ∞ — the “single large soar” precept (Foss, Korshunov & Zachary, An Introduction to Heavy-Tailed and Subexponential Distributions, Springer, 2nd ed. 2013, eqs. 1.3 & 1.6). It’s an asymptotic assertion and assumes independence, so deal with it because the instinct for why one sluggish step dominates, not a plug-in system.
3: If every unbiased try is sluggish with likelihood q, two parallel makes an attempt are each sluggish with likelihood q²; n makes an attempt, qⁿ. The traditional hedged-request end result (Dean & Barroso, “The Tail at Scale,” CACM 2013); in an agent setting, Winston et al. (arXiv:2605.21470, ICML 2026) select between serial, parallel, and hedged execution from measured latency curves. On our manufacturing information, racing two makes an attempt lower p99 on longer steps by greater than half (≈60s→25s) whereas sequential re-draw on complete time didn’t.
4: The break up is normal in inference work: “time to first token” (queue + prefill) versus per-token technology. See e.g. Agrawal et al., Taming the Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (arXiv:2403.02310, 2024). In our manufacturing visitors the tail for longer calls sits within the technology part, not the first-token wait — which is why we lower lengthy steps on complete elapsed time slightly than time-to-first-token.
5: Temporal’s exercise timeouts are designed to complete finally, together with retries — therefore Begin-To-Shut set above the sluggish tail.
6: Google SRE, gRPC deadlines, and Spanner all advise propagating a complete funds and dropping work that may not assist the caller. We lengthen the identical precept to a sync, non-resumable buyer funds.
