Cease Giving Your AI Agent a Search Field and Begin Giving It Typed Instruments, Arduous Bounds, and a Gate It Can not Speak Previous

Within the second article of this collection, I retired the question router and made retrieval a hard and fast, fused pipeline the place each query runs hybrid search and typed graph traversal. The union is reranked as soon as with the connection paths seen, and the grounded bundle carries these paths as proof. I deployed it on Azure subsequent to the primary article’s stack, ran the identical artificial insurance coverage corpus by it, and measured what modified. Entity fragmentation dropped from 149 machine-extracted ideas to 120 on similar paperwork. The contradiction detector fired through the seed with out being requested. The system now walks its personal graph.

What it nonetheless can’t do is determine that one stroll was not sufficient.

That sentence is the entire motive this third article exists, so let me be exact about it. The pipeline I constructed is mounted: one retrieval move, then one era. All the things the reply will want needs to be fetched earlier than any reasoning begins, by a system that has not but reasoned. For many questions that’s precisely proper since it’s quick, low-cost, auditable and predictable. However there’s a class of questions the place you solely uncover what to retrieve after you have got partially understood what you already retrieved, and for that class, a hard and fast pipeline is the improper form irrespective of how good its single move is.

The modern reply to that is an AI agent, and I shall be sincere about my hesitation in utilizing the phrase. Most of what will get revealed below “agentic RAG” proper now’s a retry loop bolted onto vector search, a framework tutorial, or an opinion piece with no working system beneath it. I didn’t need to add to that pile. So this text takes the agent query severely in the one method I understand how, to outline the design exactly, identify the experiments that might show it improper, run them on the identical corpus and the identical Azure stack as the primary two components, and report the numbers together with those that don’t flatter the concept.

The declare I’m testing is slender and, I strongly consider, extra attention-grabbing than the thrill: an agent’s reasoning is bounded by the vocabulary of its instruments. Give a mannequin a search field and a retry price range, and iteration buys you rephrased guesses. Give it the information layer’s precise vocabulary, like typed traversal, timelines, diffs, entity decision, a contradiction register, and iteration turns into one thing nearer to navigation. The primary two articles constructed that vocabulary. This one fingers the mannequin the steering wheel, places onerous bounds on the wheel, and measures what occurs together with the query I care most about: when the mannequin is accountable for its personal loop, does the contradiction gate survive, or does a useful agent route round governance the second it turns into inconvenient?

All the things runs on the infrastructure the collection already pays for. No new providers are added, and I imply that actually since the agent is code and tokens, not sources. The implementation lands in the identical repository as Half 2 [3], as a separate additive commit, so all the pieces the earlier article describes stays precisely the place its readers discovered it.

Half I – The design revision

1. The place Half 2 ended, truthfully

A fast recap for readers becoming a member of right here, and the same old sincere accounting for readers returning.

Half 1 constructed a persistent information layer subsequent to a RAG pipeline the place an proof layer holding chunked, embedded supply paperwork, and a structured information layer holding what the group understands, like ideas, choices, relationships, and contradictions as first-class objects with homeowners and an unresolved standing the system refuses to resolve by itself. Half 2 changed the router that selected between them with fusion: each engines run on each question, a bounded two-hop typed traversal joins hybrid search, and one exterior reranker orders the union with the connection paths offered as proof. Time moved inside the perimeters, the place each relationship carries a validity window and an ingestion timestamp, and entity decision turned a two-threshold pipeline with embedding blocking and gray-zone adjudication.

The stay deployment saved the collection sincere in each instructions. The great: 149 fragmented ideas turned 120 like-for-like, the ingest-time detector opened a contradiction by itself through the seed, and each grounded bundle carried a median of ten typed, time-valid relationship paths that search alone can’t produce. The humbling: recall on the golden set was 0.75 with the graph and 0.75 with out it, as a result of twenty-one paperwork are too few for entry-point misses to happen, and the sincere studying was that fusion presently earns its price by grounding high quality relatively than recall.

One property of that system issues greater than any of its numbers for what follows. It’s a mounted pipeline. The identical phases run in the identical order for each query. Nothing in it will probably have a look at an intermediate end result and determine to return for extra. Retrieval sufficiency and the judgment of whether or not we now maintain sufficient to reply is determined implicitly, as soon as, earlier than the mannequin has reasoned about something.

2. The query one move can’t reply

Let me make the ceiling concrete with the corpus’s personal materials, as a result of an architectural limitation you can not exhibit is simply an opinion.

The roof inspection requirement advanced: a common threshold of twenty years from June 2025, then an evaluation e mail explaining why a stricter threshold was coming, then a fifteen-year threshold scoped to H3 new enterprise from 1 March 2026. Half 2 solutions “which rule utilized on 20 February?” cleanly – that’s temporal filtering, and filtering is what mounted pipelines do nicely. Now ask a query one step extra demanding:

“Did any of our recorded claims contain properties that might have been flagged below the March roof threshold however handed below the rule that was really in drive after they had been filed?”

Answering that requires a sequence. First uncover that there have been two thresholds and when every was in drive, then extract the property attributes from every declare file, then apply the March rule counterfactually to claims filed earlier than March, then evaluate. The second retrieval, which declare information to drag and what to search for in them, is determined by understanding produced by the primary. No single question, nevertheless nicely fused, expresses that. The mounted pipeline retrieves what the query’s wording reaches: in all probability the 2 threshold paperwork, presumably one declare file if its vocabulary occurs to overlap. The mannequin then solutions from no matter landed within the bundle, and the failure is quiet – {a partially} grounded reply that appears full.

I need to identify the final form, as a result of it’s the recurring character of this text: in a hard and fast pipeline, the retrieval plan is frozen earlier than reasoning begins. Half 2 made the one move a lot smarter as a result of it walks relationships, it respects time, it surfaces contradictions. Nevertheless it can’t revise its personal plan. The claims handler I hold imagining because the system’s consumer doesn’t work this manner. She reads the primary doc, realizes it names a rule she has not seen, goes and finds the rule, realizes the rule modified, checks which model governs, and solely then solutions. The attention-grabbing factor about her conduct will not be intelligence, it’s that her retrieval is interleaved with her reasoning, every informing the following step of the opposite.

That interleaving has a reputation within the analysis literature – ReAct [6] formalized it as reasoning traces and actions feeding one another, and a whole {industry} hype cycle presently orbits it. The following part is about separating the a part of that cycle value preserving from the half value ignoring.

3. The agent query, requested severely

It’s value mentioning right here that I got here near not writing this text. Someday across the center of making ready Half 2, “agent” accomplished its transition from a technical time period right into a advertising and marketing one, and the quantity of content material carrying the phrase is outstanding. Studying by it, virtually all the pieces falls into one in all three buckets: tutorials wiring collectively an orchestration framework, “agentic RAG” posts the place the agent’s whole toolset is identical vector search referred to as repeatedly, and structure suppose items with no implementation to verify. Not one of the three solutions the one query I even have as an engineer: measured towards a powerful single-pass baseline, on the identical knowledge, what does the loop purchase, and what does it price?

So this text commits to answering that query, and the dedication shapes each design choice in it. However earlier than the design, I owe you the argument for why the reply is likely to be “one thing actual,” as a result of “everyone seems to be doing brokers” will not be an argument.

Right here is the argument. When iteration fails to assist, it is actually because every step of the iteration attracts from the identical instrument. An agent whose solely software is vector_search(question) can rephrase, decompose and retry, however each try remains to be a similarity lookup towards flat chunks, so principally the identical guess with totally different phrases. Iterated guessing converges slowly, and worse, the agent has no solution to confirm something. Similarity can counsel, however it can’t affirm {that a} rule applies, {that a} model was in drive, or that two paperwork disagree. The loop provides latency and tokens to a mechanism that was already doing its greatest.

Now change the instrument as an alternative of the retry rely. The information layer constructed throughout the primary two articles speaks a richer language the place relationships have varieties, edges know after they had been true, entities resolve to canonical identities, and disagreement is a queryable object relatively than an unlucky vibe. An agent holding that vocabulary can do qualitatively various things per step: resolve the entity it’s uncertain about, stroll the precise relationship it wants relatively than hoping similarity surfaces it, pull the timeline of a requirement as an alternative of reconstructing historical past from prose, and ask immediately whether or not the group has an open contradiction on the subject earlier than asserting something. Every step could be a verification, not a guess.

That’s the thesis in a single sentence: the agent is barely pretty much as good because the vocabulary its instruments expose, and the attention-grabbing engineering lives within the instruments, not the loop. The remainder of Half I designs the vocabulary, the loop, and the governance. The strategy elaborated right here exams whether or not the thesis survives contact with the corpus together with the configuration the place I intentionally hand the agent nothing however a search field, so the vocabulary declare has a management group.

4. Eight instruments, one contract

The software floor is the API the mannequin causes by, and I handled its design with the identical seriousness as a public API, as a result of that’s what it’s – the buyer simply occurs to be a language mannequin. Three rules ruled it.

Sufficiently small to carry in a mannequin’s working consideration. Each software competes for the mannequin’s judgment on each step; the function-calling documentation itself tells you to maintain descriptions tight [10]. Eight instruments, no extra.

Aligned with the layer’s semantics, not with retrieval mechanics. The instruments expose what the information layer means (proof, understanding, relationships, time, identification, disagreement), and never how it’s saved. Nothing within the floor says “index,” “vector,” or “Gremlin.”

Learn-only, each one in all them. The agent solutions questions; it doesn’t preserve the layer. An agent that may write to the information base it cites is a special danger class with totally different failure modes, and it’s intentionally deferred. I’ll say extra in part 7.

Right here is the contract:

Each software returns the identical envelope: outcomes, supply identifiers, validity metadata, and an sincere inadequate flag when the layer has nothing as a result of an agent that can’t be instructed “there’s nothing right here” will fill the silence with confidence. Each name is logged into the hint with its arguments, end result digest, and token price. There’s little new equipment behind these – traverse, timeline and diff are skinny wrappers over the graph retailer Half 2 constructed; resolve_entity fronts the decision pipeline; the searches entrance the fused retrieval parts. The work is the floor, not the plumbing, and that’s the level. Two articles of infrastructure earn their hold right here by making the highly effective instruments low-cost to reveal.

Yet one more design choice value defending: the contract is intentionally formed so it might be mounted anyplace, not solely inside my loop. The Mannequin Context Protocol [8] has settled this sample industry-wide (instruments as typed, described, host-independent capabilities), and Half II reveals the similar floor working on Microsoft Foundry Agent Service, which speaks MCP natively. If the software floor is known as a contract, portability will not be a function, it’s the proof.

5. The loop is eighty strains on objective

Right here is the place I take a place that components of the neighborhood will disagree with, so let me state it plainly after which defend it: the agent loop on this article is about eighty strains of code, makes use of no orchestration framework, and that may be a function.

The loop itself is the textbook form and I make no declare of novelty for it: ship the query, the software contract and the system directions to the mannequin; if the mannequin requests software calls, execute them and append the outcomes; repeat till the mannequin produces a last reply or a certain journeys. Foundry’s operate calling handles the mechanics [10], together with parallel calls when the mannequin needs a number of instruments without delay. What issues, and what I couldn’t cleanly get from a managed abstraction, are three properties the experiments depend upon.

historical past = [system_contract, question]whereas rounds < MAX_ROUNDS and tokens < BUDGET and clock < TIMEOUT:    reply = mannequin(historical past, instruments=THE_EIGHT)    if reply.tool_calls is empty:        break                        # the mannequin believes it will probably reply    for name in reply.tool_calls:    # parallel calls accepted        end result = execute(name)       # bounded top_k, bounded hops        hint.append(name, digest(end result), tokens, ms)        historical past.append(end result)draft = reply.content material or compose_from(hint)   # a tripped certain nonetheless composesreply = composer.finalize(query, draft, hint)   # part 6

Arduous bounds, enforced exterior the mannequin. A most of eight tool-call rounds per query, a token price range, and a wall-clock timeout. When a certain journeys, the loop doesn’t fail, however it forces composition from regardless of the hint holds, and the reply says so. Value management by building, not by hope, and the certain numbers themselves are knobs the experiments will flip.

The hint as a first-class output. Each run returns, alongside the reply, the entire sequence of software calls: identify, arguments, end result digest, tokens, milliseconds. The hint is to this text what the paths array was to Half 2 – the artifact that turns “the agent reasoned” from a declare into one thing you’ll be able to learn, audit and disagree with. Additionally it is, as Half III will present, the debugging instrument for all the pieces that goes improper.

Equivalent mechanics throughout experimental arms. The ablation compares an agent holding all eight instruments towards an agent holding solely a search field. That comparability is barely clear if each run the identical loop, similar bounds, similar composer – one variable at a time. A framework’s inside orchestration selections, nevertheless smart, could be a confound I can’t examine.

To be honest to the frameworks: it is a analysis instrument, and devices need custody. If I had been delivery dozens of brokers with human handoffs and sturdy state, I might attain for the platforms constructed for that, and Half II reveals the identical contract working on one. The place will not be “frameworks unhealthy.” The place is that for this article’s query the loop have to be easy sufficient that it isn’t itself a suspect.

6. Company versus the gate

Now the part this text was really written for.

Half 1’s proudest conduct was refusal. When the corpus disagrees with itself (the trace-and-access contradiction being the working instance), the system presents each positions with their sources and declines to select a facet, as a result of recency will not be applicability and the group itself has not determined. Components 1 and a pair of enforced this structurally: the pipeline injected the contradiction into the context with directions the mannequin reliably adopted, and the pipeline, not the mannequin, determined the contradiction was there. The mannequin by no means had the choice of not seeing it.

An agent modifications the facility steadiness. The mannequin now decides which instruments to name, which ends matter, and when it has sufficient. And language fashions arrive with a deep bias towards being useful – towards producing the reply the consumer appears to need. So right here is the query I contemplate the sharpest one on this article, and I’m committing to a prediction earlier than working the experiment, so the end result can genuinely embarrass me:

When the mannequin controls its personal retrieval, does it respect an unresolved contradiction, or does it route across the gate to be useful? My prediction: the naive agent blows by it. Not all the time, however at a charge no ruled system ought to settle for. Generally it won’t name list_contradictions in any respect; generally it’s going to name it, word the battle in a subordinate clause, and reply anyway, as a result of each supply paperwork are individually persuasive and the mannequin needs to resolve stress, not current it.

If the prediction holds, the lesson generalizes far past my corpus: governance that lives contained in the loop is advisory; governance that survives company should stay exterior it. Which motivates the design the place the gate strikes from the retrieval stage right into a composer that owns the ultimate reply, downstream of a loop it doesn’t belief.

The composer does three issues to each draft. First, a grounding verify: every factual declare should map to a quotation from the hint or to a walked relationship path; unmapped claims are eliminated or flagged, within the spirit of Reflexion’s verbal self-correction [7] however bounded to a single move, one revision, not a self-talk spiral. Second, the gate as coverage: the composer seems up open contradictions for each entity the hint touched (its personal lookup, not trusting the agent to have made it), and if one exists, the both-positions-no-choice presentation is imposed on the reply no matter what the draft stated. Third, honesty forwarding: inadequate flags from the hint floor within the reply relatively than being papered over.

I need to be express that this design selection is itself a part of the experiment, not an assumption. Half III runs the refusal check with the composer’s coverage switched off – agent alone – and switched on. If the naive agent already respects the gate, the composer is redundant and I’ll report that. If neither configuration respects it, my design failed and I’ll report that too. The purpose of committing to the prediction now’s that both end result means one thing.

7. What the agent will not be allowed to do

A design can also be outlined by its refusals, and this one refuses three issues on objective.

No writes. Each software is read-only. The tempting subsequent step (an agent that information the contradictions it discovers, proposes supersessions, merges entities it believes are duplicates) is a genuinely thrilling design and the apparent Half 4. Additionally it is a special danger class, the place an agent that may edit the information base it cites can launder its personal errors into floor fact, and the overview workflows that make that protected deserve their very own article relatively than a paragraph on this one. The librarian agent waits.

No unbounded reflection. The composer will get one revision move. The self-critique literature is genuinely helpful [7], however in a cost-governed system, reflection is a tax paid on each query, and my expertise tuning the Half 2 reranker taught me how rapidly “another mannequin move” compounds. One bounded move, measured like all the pieces else.

No autonomy over its personal price range. The agent can’t increase its personal bounds, and certain journeys are seen within the hint and the reply. This sounds apparent till you learn agent code within the wild the place the retry logic quietly is the price range.

Furthermore, the identical self-discipline that saved Half 2 reasonably priced carries over unchanged: the knobs are express and few, tool-round cap, token price range, top_k per software, hop certain inherited from the graph retailer, and part 12 places anticipated numbers towards every. A system whose prices can’t be predicted from its knobs will not be an structure, it’s a legal responsibility with good advertising and marketing.

8. The experiments that might show me improper

Each earlier articles earned no matter belief they’ve by naming their falsification exams earlier than working them, so listed below are this one’s, designed earlier than the agent code exists.

Experiment 1 – the three-way ablation. One corpus, one golden set, three techniques: (A) Half 2’s mounted fused pipeline, untouched, because the robust baseline; (B) the agent loop holding solely search_evidence – the search-box agent the thrill builds; (C) the agent holding all eight instruments. Similar loop, similar bounds, similar composer for B and C – the one variable is vocabulary. Metrics per query: reply correctness towards the labeled set, quotation precision, software calls used, whole tokens, wall-clock latency.

The acceptance criterion is said prematurely: if C doesn’t beat B on correctness over the multi-hop subset, the vocabulary thesis of part 3 is improper and the attention-grabbing engineering was not within the instruments in any case. If C doesn’t beat A anyplace, the loop buys nothing over a powerful mounted pipeline at this corpus scale, and the sincere conclusion is that company is untimely right here. And to maintain myself sincere in regards to the probably form, at twenty-one paperwork I anticipate A to stay aggressive on the single-hop majority, and the agent’s case to relaxation on the multi-hop subset plus grounding high quality. If the margins are small, they are going to be printed small.

Experiment 2 – the refusal check. Each query within the golden set that touches the engineered contradictions, run towards A, B, C with composer coverage off, and C with coverage on. Scored on three outcomes, offered each positions with out selecting, selected a facet silently, or refused incorrectly when no contradiction utilized. That is the governance experiment of part 6, with my blown-gate prediction on file.

Experiment 3 – hint high quality on the walkthrough set. For the multi-hop questions, does the hint present the form a site skilled would acknowledge (resolve, then stroll, then verify time, then compose), or software thrashing? Partly qualitative, truthfully reported as such, with full traces printed.

The golden set itself grows from eight inquiries to fifteen for this – part 13 explains why writing these seven new questions was the toughest a part of the entire preparation.

Half II – Rising the Azure stack (barely)

9. What’s already working, and what’s intentionally not added

The Azure chapter of this text is brief, and the shortness is the discovering.

Half 2 left a useful resource group in Sweden Central working the entire layer: AI Search with the hybrid indexes, Cosmos DB for NoSQL because the system of file, Cosmos DB for Apache Gremlin holding the bitemporal graph, Microsoft Foundry serving gpt-5-mini and the embedding mannequin, the FastAPI orchestrator on Container Apps, and the projection Operate App – all keyless the place the platform permits, managed identification all over the place else. Each software in part 4’s contract maps onto one thing in that stock:

The rely of latest Azure providers required: zero. The agent is three new Python modules and one endpoint deployed as a brand new revision of the prevailing Container App. I hold repeating this as a result of it’s the article’s quietest however perhaps most transferable lesson. If the information layer is constructed correctly, company is an increment of code, not of infrastructure. The funding went into the layer throughout two articles; the agent is the layer lastly being used to its design.

The one sub-resource addition is available in part 11, a Foundry undertaking to host the Agent Service demonstration, and it idles at zero price.

10. The loop on Foundry

The loop runs on the prevailing gpt-5-mini deployment by the usual function-calling interface [10]: the eight instruments are declared with tight descriptions, tool_choice stays on auto, and parallel software calls are accepted when the mannequin requests a number of without delay – which it does, usefully, when resolving a number of entities early in a query. Two Half 2 classes carried straight over. Structured outputs hold the composer’s grounding verify machine-readable, the identical json_object self-discipline that mounted the extraction pipeline’s truncated responses. And token budgets are set with reasoning fashions in thoughts: gpt-5-mini spends a part of its price range considering earlier than it solutions, and a loop that forgets this can misattribute its prices.

Observability lands within the Utility Insights occasion that has been within the useful resource group since Half 1: each software name turns into a dependency telemetry occasion, so one agent run reads as an end-to-end transaction, the loop’s rounds, every software’s latency, the composer’s passes. For a system whose entire pitch is auditability, watching a reasoning chain render as a distributed hint in a inventory Azure monitoring view is precisely the sort of boring that enterprise readers ought to discover reassuring.

11. The identical contract on Foundry Agent Service

I argued in part 5 that the experiments want loop custody. Each Azure reader deserves the follow-up query answered head-on: then what’s Microsoft Foundry Agent Service for, and does this design match it?

Foundry Agent Service [9] is the managed path: outline an agent (directions, mannequin, instruments) and the platform runs it, with dialog state, tracing, content material filters, identification, and publishing to locations like Groups dealt with for you. It speaks the identical function-tool form my loop makes use of, and it mounts MCP servers and gear collections natively – which is precisely the door the contract design of part 4 was formed for.

So I did precisely that, and it labored on the primary sincere strive. I stood up a Foundry undertaking on the prevailing account (the one genuinely new sub-resource this path provides – no standing price), registered the identical eight instruments on a managed immediate agent, and ran the roof-threshold walkthrough by the managed runtime. It referred to as the instruments in sequence – resolve_entity, traverse, search_knowledge, three search_evidence calls, list_contradictions, get_source – and returned the identical grounded conclusion my customized loop reached: the roof was beneath the 20-year rule in drive on the date of loss, and the March H3 15-year rule, although numerically triggered, was scoped to new enterprise and didn’t retrofit onto an in-force coverage, cited to INS-SYN-013, INS-SYN-003, INS-SYN-004 and INS-SYN-018.

I need to be precise about what I did and didn’t measure: each quantity on this article comes from the customized loop, which is the instrument the experiments want, and the Agent Service run is a portability demonstration, not a benchmarked arm. However it’s a actual run, not a promise – the identical contract mounted on a managed runtime I don’t management, and produced an equal grounded reply, as a result of the contract was designed to mount anyplace. That’s the entire level of placing the worth within the software floor relatively than in my eighty strains: the loop is mine, however the instruments are transportable, and an enterprise that wishes hosted threads, scaling and publishing can have all of that with out giving up the information layer the reply really rests on.

And the sincere division of labor, having now constructed each: select the managed service once you need operational leverage – hosted execution, threads, enterprise publishing, platform tracing – and settle for its orchestration as given. Preserve the loop in your individual code when the orchestration is the factor it’s essential management or measure: customized bounds, a composer that distrusts the loop, an ablation that should maintain mechanics fixed. In manufacturing I might anticipate the mature reply to be each without delay: the contract mounted on the managed service for the fleet, and a custody loop retained for the flows the place governance is load-bearing.

12. What the agent prices

The costing self-discipline from the primary two articles continues: identify the knobs, estimate the envelope, then measure stay and publish the measured quantity even when it embarrasses the estimate.

The fixed-cost delta is zero by building – part 9’s stock gained no providers, and the Foundry undertaking idles free. Each new price is a token, and tokens scale with precisely the knobs part 7 named:

Then I measured it, and one in all my predictions was improper in a method value preserving in. On gpt-5-mini, the complete agent averaged 5.1 software calls, about 11,400 tokens, and 29 seconds per query throughout the fifteen-question set. I had anticipated company to price roughly an order of magnitude greater than the mounted pipeline. On tokens it does price extra. On wall-clock it doesn’t: the mounted pipeline averaged 31 seconds on the identical questions, slower, marginally, than the agent – as a result of Half 2’s personal reranker and reply step are themselves two reasoning-model calls, and two cautious calls take about so long as 5 low-cost software rounds plus a compose. The sincere correction: at this corpus scale the agent’s additional price is paid in tokens, not in latency, and the “order of magnitude slower” I braced for didn’t seem. That doesn’t make the agent the economical default, because the token invoice and the variance are actual, and part 18 nonetheless routes straightforward questions away from it, however it does retire a scare quantity I might have printed on instinct.

The per-model price image – the place the token and latency unfold throughout gpt-4o-mini, gpt-4.1, gpt-5-mini and gpt-5 seems to be the widest dial in the entire system – is part 17’s topic.

Half III – The corpus, reasoned over

13. A golden set that may say no

I’ve to be sincere about the place the preparation time for this text really went: not into the loop, not into the instruments, however into seven questions.

The Half 2 golden set with eight questions and sixteen labeled proof objects was constructed to check retrieval, and its questions are answerable from one well-fused move. Reusing solely it will stack the deck for the mounted baseline and inform me nothing in regards to the loop. The ablation wants questions the place single-pass retrieval ought to fail, and writing such questions truthfully is more durable than it sounds, as a result of the failure should come from the query’s construction, not from vocabulary tips that merely dodge the index.

Every new query was drafted towards two exams. It should require sequential dependency – data from a primary retrieval determines what the second retrieval must be, and it have to be answerable from the corpus in any respect, with the complete chain labeled: anticipated proof, anticipated relationship paths, and the anticipated software form, that means the sorts of calls a reliable analyst would make, so hint high quality in experiment 3 has one thing to be judged towards. Three examples of the seven, so you’ll be able to decide the flavour:

• “Did any recorded declare contain a property that might have been flagged below the March roof threshold however handed below the rule in drive when it was filed?” – the counterfactual from part 2: timeline first, claims second, comparability third.

• “Hint how the recoverable depreciation place modified throughout coverage variations, and state which model ruled declare CLM-1042 and why.” – a diff plus an as-of verify, joined.

• “Is there any subject the place our claims dealing with steering and our coverage wording presently disagree, and what would a handler be instructed as we speak?” – contradiction discovery because the query itself, not as a facet impact.

Fifteen questions whole, single-hop and multi-hop intentionally combined so the ablation can present the place every configuration earns its hold, not simply whether or not. The labeling took a night per query in locations, and I contemplate it the best-spent time within the article: an analysis set that can’t say “no” to the system it evaluates is advertising and marketing with additional steps.

14. The walkthrough: studying a machine’s working notes

I dedicated, earlier than working something, to printing the hint whether or not it flattered the design or embarrassed it. It did a few of each, so right here is the entire thing.

The query is the roof-threshold counterfactual from part 2, aimed toward CLM-1108 – a storm declare whose roof was about eighteen years previous at a loss dated 20 February 2026: “was the roof sufficiently old to require inspection below the rule in drive on the date of loss, and wouldn’t it be below the March 2026 H3 threshold?” Answering it wants the agent to find there have been two thresholds, place the declare in time, and apply the March rule counterfactually – the interleaving a hard and fast pipeline can’t do. Run on gpt-5-mini with the complete software floor, right here is the verbatim hint: six software calls, 18,304 tokens, 32 seconds.

1. resolve_entity(point out="CLM-1108")     -> clm-1108-note-historical-exception2. traverse(entity_id="clm-1108-note-historical-exception",            relation_types=[applies_to, definition_changed_by, required_for,                            depends_on, derived_from], as_of="2026-03-01", hops=2)     -> no outcomes3. search_evidence(question="H3 threshold March 2026 roof age inspection rule ...")     -> 5 outcomes; first: E mail thread - Rationale for the H3 roof inspection threshold4. list_contradictions(entity_id="clm-1108-note-historical-exception")     -> no outcomes5. search_evidence(question="CLM-1108 roof age 'main roof masking' age at loss ...")     -> 5 outcomes; first: Artificial Declare Case Notes - CLM-11086. search_evidence(question="CLM-1108 'roof age' 'main roof masking' 'age at loss' ...")     -> 8 outcomes; first: Artificial Declare Case Notes - CLM-1108

Evaluate that towards what I predicted within the plan (resolve the roof idea, learn its timeline, then pull the declare information), and the sincere verdict is: proper vacation spot, scruffier route. The agent resolved the point out to the declare’s entity relatively than the roof-age-risk idea, tried a typed traverse from there that returned nothing as a result of that entity will not be the place the edge edges stay, after which, as an alternative of stalling, fell again to proof search and recovered – three searches, the final two overlapping, to assemble the roof age and the 2 guidelines. It by no means referred to as timeline or get_source. That is the feel the design part promised to point out: an actual extraction-built graph is messier than the curated one, the mannequin’s first anchor will not be all the time the proper one, and the interleaving that issues confirmed up not as elegant graph-walking however as a restoration – a failed traversal redirecting the following search. That’s nonetheless interleaving; it’s simply not the postcard model.

And the reply it composed from that scruffy hint was, to my slight shock, probably the most cautious of the entire run:

• On the date of loss (20 Feb 2026) the related underwriting rule in drive used the usual residential threshold (inspection usually requested when the first roof masking is greater than 20 years previous) – not the H3 15-year threshold [INS-SYN-003; INS-SYN-013].

• The March 2026 H3 replace (efficient 2026-03-01) launched a 15-year inspection threshold for brand new Hearthmere enterprise in zone H3, however that replace is scoped to new enterprise and was not meant to be retro-fitted onto in-force insurance policies written earlier than the zone mannequin change [INS-SYN-004; INS-SYN-018].

• What I can’t definitively say: The paperwork don’t state the roof’s set up 12 months or precise age, so I can’t decide whether or not the roof would meet the 15-year H3 threshold on its deserves.

It bought the rule-in-force proper, bought the scope of the March change proper, refused to retrofit a new-business rule onto an previous coverage, and drew the road at what the retrieved proof couldn’t help and stated so as an alternative of guessing the roof’s age. A hard and fast single move couldn’t have reached that form, as a result of the claim-specific searches in steps 5 and 6 had been solely smart after step 3 established that two thresholds existed. The hint is the argument.

15. The ablation: one corpus, three retrieval brains

Right here is the stay run on gpt-5-mini. Correctness is scored as key phrase protection of the labelled reply – a intentionally blunt proxy, and I’ll lean on it precisely as onerous because it deserves, which isn’t very. The sturdy columns are the target ones: software calls, tokens, latency.

The pre-registered expectation was that C would separate from each on the multi-hop subset, and it half did – C beats A on multi-hop (0.92 to 0.83) – however three of those numbers embarrassed my predictions, and they’re the attention-grabbing ones.

First, A wins the single-hop subset outright (0.95), and the complete agent is the worst of the three there (0.70). On straightforward questions the agent over-works: it often mis-anchors an entity (the part 14 hint is one instance), often over-refuses (it wrongly refused one non-contested query the place its traversal engaged a contested entity), and customarily spends 5 software calls arriving someplace a single fused move reached cleanly. This isn’t a disappointment! It’s the empirical backbone of part 18. The agent is a specialist, and specialists are a poor default.

Second, B – the search-box agent – posts the very best multi-hop quantity (1.00), and it’s a lure. Learn the solutions and the metric’s blind spot is apparent: with solely a search software, B iterates the identical question and dumps massive spans of proof textual content, which is precisely what a keyword-coverage rating rewards. Its solutions are the least surgical of the three, it can’t stroll a relationship or learn a timeline, and – part 16 – it will probably solely floor a contradiction when the uncooked chunks occur to spell one out. B’s 1.00 is the metric flattering verbosity, not the search field out-reasoning the graph. It’s the clearest proof within the article that the correctness proxy is blunt, and I might relatively present you the lure than disguise it.

Third, quotation precision is low and roughly flat throughout all three (0.19-0.29), and that’s my labels’ fault, not the techniques’. The golden set names a minimal set of anticipated sources per query; the techniques routinely cite extra, genuinely related paperwork that the label doesn’t listing, so precision towards a partial key undercounts everybody. I’m reporting it as a result of I stated I might, not as a result of it separates something.

So the sincere ablation end result, stripped of the noisy metric: the agent earns its carry on multi-hop and loses on single-hop, at a token premium and comparable latency. That may be a narrower declare than “the agent is best,” and it’s the true one at twenty-one paperwork. The magnitude belongs to a bigger corpus; the form – specialist beats generalist solely the place the plan should change mid-flight – is what this scale can truthfully present, and it is sufficient to design round.

16. The refusal check: does governance survive company?

I predicted in part 6 that the naive agent would blow by the gate. It didn’t, and I owe you the end result plainly: my prediction was improper, within the reassuring path. Throughout the three contested questions, each configuration disclosed the contradiction relatively than silently choosing a facet, and I imply each one.

And the identical naive row, that the complete agent with the composer switched off, held throughout all 4 fashions, not simply gpt-5-mini:

I learn all twelve naive solutions by hand relatively than belief a key phrase classifier, which, once I first ran it, mislabelled a number of of those in each instructions and almost handed me a dramatic false headline. The reality is calmer and, I feel, extra helpful. Each mannequin, when the retrieved materials or a list_contradictions name surfaced the battle, selected to report it as a result of the system immediate tells it to, and at this scale, with this immediate, that instruction was sufficient. The one comfortable spot: gpt-4o-mini as soon as opened with the endorsement place as if it had been the reply earlier than disclosing the battle two sentences later. No one selected a facet outright.

That is the department I pre-registered in part 6 – “if each agent rows rating clear, I’ll fortunately report that instruction-following made my composer redundant at this scale” – so right here is that report, with out spin. At twenty-one paperwork and one well-designed immediate, the advisory instruction sufficed, and the composer’s deterministic gate was, strictly, redundant for the result. I’m preserving it anyway, for 3 sincere causes. It converts the mannequin complied this time into the system enforces it each time, and you can not confirm prompt-compliance prematurely the way in which you’ll be able to confirm a code path. It nonetheless runs the grounding verify that the immediate doesn’t. And, the rationale that outranks the opposite two, the one real governance failure on this entire desk was not the agent in any respect: the mounted pipeline returned an empty reply on the customer-phrased contested query (q13, “are trace-and-access prices mechanically included?”). Its gate is determined by retrieval surfacing the contested idea, and that pleasant phrasing didn’t retrieve it, so it produced nothing. The agent, which may intentionally name list_contradictions relatively than hope the index cooperates, dealt with that phrasing on each mannequin. Governance that is determined by the wording reaching the proper chunk is precisely the fragility this collection has been retiring since Half 2.

One caveat I can’t bury: the composer’s gate fires solely on entities the agent engaged, so it’s a assure conditional on the agent having seemed, not an unconditional one. With coverage on, its template fired on two of the three contested questions and the mannequin’s personal disclosure lined the third. A composer that scanned the drafted reply for contested matters, not simply the hint, would shut that hole; that refinement is famous and deferred. What I can’t do is common any of this right into a single rating. A blown gate on a contested claims query will not be offset by ten well-cited straightforward solutions – and the sincere headline right here is that, at this scale, no person blew it, the mounted pipeline included proper up till a buyer requested politely.

17. The identical system, 4 fashions

Here’s a query the primary two articles by no means needed to ask, as a result of they ran on one mannequin. The second retrieval high quality is determined by a mannequin’s judgment (which software to name, when it has sufficient, whether or not to respect a contradiction) the mannequin turns into a design variable, not a hard and fast backend. So I did one thing the thrill not often does: I held the whole system fixed and swapped solely the mind.

The Foundry account already served gpt-5-mini. For this part I added three extra deployments on the identical account – gpt-4o-mini, gpt-4.1, and gpt-5 – and ran config C, the complete agent with the gate on, throughout all 4 on the similar fifteen-question golden set. Nothing else modified: similar instruments, similar bounds, similar composer, similar corpus, similar prompts. This isn’t a mannequin leaderboard, these exist and are higher than something I might produce on twenty-one paperwork. It’s a narrower and, I feel, extra helpful query for anybody really constructing one in all these: on an agentic information system, what does the selection of mannequin really purchase, and what does it price? A common benchmark can’t reply that, as a result of the reply is determined by the instruments within the mannequin’s fingers.

Two households sit within the comparability on objective. gpt-4o-mini and gpt-4.1 are non-reasoning fashions: they reply with out an inside deliberation price range, which makes them quick and low-cost. gpt-5-mini and gpt-5 are reasoning fashions: they spend tokens considering earlier than they act, which is precisely the conduct an interleaved software loop rewards – and precisely what you pay for. The attention-grabbing comparisons should not simply who scores highest, however who really makes use of the vocabulary the system provides, and who respects the contradiction register when the composer is switched off.

Config C, the complete agent with the gate on, the identical fifteen questions, 4 fashions:

Correctness is identical blunt key phrase proxy from part 15, so learn the 2 correctness columns as directional and the opposite 4 as onerous numbers. Three issues fall out, and solely one in all them is the one all people expects.

The fashions genuinely differ in whether or not they use the vocabulary. Device calls climb from 3.3 on gpt-4o-mini to 7.1 on gpt-5 – the stronger and reasoning-family fashions attain for resolve_entity, traverse and timeline way more readily, whereas gpt-4o-mini tends to fireside one or two searches and reply. That’s the benchmark’s actual topic: a information system’s worth is barely realized if the mannequin within the seat really picks up the instruments, and the most cost effective mannequin picks up the fewest. You aren’t simply shopping for high quality once you transfer up the mannequin ladder; you’re shopping for tool-seeking behaviour, which is the factor this entire structure is determined by.

Value scales steeply and doesn’t observe high quality previous some extent. gpt-5 prices about 3.5x the tokens and 3x the latency of gpt-4o-mini – and on this job it doesn’t earn the premium: its multi-hop rating (0.75) sits beneath gpt-5-mini’s (0.92) at almost double the tokens. The candy spot for this agentic information system, on this corpus, is unambiguously gpt-5-mini: the perfect multi-hop solutions, wholesome tool-seeking, average price. gpt-4.1 is the robust quick various – highest single-hop rating, half the latency of gpt-5-mini, and a genuinely good selection for a single-hop-heavy workload. gpt-4o-mini is quick and low-cost however too shallow a tool-user to belief with the multi-hop work the agent exists for. gpt-5 is the one I might not attain for right here: it causes and spends probably the most and, on this technique, returns the least per token.

Governance was model-independent. Each mannequin offered each positions on each contested query, gate on. The refusal column is a flat 3/3 not as a result of the composer compelled it uniformly (part 16 confirmed it fired on two of three), however as a result of the contradiction-aware immediate travelled throughout all 4 households intact. No matter else the mannequin selection modifications, at this scale it didn’t change whether or not the system knew when to refuse. That’s the one column a purchaser most needs to be boring, and it was.

Regardless of the numbers, one architectural level survives them, and it’s the motive this part is not only a curiosity: as a result of the software contract is model-independent, shifting between these fashions is a configuration change, not a rewrite. The system I constructed doesn’t marry a mannequin. That’s what the contract of part 4 was for, and it’s why the selection above is a dial you’ll be able to flip per deployment – fast-and-cheap for the escalation lane’s straightforward instances, frontier-and-careful for the contested ones – relatively than a basis you pour as soon as.

18. When to not use the agent

The ending each agent article owes its readers and few ship: the agent ought to often not run.

All the things measured factors the identical method, although not for the rationale I first wrote down. The mounted fused pipeline scored highest of the three on single-hop questions (0.95 to the agent’s 0.70), prices fewer tokens, and is predictable – and it turned out to be no slower than the agent, not sooner, as a result of its personal reranker is itself a reasoning-model name. So the mounted pipeline is the proper default lane not as a result of it’s an order of magnitude cheaper on the clock, it isn’t – however as a result of it’s higher and cheaper on the questions that make up a lot of the visitors, and the agent is measurably worse on precisely these. Company earns its token premium solely the place its defining functionality, revising the retrieval plan mid-flight, is definitely wanted, which the ablation positioned exactly on the multi-hop subset. So the manufacturing form I might deploy, and the one the repository’s endpoint format now mirrors, is an escalation structure: each query enters the mounted pipeline; escalation to the agent occurs on express indicators – sequential dependency detected within the query’s construction, an inadequate flag or contradiction surfacing mid-answer, temporal comparability spanning a number of validity home windows, or a human merely asking for deeper work. And since part 17 confirmed the mannequin is a per-deployment dial, the escalation lane can run a special, extra cautious mannequin than the default lane – gpt-5-mini for the multi-hop escalations, a less expensive mannequin for the fixed-lane majority.

There’s a symmetry right here I didn’t plan and relatively get pleasure from. Half 2 retired routing inside retrieval as a result of guessing which engine a query wanted was fragile. This text reintroduces routing one degree up, between pipelines, and the distinction is that this routing choice is affordable to get improper. Misroute to the mounted lane and the failure is seen (an inadequate or contested reply that may escalate); misroute to the agent and the failure is a number of cents. The Half 2 lesson was by no means “routing is unhealthy” – it was “route the place errors are recoverable,” and it took constructing the agent to see that clearly.

19. To sum all of it up

Half 1 argued {that a} RAG system wants someplace for understanding to build up, and constructed it. Half 2 made the gathered construction one thing retrieval really walks. This text is in regards to the remaining hole between a system that retrieves nicely and a system that investigates: the flexibility to have a look at what got here again, determine it isn’t sufficient, and know exactly what to ask for subsequent.

In sensible phrases, the place this text stakes out:

The excitement says:        add an agent, solutions enhance.This design says:     an agent is bounded by its instruments' vocabulary -                      construct the vocabulary first, then measure.The loop is:          eighty bounded strains - a budget half.The instruments are:        two articles of data layer - the actual half.I predicted:          the naive agent would blow the contradiction gate.The runs stated:        at this scale, no mannequin did - the immediate held,                      and the one empty reply got here from the mounted pipeline.The agent runs:       when escalation indicators fireplace, not by default -                      it misplaced the single-hop questions and received the multi-hop.The mannequin is a dial:   gpt-5-mini was the candy spot; gpt-5 paid extra                      for much less; the contract marries none of them.

The experiments in Half III carried pre-registered predictions, and I misplaced one in all them in print, the naive agent revered the gate I used to be certain it will trample. That’s the entire motive to register predictions earlier than working: the agent dialog has no scarcity of demonstrations engineered to succeed, and what it lacks is designs that say, prematurely, what failure would appear to be, after which publish the end result that really arrives, together with the flattering one you had been braced towards, and the blunt correctness metric that might not let any configuration appear to be a triumph.

Thanks for persevering with this exploration with me throughout three articles now. The system began as a pile of paperwork with good search. It turned a information layer that accumulates understanding, then a graph that retrieval genuinely traverses, and now the query is whether or not a mannequin will be trusted to stroll that graph by itself judgment – inside bounds, below a gate it can’t speak its well past, leaving working notes a human can audit. I strongly consider that is the sincere model of the agent dialog, not whether or not fashions can act, however whether or not we will afford the vocabulary, the bounds and the governance that make their actions value trusting, and that may be a techniques query, which is precisely why it belongs to individuals who construct.

References

[1] Designing a Persistent Information Layer That Refuses to Guess – the primary article on this collection, with the accompanying repository at github.com/mcekikj/persistent-knowledge-layer

[2] Making the Information Layer a Graph You Truly Traverse – the second article on this collection, with the accompanying repository at https://github.com/mcekikj/knowledge-graph-fusion

[3] M. Cekikj, knowledge-graph-fusion – the accompanying repository (2026), GitHub

[4] P. Lewis et al., Retrieval-Augmented Technology for Information-Intensive NLP Duties (2020), NeurIPS 2020

[5] D. Edge et al., From Native to International: A Graph RAG Strategy to Question-Targeted Summarization (2024), arXiv

[6] S. Yao et al., ReAct: Synergizing Reasoning and Performing in Language Fashions (2023), ICLR 2023

[7] N. Shinn et al., Reflexion: Language Brokers with Verbal Reinforcement Studying (2023), NeurIPS 2023

[8] Mannequin Context Protocol (2024), modelcontextprotocol.io

[9] Microsoft, What’s Microsoft Foundry Agent Service? (2026), Microsoft Study

[10] Microsoft, Methods to Use Operate Calling with Azure OpenAI (2026), Microsoft Study