How Postman runs Agent Mode for 40 million builders on Amazon Bedrock

Constructing an AI agent for a demo and working one for 40 million builders are completely different engineering issues. Postman got down to construct Agent Mode, an AI-native approach to work throughout API testing, documentation, discovery, and implementation. The crew anticipated mannequin high quality and immediate design to be the toughest issues. The deeper challenges got here from integrating an agent right into a mature product with years of interface-driven assumptions, a large floor space, and specialised ideas.

On this publish, Postman and AWS describe the architectural patterns that emerged whereas making a mature product legible to an AI agent. These patterns embrace controlling software sprawl, exposing schema-based reads, and treating context somewhat than functionality as the first bottleneck.

We additionally clarify how Agent Mode makes use of Amazon Bedrock for mannequin flexibility, geographically scoped cross-Area inference, model-dependent zero information retention, and multi-tier immediate caching. Collectively, these classes may help groups transfer manufacturing brokers past prototypes.

Why Postman constructed Agent Mode

Agent Mode is Postman’s portal for working with the product in an AI-native method throughout testing, documentation, discovery, and implementation. Postman has advanced over 11 years, and builders and customers discovered to find info by the interface by increasing sidebars, checking tabs, and opening requests. Re-engineering that consciousness for an agent surfaced structural assumptions within the product’s APIs, person expertise, and distribution of product data. An agent causes over information somewhat than navigating a display. Determine 1 illustrates how Agent Mode works immediately towards the applying.

Determine 1: Agent Mode works immediately towards the Postman software. On this instance, it opens a pull request and proposes subsequent steps with out requiring the person to navigate by the interface

Agent Mode runs on Amazon Bedrock, which offers managed entry to basis fashions behind the agent. Supporting Postman’s international developer group creates variable, latency-sensitive demand with sharp visitors bursts. With Amazon Bedrock, Postman can scale this manufacturing workload with out working its personal model-serving infrastructure whereas retaining flexibility in mannequin choice and management over throughput, geographic processing, and value. Determine 2 offers a high-level view of the manufacturing structure earlier than the next sections look at its parts.

Determine 2: Postman Agent Mode combines client-side instruments, agent orchestration, purpose-built context, and Amazon Bedrock mannequin inference. Instruments are scoped for every process, and person approval stays a part of actions that modify software state

Human oversight is a part of the manufacturing design. Agent Mode requires person approval earlier than actions that modify software state. Postman additionally scopes obtainable instruments to the duty, selects purpose-built context, and applies model-dependent data-retention settings. These controls scale back unintended actions and pointless information publicity, whereas manufacturing testing and monitoring stay essential. As a accountable AI management, Postman makes use of Amazon Bedrock Guardrails to redact personally identifiable info earlier than it reaches the underlying giant language mannequin (LLM). Enterprise admins can flip this on in Agent Mode’s guardrail settings.

In Agent Mode, instruments outline how the agent acts inside Postman. Early on, the crew leaned towards extremely atomic instruments: small, exact actions resembling opening a request, updating one subject, or fetching a particular piece of metadata. That method supported correctness and management in early iterations, but it surely additionally revealed a number of issues.

Many real-world workflows require lengthy sequences of software calls. Even when every step was quick, the general expertise felt gradual, as a result of each motion needed to return to the mannequin earlier than the subsequent one might start. Customers watched the agent step by actions that they had mentally grouped as a single operation.

In Postman’s testing, tool-selection errors elevated as soon as the seen toolset exceeded roughly 40 instruments. The agent might name nonexistent instruments, cross incorrect arguments regardless of legitimate schemas, or choose instruments that appeared semantically cheap however have been unsuitable in context. Bigger or newer fashions decreased this habits however didn’t take away it.

Previous a sure toolset measurement, exposing extra instruments can scale back agent effectiveness. The present structure selects instruments primarily based on want and context and isolates particular person execution threads. The mannequin sees solely the instruments related to the present process. Determine 3 illustrates this dynamic choice course of.

Determine 3: The basis agent queries a vector database of software embeddings and narrows greater than 170 instruments to roughly 15 related to the request. It then fingers these instruments to a context-isolated sub-agent, so the mannequin sees solely the instruments wanted for the duty

A subtler downside was that many consumer APIs have been implicitly coupled to interface state. Instruments that changed requests wanted sure components to be open, whereas different instruments opened new tabs as uncomfortable side effects. The agent needed to open a request tab to learn it, mimicking interface interactions as a substitute of reasoning about information. Postman is actively decoupling instruments from tabs, and its Native Git characteristic makes in depth use of this method. For instance, Agent Mode can now ship requests within the background with out an open tab, though person approval remains to be required.

Builder takeaway: Deal with your software catalog as a part of the context price range. Dynamically scope the instruments uncovered to the mannequin per process, and decouple “what the agent can do” from “what the UI occurs to have open.”

Exposing schema-based reads

For merchandise such because the API Catalog, Postman consolidated a number of slim views right into a single question software. These merchandise expose structured information resembling service uptime, take a look at outcomes, and endpoint response instances throughout many providers.

Given the schemas of the underlying ClickHouse tables, the agent can generate complicated queries with joins and WHERE clauses. This considerably reduces the variety of distinct instruments wanted to reply an evaluation query:

SELECT toString(service_id) AS service_id,
    countMerge(total_events_state) AS total_requests,
    countMerge(error_events_state) AS total_errors,
    spherical(countMerge(error_events_state) * 100.0
        / countMerge(total_events_state), 4) AS error_rate_pct,
    avgMerge(avg_latency_state) AS avg_latency_ms,
    quantileMerge(0.95)(p95_latency_state) AS p95_latency_ms
FROM http_events_summary_1d
WHERE service_id IN ('...listing of service IDs')
    AND bucket_1d >= as we speak() - 7
GROUP BY service_id
HAVING p95_latency_ms < 100
    AND total_requests > 0
ORDER BY error_rate_pct DESC;

With this method, the engineering job shifts from constructing a software per query to modeling the information nicely as soon as. The agent can then generate a far wider number of queries than the crew might ever have enumerated as particular person instruments.

Builder takeaway: The place you have got well-structured information, give the agent schema-aware learn entry to a question engine as a substitute of a proliferation of single-purpose learn instruments. You commerce software rely for information modeling, which produces a greater scaling curve.

Context was the true bottleneck

Postman initially assumed lacking instruments could be the most important blocker. In apply, lacking or incomplete context precipitated extra failures than lacking capabilities.

Context is the agent’s understanding of the place the person is in Postman, which entities are lively, and what state has already been established. When that context was unsuitable or absent, even right instruments grew to become ineffective. Determine 4 distinguishes the 2 types of context equipped to the agent.

Determine 4: Two sorts of context feed the agent. Broad, shallow background context is gathered mechanically and minified for the immediate. Deep, targeted chosen context is chosen by the person and routed by a devoted handler for every entity kind. Every handler distills the entity into the data the agent wants

The problem was structural. Over 11 years, builders and customers discovered to search out info by the interface. Re-engineering that consciousness for an agent required a number of iterations to find out what mattered for every workflow and what was noise. Serializing the present interface information mannequin didn’t produce helpful context as a result of these objects have been formed for rendering and information switch, not reasoning. Postman due to this fact constructed devoted context handlers that distilled every entity into what the agent wanted to know.

As extra objects gained handlers, truncation grew to become the subsequent downside. Many fields include open-ended user-generated information, together with request descriptions, OpenAPI specs, and request payloads. This information can crowd the context window. Managing the context price range fastidiously is important at scale and helps the filesystem-backed method the crew is exploring, the place every handler doesn’t want customized truncation and enlargement logic.

Builder takeaway: Don’t feed the mannequin your rendering information mannequin. Construct purpose-shaped context handlers and deal with the context window as a scarce, actively managed price range. Noise crowds out sign lengthy earlier than the mannequin reaches its restrict.

Placing all of it collectively

As Agent Mode advanced, it grew to become clear the system needed to mixture three distinct parts, every fixing a unique downside.

1. Consumer-side instruments dwell within the Postman software and symbolize the ultimate actions the agent can take, resembling opening requests, modifying settings, operating collections, and inspecting authentication. Agent Mode additionally makes use of server-side instruments for capabilities resembling net search and agent-loop administration, however most instruments function on the Postman software.
2. Generic agent directions outline system-level habits, together with how proactive Agent Mode needs to be, the way it communicates uncertainty, and what baseline product data it carries.
3. A data base makes use of a Retrieval Augmented Technology (RAG) method. Postman has a big product floor that features a number of request protocols, mock servers, screens, documentation, the API Community, workspace governance, variables, helpers, code technology, request settings, and assortment runs.

Encoding all of this in static prompts was not possible, and most of it’s irrelevant to a given question. For preliminary seeding, the crew used Postman’s Studying Middle to generate concise feature-specific articles. At runtime, Agent Mode selects data articles primarily based on the incoming question and obtainable context. For instance, when a person selects a mock server, Agent Mode injects the associated article mechanically. This retains the agent light-weight by default whereas offering depth when wanted. The data base evolves with the applying, so groups can ship Agent Mode documentation with new options.

Operating Agent Mode on Amazon Bedrock

The three parts beforehand described resolve to the identical runtime motion: an inference name to a basis mannequin (FM). At Postman’s scale, visitors is bursty and developer-driven. Routing, caching, and geographic processing controls assist Postman accommodate visitors bursts, handle inference value, and handle workload-specific processing necessities. Amazon Bedrock offers 4 capabilities that matter most right here.

Mannequin flexibility throughout the Claude household

Agent Mode isn’t tied to 1 mannequin. Via Amazon Bedrock mannequin inference APIs, Postman can entry supported Anthropic Claude fashions and route every workload to an applicable mannequin. A sooner mannequin can serve high-volume, latency-sensitive interactions, whereas a bigger mannequin can deal with complicated reasoning the place high quality issues greater than value. Transferring between supported Claude fashions is primarily a configuration change somewhat than a brand new integration. This flexibility immediately helps the tool-sprawl and context challenges described earlier. In Postman’s testing, newer and bigger fashions decreased software hallucinations, and Postman can undertake supported fashions with out rebuilding the combination. See supported fashions by AWS Area in Amazon Bedrock.

Cross-Area inference for prime throughput

Developer visitors is spiky, and provisioning for peak demand in a single AWS Area will be pricey. Agent Mode makes use of Amazon Bedrock cross-Area inference to mechanically route requests among the many vacation spot Areas outlined by an inference profile. At runtime, the applying passes the chosen inference profile ID or Amazon Useful resource Identify (ARN) because the modelId in Converse or InvokeModel. The profile, relevant AWS Identification and Entry Administration (IAM) and repair management insurance policies, and quotas should allow each vacation spot Area that Bedrock may choose.

1. Geographic inference profiles route requests solely amongst supported Areas inside an outlined geography, resembling america or European Union. This selection combines elevated throughput with a configured geographic processing boundary.
2. World inference profiles can route requests amongst supported vacation spot Areas worldwide to supply extra throughput throughout visitors bursts. They’re applicable solely when the workload doesn’t require a geographically constrained processing boundary.

Postman can choose the inference profile per workload: a worldwide profile for max obtainable throughput or a geographic profile when processing should stay throughout the profile’s outlined geography. This selection is specific within the modelId used for every Bedrock inference request.

# Schematic Converse request
response = bedrock_runtime.converse(
    modelId="",
    messages=messages,
    system=system_blocks,
)

Information residency and enterprise controls

For enterprise clients, permitted processing geography will be as essential as throughput. Geographic inference profiles constrain Bedrock routing to the profile’s supported vacation spot Areas throughout the chosen geography. This doesn’t imply that inference runs inside Postman’s personal AWS setting. Amazon Bedrock processes requests within the eligible AWS Areas for that profile, with information encrypted in transit and at relaxation. AWS states that Bedrock doesn’t use prompts and completions to coach AWS fashions or distribute them to 3rd events. Postman has configured zero information retention with data_retention_mode set to none for supported Agent Mode fashions. Availability and habits are model-dependent, so every manufacturing mannequin should be checked towards the present Amazon Bedrock data-protection and retention documentation.

Immediate caching to maintain prices in verify

A manufacturing agent resends substantial steady context on every flip, together with system directions, generic agent habits, a core software set, chosen data, and dialog context. Reprocessing the unchanged prefix on each request provides avoidable latency and value.

Agent Mode makes use of Amazon Bedrock immediate caching to reuse steady immediate prefixes. The near-immutable core, together with the system immediate, agent directions, and core software definitions, makes use of a one-hour cache checkpoint. Extra variable context makes use of a five-minute checkpoint that refreshes on a cache hit. Bedrock requires the longer-lived checkpoint to look earlier than the shorter-lived checkpoint. The shorter tier fits interactive classes as a result of idle context expires, whereas the one-hour tier can amortize its larger cache-write value throughout many reads. Cache advantages and supported TTLs rely on the chosen mannequin. Groups can confirm habits by the cacheReadInputTokens and cacheWriteInputTokens utilization fields and measure time to first token for their very own workloads.

# Schematic cache checkpoints in Converse content material blocks
{"cachePoint": {"kind": "default", "ttl": "1h"}}  # steady core
{"cachePoint": {"kind": "default", "ttl": "5m"}}  # variable layer

Builder takeaway: Deal with inference as a routing-and-caching downside, not solely a model-selection choice. Choose the Claude mannequin per workload, select the suitable cross-Area inference profile, and cache the steady immediate prefix with TTLs that match how steadily every layer modifications.

Finest practices for scaling brokers in manufacturing

Distilled from Postman’s journey, for builders engaged on Amazon Bedrock:

1. Price range instruments as fastidiously as tokens. Dynamically choose the instruments uncovered per process. In Postman’s testing, tool-selection errors elevated because the seen toolset grew to become giant.
2. Favor schema-aware reads over software proliferation. Mannequin your information nicely and let the agent question it.
3. Decouple agent actions from interface state. If a software requires an open tab, the agent is navigating the interface somewhat than reasoning immediately over information.
4. Engineer context intentionally. Function-built context handlers beat serializing your rendering mannequin each time.
5. Handle the context window as a scarce useful resource. Truncation and enlargement technique is a first-class design downside, not an afterthought.
6. Ship docs with options. A RAG data base solely stays helpful if it evolves in lockstep with the product.
7. Route and cache on Bedrock. Match every workload to the suitable Claude mannequin, select cross-Area inference primarily based on throughput and geographic necessities, and apply tiered caching to steady immediate prefixes.

Conclusion

Constructing Agent Mode required Postman to confront the hole between giant language mannequin capabilities and the construction of mature merchandise: interface assumptions, coupled purchasers, sprawling software catalogs, and data distributed throughout documentation and groups. Dynamic software choice, schema-based reads, and deliberate context engineering emerged as repeatable patterns on the scale of Postman’s developer group. Amazon Bedrock offers the managed mannequin entry, cross-Area inference, model-dependent retention controls, and immediate caching that assist the manufacturing structure.

Whether or not you’re constructing your first agent or scaling an current one, these patterns may help groups keep away from widespread agent-integration and scaling challenges.

To be taught extra, see the Amazon Bedrock documentation, together with steering for cross-Area inference, immediate caching, and information safety and retention. For associated implementation steering, learn Successfully use immediate caching on Amazon Bedrock and Amazon Bedrock pronounces international cross-Area inference for elevated throughput on the AWS Machine Studying Weblog. To discover the product, see the Postman Agent Mode documentation.

Postman’s manufacturing implementation is proprietary and isn’t obtainable as a public pattern repository.

In regards to the authors