Automate legacy net purposes with Amazon Bedrock AgentCore Browser Device

Enterprises throughout healthcare, manufacturing, retail, and monetary providers battle to automate legacy net purposes that demand human-like interplay past what customary Robotic Course of Automation (RPA) can present at scale. Amazon Bedrock AgentCore Browser Device, mixed with Strands Brokers, addresses this hole with a totally managed browser service that lets AI brokers drive these legacy interfaces by means of safe, remoted classes. With the vast majority of enterprises nonetheless operating important workloads on legacy know-how, and with most of these programs exposing solely HTML rendered by server-side middleware somewhat than fashionable APIs, high-volume guide knowledge entry stays a major price heart and a barrier to digital transformation.

Think about a state of affairs acquainted to enterprises throughout industries: a big insurance coverage firm processes tens of hundreds of operational adjustments yearly throughout legacy coverage administration programs, together with plan modifications, claims changes, protection updates, and endorsements. With devoted employees navigating complicated net interfaces generated by legacy server-side middleware, the group faces vital annual losses from guide errors alone. Every change requires navigating multi-step workflows, getting into knowledge throughout a number of screens, and validating towards enterprise guidelines. These duties devour hundreds of hours. This sample repeats throughout industries wherever legacy net purposes stay the spine of important enterprise operations.

The Amazon Bedrock AgentCore Browser Device addresses these limitations by offering a totally managed, cloud-based browser service that AI brokers use to work together with legacy net interfaces by means of safe, remoted browser classes. The Browser Device makes use of Playwright integration by means of WebSocket-based Chrome DevTools Protocol (CDP) connections, so AI brokers can work together with legacy net purposes no matter their underlying know-how stack. The Browser Device runs a managed Chromium occasion within the cloud. The goal legacy utility must be accessible over HTTP or HTTPS, no matter which browser it was initially designed for.

Mixed with Strands Brokers for model-driven orchestration, organizations can implement refined automation workflows that scale from single-step automations to complicated multi-agent workflows. The answer integrates with Amazon Bedrock basis fashions (FMs) by means of Amazon Bedrock AgentCore runtime, offering session-isolated safety with AWS Identification and Entry Administration (IAM) controls and full audit trails. This may also help corporations modernize important workflows whereas supporting their regulatory compliance necessities and preserving human oversight.

On this submit, we stroll by means of a reference implementation of an AI-powered digital employee constructed with Amazon Bedrock AgentCore Browser Device and Strands Brokers. We cowl the structure, the important thing design choices, and an enterprise blueprint with terraform deployment. The whole supply code is out there on GitHub.

Understanding the enterprise and technical problem

Enterprise RPA implementations usually face three vital technical challenges that conventional automation approaches battle to handle. These are legacy net utility integration complexity, regulatory compliance necessities in regulated industries, and scalability limitations that stop profitable manufacturing deployment.

Legacy net utility complexity

Returning to our insurance coverage firm state of affairs, the coverage administration system, like many legacy purposes throughout industries, was constructed a long time in the past on server-side middleware that generates HTML, CSS, and JavaScript for the browser. These programs lack fashionable REST APIs, requiring automation options to work together by means of net interfaces initially designed for human customers. The purposes function complicated multi-step workflows, dynamic type validation, and session-dependent state administration that conventional RPA bots can’t reliably navigate. Documentation gaps after a long time of modifications create further complexity, with important enterprise logic present solely in institutional data somewhat than accessible system specs.

Authentication mechanisms compound the issue. Some legacy purposes require multi-factor authentication (MFA), some use proprietary single sign-on (SSO), and plenty of depend on session tokens with unpredictable expiration guidelines. Guidelines-based RPA bots battle right here, which is why manufacturing deployments find yourself with an extended tail of guide interventions. In our insurance coverage state of affairs, operators navigate a fancy net interface to make plan modifications. They replace protection quantities, modify beneficiaries, and course of endorsements, every requiring a number of screens and validation steps {that a} brittle, rules-based bot can not reliably deal with when the UI renders otherwise than anticipated.

Compliance and audit necessities

Regulated industries should implement complete audit trails that seize consumer identification, timestamps, knowledge modifications, and system interactions for GDPR, HIPAA, and monetary rules. Technical implementation requires immutable logging programs, encrypted knowledge transmission, and role-based entry controls that conventional RPA programs can not present with out in depth customized growth. Monetary rules mandate tamper-proof file retention for six years with rapid accessibility for the newest 90 days. These necessities necessitate refined logging architectures that almost all RPA programs lack.

Each coverage modification on this state of affairs have to be traceable. Meaning capturing who initiated the change, what was modified, when, and whether or not it was accepted. An AI-powered digital employee should present the identical degree of auditability as a human operator, if no more.

Scalability and brittleness

Conventional RPA options face basic architectural constraints that stop enterprise-scale success. UI-based automation creates brittle dependencies on display layouts, component positioning, and utility timing that break with minor system updates. The principles-based nature of conventional RPA can’t deal with exceptions, dynamic content material, or cognitive decision-making required for complicated enterprise processes. Efficiency limitations compound these points. Conventional RPA bots require devoted digital machines or bodily programs, creating useful resource overhead and scaling constraints. Session administration turns into complicated when dealing with a number of concurrent processes, and error restoration mechanisms are restricted to predefined eventualities somewhat than adaptive problem-solving.

For our insurance coverage firm, the group processes tens of hundreds of plan adjustments yearly. Scaling conventional RPA to deal with peak volumes corresponding to enrollment durations, regulatory adjustments, and year-end processing whereas sustaining reliability proved impractical. The group wanted an strategy that might scale elastically and get better intelligently when a web page renders in an surprising state.

Resolution structure

Addressing these challenges requires automation that may deal with JavaScript-heavy net interfaces, isolate classes between processes, expose programmatic management by means of APIs, log each interplay immutably, defer to a human when confidence is low, and scale elastically. The reference implementation meets these necessities by means of a small set of elements with tightly scoped tasks. These embrace a React single-page app for the operator, a TLS-terminating proxy that solves a selected browser limitation, a Python employee operating Strands Brokers on Amazon Bedrock AgentCore runtime, and the managed browser setting supplied by Amazon Bedrock AgentCore Browser Device. Amazon Cognito indicators the operator in and points the JWT that flows by means of the stack. The structure is designed to assist enhance automation return on funding whereas sustaining safety, compliance, and human oversight.

Determine 1: Resolution structure

The numbered steps within the diagram hint the end-to-end circulation:

1. Load UI. The consumer’s browser masses the React chat interface from Amazon CloudFront, backed by a personal Amazon Easy Storage Service (Amazon S3) bucket internet hosting the static UI belongings.
2. OIDC Login. The consumer authenticates by means of Amazon Cognito (configured with SPA and machine-to-machine app shoppers) by means of the usual OpenID Join (OIDC) sign-in circulation.
3. WSS. The browser opens a WebSocket connection by means of an AWS Software Load Balancer (ALB) that terminates TLS utilizing an AWS Certificates Supervisor (ACM) certificates.
4. Plain HTTP. The ALB forwards the WebSocket connection to an NGINX reverse proxy operating as an AWS Fargate process in a personal subnet. The proxy extracts the JWT from the question string and prepares it for the following hop.
5. WSS Auth Bearer. The NGINX proxy forwards the connection to Amazon Bedrock AgentCore runtime with the JWT within the Authorization header. AgentCore runtime validates the token by means of its JWT authorizer and routes to the browser-agent container operating Strands Brokers, VisualBrowserTool, and the human-in-the-loop handler.
6. InvokeModel. The agent calls a vision-capable basis mannequin on Amazon Bedrock to research screenshots and decide the following browser motion.
7. CDP / WSS (Playwright). The agent drives an remoted Chrome occasion in Amazon Bedrock AgentCore Browser Device over the Chrome DevTools Protocol. The browser navigates the goal legacy net purposes on the client’s community.
8. PutObject. The agent shops session transcripts and screenshots to an Amazon S3 bucket, generates a pre-signed URL for every screenshot, and streams the URL again to the consumer’s browser over the prevailing WebSocket connection.
9. Pre-Signed URL Get. The consumer’s browser fetches the screenshot picture immediately from S3 utilizing the pre-signed URL, rendering it within the chat UI alongside the agent’s reasoning hint. Picture bytes don’t journey by means of the WebSocket. Solely the short-lived URL does.

Supporting infrastructure contains Amazon Elastic Container Registry (Amazon ECR) (internet hosting the NGINX gateway and browser-agent container photographs) and Amazon CloudWatch (audit logging and observability).

Amazon Bedrock AgentCore Browser Device gives a totally managed, cloud-based browser service. AI brokers work together with legacy net interfaces by means of safe, remoted browser classes. Every session runs in a devoted, remoted setting with its personal CPU, reminiscence, and filesystem. When a session completes, the setting is terminated and the state is sanitized.

Brokers join by means of Playwright over WebSocket-based CDP for programmatic management of net purposes, together with JavaScript-heavy interfaces, dynamic kinds, and multi-step workflows, whatever the underlying know-how stack.

Three capabilities immediately handle the challenges described earlier:

• Browser profiles persist authentication state throughout classes. Authenticate as soon as, or have a human operator carry out the preliminary login by means of live-view, and subsequent classes resume already signed in, assuaging repeated SSO and MFA flows.
• Proxy configuration routes site visitors by means of company proxy infrastructure with domain-based routing and credentials on AWS Secrets and techniques Supervisor, making inner legacy purposes behind company networks or IP-allowlisted portals reachable.
• Session recording captures browser interactions, together with clicks, type inputs, and web page navigation, and shops them in Amazon S3. Mixed with AWS CloudTrail logging, this helps the audit path necessities of monetary rules.

Returning to our insurance coverage instance, the Browser Device navigates the coverage administration system precisely as a human operator would. It indicators in with profile-persisted session state, enters modifications throughout a number of screens, and submits adjustments, all inside a safe, remoted session that produces a whole, replayable audit file.

Amazon Bedrock basis fashions and Strands Brokers orchestration

Amazon Bedrock gives the multi-modal basis mannequin that drives the agent’s determination loop. The reference implementation makes use of a vision-capable basis mannequin on Amazon Bedrock by means of cross-region inference to research screenshots of the browser’s present state and determine what motion to take subsequent. That is the core functionality that differentiates the answer from conventional RPA. The mannequin sees the web page, causes about what’s on display, and determines the following browser motion, whether or not that could be a click on, a fill, a scroll, or a pause for human affirmation.

Strands Brokers is the orchestration framework that turns the mannequin’s choices into concrete software calls. Fairly than following a hard and fast script, the mannequin drives its personal conduct by means of a ReAct (Purpose + Act) loop. The mannequin captures a screenshot, analyzes the web page visually, decides which software to name, executes it, observes the end result, and repeats. The framework helps a number of orchestration patterns, together with Brokers-as-Instruments for hierarchical delegation, Graphs for structured workflows, and collaborative patterns for complicated multi-step processes.

When an operator gives pure language directions corresponding to “open coverage #12345, replace the protection quantity to $500,000, and submit for approval,” the mannequin decomposes these into sequential browser actions. After every motion, the mannequin takes a screenshot, interprets the brand new web page state visually, and decides the following step. If a affirmation dialog seems unexpectedly or a subject validates otherwise than anticipated, the mannequin causes by means of the brand new state somewhat than failing. This alleviates the brittleness that makes conventional RPA impractical at scale.

Utilized to our insurance coverage state of affairs, a single pure language instruction can drive a whole plan modification workflow. The mannequin visually navigates to the best coverage, identifies type fields from the screenshot, enters adjustments throughout a number of screens, handles surprising validation prompts, and submits for approval. No human must script every step or preserve brittle selectors because the UI evolves.

Human-in-the-loop integration

The operator sees the automation because it occurs. The agent streams reasoning traces and screenshots to the chat interface in actual time. When the mannequin determines it wants human enter earlier than, for instance confirming that the right coverage was opened earlier than submitting a change, it calls the handoff_to_user software, which pauses the automation and presents the operator with a query and the present screenshot.

The browser session stays alive whereas the agent waits for a response (as much as 300 seconds by default). The operator can approve, reject, present further directions, or ask the agent to strive a distinct strategy. As soon as the response arrives, the mannequin resumes the ReAct loop with the operator’s enter. It takes a contemporary screenshot to evaluate the present state and continues from there. If the operator doesn’t reply inside the timeout window, the mannequin decides whether or not to retry, try another strategy, or abort gracefully.

Within the context of our insurance coverage workflow, this implies the digital employee pauses earlier than important actions corresponding to submitting a plan modification, confirming a beneficiary change, or processing an endorsement that exceeds a threshold. This offers the operator the chance to confirm accuracy on display earlier than the change turns into everlasting. The operator stays in management with no need to drive each step manually.

Safety and compliance

The structure gives session isolation, IAM controls for entry administration, and ephemeral classes the place the state is cleared after every use. The reference implementation makes use of Amazon Cognito because the identification supplier, with Microsoft Entra ID and Okta supported by swapping the OIDC discovery URL. Requirements-aligned OIDC suppliers work with the identical configuration floor.

For our insurance coverage firm, this implies each coverage modification carried out by the digital employee is absolutely traceable, with session recordings saved in Amazon S3 and actions logged by means of Amazon CloudWatch. This helps the retention necessities of monetary rules.

Implementation walkthrough

The reference implementation ships as a Python employee constructed on Strands Brokers, deployed on Amazon Bedrock AgentCore runtime. The agent makes use of the AgentCore Browser Device SDK to create remoted browser classes by means of the Strands AgentCoreBrowser base class. The whole supply code is out there within the GitHub repository.

How the agent drives the browser

The next diagram traces a single operator instruction by means of the complete automation loop. The operator points a pure language command, the agent begins a browser session, and the inspiration mannequin enters a ReAct cycle. It captures screenshots, analyzes web page state visually, decides the following motion, executes it by means of Playwright, and repeats till the duty is full.

Determine 2: Chat circulation, from operator instruction by means of the ReAct loop

When the mannequin determines it wants human affirmation earlier than a important step, corresponding to submitting a type or confirming a file choice, it pauses the automation and presents the operator with the present screenshot and a query. The operator responds, and the mannequin resumes.

Determine 3: Human-in-the-loop, the place the agent pauses for an operator determination

Key design choices

Two implementation decisions are price calling out as a result of they immediately have an effect on manufacturing reliability:

Occasion loop isolation. The browser agent runs alongside a WebSocket server that streams updates to the operator. The upstream browser software’s threading mannequin interferes with the server’s occasion loop, inflicting connections to drop after about 15 seconds. The reference implementation overrides this so browser operations run on their very own remoted loop, preserving the operator’s connection secure all through the session. See visual_browser_tool.py for the complete implementation.

Semantic locators over CSS selectors. Conventional RPA bots break when UI parts transfer or get renamed. The semantic_action software takes a distinct strategy. The mannequin describes what it sees (“click on the button labeled Submit”) and the software maps that to Playwright’s semantic Locator API utilizing case-insensitive fuzzy matching. This makes the automation resilient to the minor UI variations widespread in legacy purposes. That is precisely the brittleness downside the answer got down to remedy. See semantic_action_tool.py for the complete implementation.

Deployment and testing

The complete resolution deploys with a single Terraform stack. The next steps take you from git clone to a working chat URL.

Conditions

You have to have the next:

• An AWS account with Amazon Bedrock mannequin entry enabled for a vision-capable basis mannequin.
• AWS Command Line Interface (AWS CLI) v2 configured with credentials.
• Docker 24+.
• Terraform 1.5+.
• Node 20+ and Yarn.
• Python 3.12.

Deploy

git clone  && cd 
cp deployment/terraform/stacks/all/terraform.tfvars.pattern 
deployment/terraform/stacks/all/terraform.tfvars
# Edit terraform.tfvars - set aws_region, project_name, cognito_domain_prefix

terraform -chdir=deployment/terraform/stacks/all init
terraform -chdir=deployment/terraform/stacks/all apply

The stack provisions the Cognito Person Pool, CloudFront distribution, ECR repositories, ECS proxy service, AgentCore runtime with JWT authorizer, and S3 session storage. Terraform provisioners construct and push each container photographs, compile the React UI, sync it to S3, and invalidate CloudFront, multi function cross. First apply takes roughly 20–half-hour, most of which is CloudFront propagation and container picture builds.

Check the answer

After deployment completes, Terraform outputs the CloudFront URL. Open it in a browser, register with a Cognito consumer, and check out a immediate corresponding to:

“Open https://httpbin.org/kinds/submit, fill the client title with John Smith, decide medium pizza, examine bacon topping, and submit the shape.”

You must see the reasoning hint stream into the chat, screenshots seem alongside every step, and a human-in-the-loop immediate earlier than the ultimate submit. For added check eventualities, instance prompts, and screenshots of the UI in motion, see the repository README.

Clear up

To keep away from ongoing prices, destroy the infrastructure if you’re performed:

terraform -chdir=deployment/terraform/stacks/all destroy

This removes the provisioned assets together with the Cognito Person Pool, CloudFront distribution, ECR repositories, ECS providers, AgentCore runtime, and S3 buckets.

Conclusion and subsequent steps

The AI-powered digital employee described on this submit makes that case achievable. The place conventional RPA is brittle as a result of it {couples} to actual UI layouts, the mixture of the vision-capable basis fashions in Amazon Bedrock, the model-driven orchestration of Strands Brokers, and the remoted browser classes of AgentCore Browser Device produces automation that causes over web page state, recovers from surprising branches, and defers to a human when confidence is low. It does this with out sacrificing the audit path that regulated industries require.

The reference implementation leaves room to develop. In case your workflow wants cross-session context, Amazon Bedrock AgentCore reminiscence gives managed short-term and long-term recall. If you must combine present REST APIs as instruments the agent can name alongside its browser actions, Amazon Bedrock AgentCore Gateway handles that with out writing customized Strands instruments. For portals that require customized browser conduct, Chrome extensions may be loaded into AgentCore Browser Device classes at creation time. The whole supply code is out there on GitHub.

In regards to the authors