Embed a dwell AI browser agent in your React app with Amazon Bedrock AgentCore

If you construct AI-powered purposes, your customers should perceive and belief AI brokers that navigate web sites and work together with net content material on their behalf. When an agent interacts with net content material autonomously, your customers require visibility into these actions to keep up confidence and management, which they don’t presently have.

The Amazon Bedrock AgentCore Browser BrowserLiveView element addresses this problem by offering a real-time video feed of the agent’s searching session straight inside your React utility. This element, a part of the Bedrock AgentCore TypeScript SDK, streamlines the combination by embedding a dwell browser stream with three traces of JavaScript XML (JSX).

The BrowserLiveView element makes use of the Amazon DCV protocol to render the browser session, creating transparency into agent actions. Implementation requires solely a presigned URL out of your server, with out requiring you to construct streaming infrastructure.

This put up walks you thru three steps: beginning a session and producing the Reside View URL, rendering the stream in your React utility, and wiring up an AI agent that drives the browser whereas your customers watch. On the finish, you should have a working pattern utility you’ll be able to clone and run.

Why embed Reside View in your utility

Embedding Reside View inside your personal utility unlocks further worth in your customers at scale.

With an embedded Reside View, your customers comply with each navigation, kind submission, and search question because the agent performs it. They get quick visible affirmation that the agent is on the proper web page, interacting with the right components, and progressing via the workflow. This real-time suggestions loop offers finish customers direct perception into agent habits with out ready for the ultimate end result.

Customers who delegate searching duties to an AI agent are extra assured once they can observe the work. Watching the agent fill in a kind area by area is extra reassuring than receiving a textual content affirmation. For regulated workflows, visible proof of agent actions can help audit necessities.

In workflows that require human supervision, like dealing with buyer accounts and processing delicate knowledge, a supervisor can use the embedded Reside View to observe the agent in actual time and intervene if wanted, with out leaving your utility.

Organizations additionally achieve audit path help via visible proof of agent actions, which proves invaluable for compliance necessities and troubleshooting situations. Mixed with session recordings to Amazon Easy Storage Service (Amazon S3) and console-based session replay, you get each real-time remark and post-hoc evaluate.

The way it works

The combination has three parts.

The person’s net browser runs a React utility containing the BrowserLiveView element, which receives a SigV4-presigned URL and establishes a persistent WebSocket connection to obtain the DCV video stream from a distant browser session. The React utility handles video rendering and person interface presentation whereas sustaining the WebSocket connection for steady streaming.

The appliance server capabilities as an AI agent inside the Amazon Bedrock session lifecycle, orchestrating the connection between shopper browsers and cloud-hosted browser classes. It begins classes utilizing the Amazon Bedrock AgentCore API and generates SigV4-presigned URLs that grant safe, time-limited entry to the Reside View stream. This layer handles session administration, authentication, and stream distribution.

AWS Cloud hosts Amazon Bedrock AgentCore Browser and Amazon Bedrock companies that present the underlying browser automation and streaming capabilities. Amazon Bedrock AgentCore hosts the remoted cloud browser classes inside AWS Cloud and supplies each the automation endpoint (Playwright CDP) and the Reside View streaming endpoint (DCV).

The important thing effectivity benefit with this structure is that the DCV Reside View stream flows straight from Amazon Bedrock AgentCore to the person’s browser. It doesn’t cross via your utility server. Your server generates the URL and runs the agent, however the video stream is a direct WebSocket connection from AWS to the shopper. This helps reduce latency and scale back infrastructure necessities.

Determine 1: Resolution structure exhibiting the info circulation between three parts. The numbered arrows within the diagram characterize the next knowledge flows:

Arrow 1 (grey, stable): The shopper sends prompts and polls standing from the Utility Server utilizing REST.

Arrow 2 (orange, stable): The Utility Server calls the Amazon Bedrock Converse API for AI mannequin reasoning.

Arrow 3 (blue, stable): The Utility Server runs browser instruments in opposition to Amazon Bedrock AgentCore Browser utilizing Playwright Chrome DevTools Protocol (CDP).

Arrow 4 (pink, dashed): The DCV Reside View stream flows straight from Amazon Bedrock AgentCore to the Person Browser, bypassing the Utility Server.

Conditions

Earlier than you start, confirm that you’ve got the next:

Necessary: Reside View (Steps 1 and a pair of) requires solely Amazon Bedrock AgentCore permissions. It doesn’t rely on Amazon Bedrock or any particular AI mannequin. The AI agent in Step 3 makes use of the Amazon Bedrock Converse API, which requires further Amazon Bedrock permissions, however that is particular to our pattern. You’ll be able to substitute a mannequin supplier or agent framework of your alternative. Use momentary credentials from AWS IAM Id Middle or AWS Safety Token Service (AWS STS). Don’t use long-lived entry keys. Comply with the precept of least privilege when configuring AWS Id and Entry Administration (IAM) permissions.

Set up the Amazon Bedrock AgentCore TypeScript SDK:

npm set up bedrock-agentcore

For the AI agent in Step 3, you additionally want the AWS SDK for JavaScript:

npm set up @aws-sdk/client-bedrock-runtime

The code on this put up runs in two environments: the server-side code (Steps 1 and three) runs in Node.js, and the client-side code (Step 2) runs in a React utility bundled with Vite. The pattern utility on the finish of this put up packages all the pieces collectively.

Step-by-step implementation

1: Begin a browser session and generate the Reside View URL

In your utility server, use the Browser class to begin a session and generate the presigned URL. The API returns a session identifier and streaming URL, which the server converts right into a presigned URL with an outlined expiration time of 300 seconds by default. It comprises SigV4 credentials within the question parameters, so no secrets and techniques attain the browser. Go this URL to your frontend via an API endpoint.

import { Browser } from 'bedrock-agentcore/browser'

const browser = new Browser({
  area: 'us-west-2'
})
await browser.startSession({
  viewport: { width: 1920, peak: 1080 }
})

const signedUrl =
  await browser.generateLiveViewUrl()
// Ship signedUrl to your frontend by way of API

2: Render the BrowserLiveView element in your React app

In your browser, import the BrowserLiveView element from the Bedrock AgentCore TypeScript SDK and render it with the presigned URL. The element handles WebSocket connection, DCV protocol negotiation, video stream decoding, and body rendering. It auto scales to suit its mother or father container whereas preserving its side ratio. The remoteWidth and remoteHeight should match the viewport that you just set in Step 1. Mismatched values trigger cropping or black bars.

import { BrowserLiveView }
  from 'bedrock-agentcore/browser/live-view'

After including this element, the Reside View begins streaming as quickly because the presigned URL is legitimate and the browser session is lively. You must see the distant browser’s desktop seem inside the element’s container. If the container stays empty, confirm that the presigned URL hasn’t expired and that the browser session remains to be working.

3: Join an AI agent to drive browser actions

With the Reside View streaming, you want one thing fascinating to observe. The next instance makes use of the Amazon Bedrock Converse API, however Reside View is mannequin agnostic. You need to use an AI mannequin or agent framework of your option to drive the browser.

The code creates a PlaywrightBrowser shopper, which begins a brand new AgentCore Browser session and connects to it utilizing the Playwright Chrome DevTools protocol. This is similar sort of cloud browser session as Step 1 however accessed via the Playwright automation interface relatively than the Reside View interface.

The mannequin decides which browser instruments to name, together with navigate, click on, sort, getText, getHtml, and pressKey. Your server runs these instruments and feeds the outcomes again to the mannequin for the subsequent iteration.

import { BedrockRuntimeClient, ConverseCommand }
  from '@aws-sdk/client-bedrock-runtime'
import { PlaywrightBrowser }
  from 'bedrock-agentcore/browser/playwright'

const browser = new PlaywrightBrowser({
  area: 'us-west-2'
})
await browser.startSession()

// Outline browser instruments as JSON Schema
// (navigate, click on, sort, getText, and extra)

whereas (step < maxSteps) {
  const response = await bedrockClient.ship(
    new ConverseCommand({
      modelId: modelId,
      system: [{ text: systemPrompt }],
      messages,
      toolConfig: browserTools,
    })
  )

  if (response.stopReason === 'tool_use') {
    // Run browser instrument, add end result
    // to dialog, proceed loop
  } else {
    break // Last reply from mannequin
  }
}

The mannequin is configurable. You need to use Anthropic Claude, Amazon Nova, or an Amazon Bedrock mannequin that helps instrument use. Each instrument name that the mannequin makes is seen to your person via the Reside View. They see the browser navigate, the search field fill in, and the outcomes web page load.

Observe: The TypeScript SDK additionally features a Vercel AI SDK integration (BrowserTools) that wraps these browser operations as framework-native instruments.

Strive it utilizing the pattern utility

We constructed an entire pattern utility on GitHub that places Steps 1–3 collectively. The pattern features a React dashboard with the embedded Reside View, an exercise log exhibiting agent reasoning and actions, and a Fastify server working the AI agent. The agent navigates to Wikipedia, searches for a subject, reads the web page content material, and summarizes what it finds whilst you watch each step. You’ll be able to obtain it from the GitHub repository.

Determine 2: The pattern utility mid-run. The left panel reveals the BrowserLiveView element streaming a Wikipedia web page that the agent has navigated to. The suitable panel reveals the exercise log with timestamped instrument calls (navigate, getText, click on). On the backside, the immediate enter area and Launch Agent button are seen.

To clone and run the pattern utility

Full the next steps to clone and run the pattern utility.

1. Clone the repository and navigate to the pattern folder.

git clone https://github.com/awslabs/bedrock-agentcore-samples-typescript.git
cd bedrock-agentcore-samples-typescript
cd use-cases/browser-live-view-agent

2. Set up the dependencies.

3. Export your AWS credentials.

export AWS_ACCESS_KEY_ID=
export AWS_SECRET_ACCESS_KEY=
export AWS_SESSION_TOKEN=
export AWS_REGION=us-west-2

Necessary: Use momentary credentials. Don’t commit credentials to supply management.

4. Begin the applying.

5. Open http://localhost:5173, enter a immediate, and select Launch Agent.

Bundler configuration

The BrowserLiveView element makes use of the Amazon DCV Internet Consumer SDK, which ships vendored recordsdata contained in the bedrock-agentcore npm package deal. You don’t must obtain or set up DCV individually. Your Vite configuration wants three additions:

• resolve.alias factors the dcv and dcv-ui naked specifiers to the vendored SDK recordsdata.
• resolve.dedupe verifies that React and shared dependencies resolve out of your node_modules, not from the vendored path.
• viteStaticCopy copies DCV runtime recordsdata (employees, WASM decoders) to your construct output.

The pattern utility’s vite.config.ts has the entire configuration prepared to make use of. For extra particulars on the BrowserLiveView element, see the live-view supply listing within the TypeScript SDK.

Clear up sources

To keep away from incurring prices, cease the browser session and shut down the applying whenever you’re executed:

1. Within the utility UI, select Cease Session to finish the Amazon Bedrock AgentCore Browser session.
2. In your terminal, press Ctrl+C to cease the event servers.
3. In the event you created any IAM roles or insurance policies particularly for this demo, delete them from the IAM console.

Amazon Bedrock AgentCore Browser classes incur prices whereas lively. For pricing particulars, check with the Amazon Bedrock AgentCore pricing web page.

Subsequent steps

Now that you’ve got a working Reside View integration, listed below are some issues to discover.

To get began, clone the pattern utility, fill in your AWS credentials, and run npm run dev to see the complete demo in motion. For directions, check with the To clone and run the pattern utility part on this put up.

The pattern utility defaults to Anthropic Claude, however you’ll be able to swap to Amazon Nova or one other Amazon Bedrock mannequin that helps instrument use by setting the BEDROCK_MODEL_ID atmosphere variable. For an inventory of accessible fashions and their instrument use capabilities, check with the Amazon Bedrock mannequin documentation.

The React dashboard within the pattern utility is a place to begin in your personal implementation. You’ll be able to adapt the format to match your design system, combine the Reside View into an present utility, or add controls that allow customers intervene mid-workflow. For steerage on constructing React purposes with the AgentCore SDK, check with the Bedrock AgentCore TypeScript SDK documentation.

The BrowserLiveView element helps a number of cases on the identical web page, every streaming a special browser session. This functionality is beneficial for monitoring dashboards. The element’s supply code, together with scaling logic and DCV authentication circulation, is on the market within the live-view supply listing within the TypeScript SDK.

Conclusion

On this put up, you discovered the right way to use the BrowserLiveView element to embed a Reside View of an Amazon Bedrock AgentCore Browser session into your React utility. The three-step implementation and structure that streams video straight from AWS to shopper browsers makes dwell agent visualization accessible with out specialised streaming experience.

For a deeper take a look at Amazon Bedrock AgentCore Browser capabilities, check with the Amazon Bedrock AgentCore Browser documentation. In case you have suggestions or questions, open a problem within the GitHub repository.

Necessary: This pattern utility is meant for native improvement and demonstration. For manufacturing use, add authentication to your API endpoints, allow HTTPS, prohibit CORS origins, implement charge limiting, and comply with the AWS Effectively-Architected Framework safety pillar.

Concerning the authors