Getting began with pc use in Amazon Bedrock Brokers

Pc use is a breakthrough functionality from Anthropic that enables basis fashions (FMs) to visually understand and interpret digital interfaces. This functionality permits Anthropic’s Claude fashions to determine what’s on a display, perceive the context of UI components, and acknowledge actions that must be carried out reminiscent of clicking buttons, typing textual content, scrolling, and navigating between functions. Nevertheless, the mannequin itself doesn’t execute these actions—it requires an orchestration layer to soundly implement the supported actions.

At present, we’re saying pc use assist inside Amazon Bedrock Brokers utilizing Anthropic’s Claude 3.5 Sonnet V2 and Anthropic’s Claude Sonnet 3.7 fashions on Amazon Bedrock. This integration brings Anthropic’s visible notion capabilities as a managed device inside Amazon Bedrock Brokers, offering you with a safe, traceable, and managed solution to implement pc use automation in your workflows.

Organizations throughout industries wrestle with automating repetitive duties that span a number of functions and methods of report. Whether or not processing invoices, updating buyer data, or managing human useful resource (HR) paperwork, these workflows typically require staff to manually switch info between completely different methods – a course of that’s time-consuming, error-prone, and troublesome to scale.

Conventional automation approaches require customized API integrations for every utility, creating important growth overhead. Pc use capabilities change this paradigm by permitting machines to understand present interfaces simply as people.

On this submit, we create a pc use agent demo that gives the important orchestration layer that transforms pc use from a notion functionality into actionable automation. With out this orchestration layer, pc use would solely determine potential actions with out executing them. The pc use agent demo powered by Amazon Bedrock Brokers offers the next advantages:

• Safe execution atmosphere – Execution of pc use instruments in a sandbox atmosphere with restricted entry to the AWS ecosystem and the online. It’s essential to notice that at the moment Amazon Bedrock Agent doesn’t present a sandbox atmosphere
• Complete logging – Potential to trace every motion and interplay for auditing and debugging
• Detailed tracing capabilities – Visibility into every step of the automated workflow
• Simplified testing and experimentation – Diminished threat when working with this experimental functionality via managed controls
• Seamless orchestration – Coordination of advanced workflows throughout a number of methods with out customized code

This integration combines Anthropic’s perceptual understanding of digital interfaces with the orchestration capabilities of Amazon Bedrock Brokers, creating a robust agent for automating advanced workflows throughout functions. Moderately than construct customized integrations for every system, builders can now create brokers that understand and work together with present interfaces in a managed, safe means.

With pc use, Amazon Bedrock Brokers can automate duties via fundamental GUI actions and built-in Linux instructions. For instance, your agent may take screenshots, create and edit textual content information, and run built-in Linux instructions. Utilizing Amazon Bedrock Brokers and suitable Anthropic’s Claude fashions, you need to use the next motion teams:

• Pc device – Permits interactions with consumer interfaces (clicking, typing, scrolling)
• Textual content editor device – Offers capabilities to edit and manipulate information
• Bash – Permits execution of built-in Linux instructions

Resolution overview

An instance pc use workflow consists of the next steps:

1. Create an Amazon Bedrock agent and use pure language to explain what the agent ought to do and the way it ought to work together with customers, for instance: “You might be pc use agent able to utilizing Firefox net browser for net search.”
2. Add the Amazon Bedrock Brokers supported pc use motion teams to your agent utilizing CreateAgentActionGroup API.
3. Invoke the agent with a consumer question that requires pc use instruments, for instance, “What’s Amazon Bedrock, are you able to search the online?”
4. The Amazon Bedrock agent makes use of the device definitions at its disposal and decides to make use of the pc motion group to click on a screenshot of the atmosphere. Utilizing the return management functionality of Amazon Bedrock Brokers, the agent the responds with the device or instruments that it desires to execute. The return management functionality is required for utilizing pc use with Amazon Bedrock Brokers.
5. The workflow parses the agent response and executes the device returned in a sandbox atmosphere. The output is given again to the Amazon Bedrock agent for additional processing.
6. The Amazon Bedrock agent continues to reply with instruments at its disposal till the duty is full.

You may recreate this instance within the us-west-2 AWS Area with the AWS Cloud Improvement Equipment (AWS CDK) by following the directions within the GitHub repository. This demo deploys a containerized utility utilizing AWS Fargate throughout two Availability Zones within the us-west-2 Area. The infrastructure operates inside a digital personal cloud (VPC) containing public subnets in every Availability Zone, with an web gateway offering exterior connectivity. The structure is complemented by important supporting companies, together with AWS Key Administration Service (AWS KMS) for safety and Amazon CloudWatch for monitoring, making a resilient, serverless container atmosphere that alleviates the necessity to handle underlying infrastructure whereas sustaining strong safety and excessive availability.

The next diagram illustrates the answer structure.

On the core of our answer are two Fargate containers managed via Amazon Elastic Container Service (Amazon ECS), every protected by its personal safety group. The primary is our orchestration container, which not solely handles the communication between Amazon Bedrock Brokers and finish customers, but in addition orchestrates the workflow that allows device execution. The second is the environment container, which serves as a safe sandbox the place the Amazon Bedrock agent can safely run its pc use instruments. The atmosphere container has restricted entry to the remainder of the ecosystem and the web. We make the most of service discovery to attach Amazon ECS companies with DNS names.

The orchestration container consists of the next parts:

• Streamlit UI – The Streamlit UI that facilitates interplay between the tip consumer and pc use agent
• Return management loop – The workflow answerable for parsing the instruments that the agent desires to execute and returning the output of those instruments

The atmosphere container consists of the next parts:

• UI and pre-installed functions – A light-weight UI and pre-installed Linux functions like Firefox that can be utilized to finish the consumer’s duties
• Device implementation – Code that may execute pc use device within the atmosphere like “screenshot” or “double-click”
• Quart (RESTful) JSON API – An orchestration container that makes use of Quart to execute instruments in a sandbox atmosphere

The next diagram illustrates these parts.

Conditions

1. AWS Command Line Interface (CLI), observe directions right here. Be certain to setup credentials, observe directions right here.
2. Require Python 3.11 or later.
3. Require Node.js 14.15.0 or later.
4. AWS CDK CLI, observe directions right here.
5. Allow mannequin entry for Anthropic’s Claude Sonnet 3.5 V2 and for Anthropic’s Claude Sonnet 3.7.
6. Boto3 model >= 1.37.10.

Create an Amazon Bedrock agent with pc use

You should use the next code pattern to create a easy Amazon Bedrock agent with pc, bash, and textual content editor motion teams. It’s essential to supply a suitable motion group signature when utilizing Anthropic’s Claude 3.5 Sonnet V2 and Anthropic’s Claude 3.7 Sonnet as highlighted right here.

import boto3
import time

# Step 1: Create the bedrock agent shopper

bedrock_agent = boto3.shopper("bedrock-agent", region_name="us-west-2")

# Step 2: Create an agent

create_agent_response = create_agent_response = bedrock_agent.create_agent(
        agentResourceRoleArn=agent_role_arn, # Amazon Bedrock Agent execution position
        agentName="computeruse",
        description="""Instance agent for pc use. 
				This agent ought to solely function on 
				Sandbox environments with restricted privileges.""",
        foundationModel="us.anthropic.claude-3-7-sonnet-20250219-v1:0",      
		instruction="""You might be pc use agent able to utilizing Firefox 
                 net browser for net search.""",
)

time.sleep(30) # watch for agent to be created

# Step 3.1: Create and fasten pc motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="ComputerActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.Pc",
    parentActionGroupSignatureParams={
        "kind": "computer_20250124",
        "display_height_px": "768",
        "display_width_px": "1024",
        "display_number": "1",
    },
)

# Step 3.2: Create and fasten bash motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="BashActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.Bash",
    parentActionGroupSignatureParams={
        "kind": "bash_20250124",
    },
)

# Step 3.3: Create and fasten textual content editor motion group

bedrock_agent.create_agent_action_group(
    actionGroupName="TextEditorActionGroup",
    actionGroupState="ENABLED",
    agentId=create_agent_response["agent"]["agentId"],
    agentVersion="DRAFT",
    parentActionGroupSignature="ANTHROPIC.TextEditor",
    parentActionGroupSignatureParams={
        "kind": "text_editor_20250124",
    },
)

# Step 3.4 Create Climate Motion Group

bedrock_agent.create_agent_action_group(
        actionGroupName="WeatherActionGroup",
        agentId=create_agent_response["agent"]["agentId"],
        agentVersion="DRAFT",
        actionGroupExecutor = {
            'customControl': 'RETURN_CONTROL',
        },
        functionSchema = {
            'features': [
                {
                    "name": "get_current_weather",
                    "description": "Get the current weather in a given location.",
                    "parameters": {
                        "location": {
                            "type": "string",
                            "description": "The city, e.g., San Francisco",
                            "required": True,
                        },
                        "unit": {
                            "type": "string",
                            "description": 'The unit to use, e.g., 
									fahrenheit or celsius. Defaults to "fahrenheit"',
                            "required": False,
                        },
                    },
                    "requireConfirmation": "DISABLED",
                }
            ]
        },
)
time.sleep(10)
# Step 4: Put together agent

bedrock_agent.prepare_agent(agentId=create_agent_response["agent"]["agentId"])

Instance use case

On this submit, we exhibit an instance the place we use Amazon Bedrock Brokers with the pc use functionality to finish an internet kind. Within the instance, the pc use agent may also change Firefox tabs to work together with a buyer relationship administration (CRM) agent to get the required info to finish the shape. Though this instance makes use of a pattern CRM utility because the system of report, the identical strategy works with Salesforce, SAP, Workday, or different methods of report with the suitable authentication frameworks in place.

Within the demonstrated use case, you may observe how effectively the Amazon Bedrock agent carried out with pc use instruments. Our implementation accomplished the shopper ID, buyer title, and e-mail by visually analyzing the excel knowledge. Nevertheless, for the overview, it determined to pick out the cell and duplicate the information, as a result of the knowledge wasn’t utterly seen on the display. Lastly, the CRM agent was used to get further info on the shopper.

Finest practices

The next are some methods you may enhance the efficiency to your use case:

Issues

The pc use function is made obtainable to you as a beta service as outlined within the AWS Service Phrases. It’s topic to your settlement with AWS and the AWS Service Phrases, and the relevant mannequin EULA. Pc use poses distinctive dangers which are distinct from customary API options or chat interfaces. These dangers are heightened when utilizing the pc use function to work together with the web. To attenuate dangers, take into account taking precautions reminiscent of:

• Function pc use performance in a devoted digital machine or container with minimal privileges to attenuate direct system exploits or accidents
• To assist forestall info theft, keep away from giving the pc use API entry to delicate accounts or knowledge
• Restrict the pc use API’s web entry to required domains to scale back publicity to malicious content material
• To implement correct oversight, hold a human within the loop for delicate duties (reminiscent of making choices that would have significant real-world penalties) and for something requiring affirmative consent (reminiscent of accepting cookies, executing monetary transactions, or agreeing to phrases of service)

Any content material that you simply allow Anthropic’s Claude to see or entry can doubtlessly override directions or trigger the mannequin to make errors or carry out unintended actions. Taking correct precautions, reminiscent of isolating Anthropic’s Claude from delicate surfaces, is crucial – together with to keep away from dangers associated to immediate injection. Earlier than enabling or requesting permissions essential to allow pc use options in your individual merchandise, inform finish customers of any related dangers, and acquire their consent as acceptable.

Clear up

If you find yourself performed utilizing this answer, be certain to scrub up all of the sources. Comply with the directions within the supplied GitHub repository.

Conclusion

Organizations throughout industries face important challenges with cross-application workflows that historically require guide knowledge entry or advanced customized integrations. The combination of Anthropic’s pc use functionality with Amazon Bedrock Brokers represents a transformative strategy to those challenges.

Through the use of Amazon Bedrock Brokers because the orchestration layer, organizations can alleviate the necessity for customized API growth for every utility, profit from complete logging and tracing capabilities important for enterprise deployment, and implement automation options shortly.

As you start exploring pc use with Amazon Bedrock Brokers, take into account workflows in your group that would profit from this strategy. From bill processing to buyer onboarding, HR documentation to compliance reporting, the potential functions are huge and transformative.

We’re excited to see how you’ll use Amazon Bedrock Brokers with the pc use functionality to securely streamline operations and reimagine enterprise processes via AI-driven automation.

Sources

To be taught extra, confer with the next sources:

In regards to the Authors

Eashan Kaushik is a Specialist Options Architect AI/ML at Amazon Net Companies. He’s pushed by creating cutting-edge generative AI options whereas prioritizing a customer-centric strategy to his work. Earlier than this position, he obtained an MS in Pc Science from NYU Tandon College of Engineering. Outdoors of labor, he enjoys sports activities, lifting, and working marathons.

Maira Ladeira Tanke is a Tech Lead for Agentic workloads in Amazon Bedrock at AWS, the place she permits clients on their journey to develop autonomous AI methods. With over 10 years of expertise in AI/ML. At AWS, Maira companions with enterprise clients to speed up the adoption of agentic functions utilizing Amazon Bedrock, serving to organizations harness the ability of basis fashions to drive innovation and enterprise transformation. In her free time, Maira enjoys touring, taking part in together with her cat, and spending time together with her household someplace heat.

Raj Pathak is a Principal Options Architect and Technical advisor to Fortune 50 and Mid-Sized FSI (Banking, Insurance coverage, Capital Markets) clients throughout Canada and america. Raj makes a speciality of Machine Studying with functions in Generative AI, Pure Language Processing, Clever Doc Processing, and MLOps.

Adarsh Srikanth is a Software program Improvement Engineer at Amazon Bedrock, the place he develops AI agent companies. He holds a grasp’s diploma in pc science from USC and brings three years of business expertise to his position. He spends his free time exploring nationwide parks, discovering new mountaineering trails, and taking part in varied racquet sports activities.

Abishek Kumar is a Senior Software program Engineer at Amazon, bringing over 6 years of useful expertise throughout each retail and AWS organizations. He has demonstrated experience in growing generative AI and machine studying options, particularly contributing to key AWS companies together with SageMaker Autopilot, SageMaker Canvas, and AWS Bedrock Brokers. All through his profession, Abishek has proven ardour for fixing advanced issues and architecting large-scale methods that serve thousands and thousands of consumers worldwide. When not immersed in know-how, he enjoys exploring nature via mountaineering and touring adventures together with his spouse.

Krishna Gourishetti is a Senior Software program Engineer for the Bedrock Brokers group in AWS. He’s keen about constructing scalable software program options that clear up buyer issues. In his free time, Krishna likes to go on hikes.