Use Amazon Bedrock tooling with Amazon SageMaker JumpStart fashions

At the moment, we’re excited to announce a brand new functionality that means that you can deploy over 100 open-weight and proprietary fashions from Amazon SageMaker JumpStart and register them with Amazon Bedrock, permitting you to seamlessly entry them by way of the highly effective Amazon Bedrock APIs. Now you can use Amazon Bedrock options comparable to Amazon Bedrock Information Bases and Amazon Bedrock Guardrails with fashions deployed by way of SageMaker JumpStart.

SageMaker JumpStart helps you get began with machine studying (ML) by offering absolutely customizable options and one-click deployment and fine-tuning of greater than 400 common open-weight and proprietary generative AI fashions. Amazon Bedrock is a completely managed service that gives a single API to entry and use varied high-performing basis fashions (FMs). It additionally gives a broad set of capabilities to construct generative AI functions. The Amazon Bedrock Converse API is a runtime API that gives a constant interface that works with totally different fashions. It means that you can use superior options in Amazon Bedrock such because the playground, guardrails, and power use (perform calling).

SageMaker JumpStart has lengthy been the go-to service for builders and knowledge scientists searching for to deploy state-of-the-art generative AI fashions. By means of this integration, now you can mix the pliability of internet hosting fashions on SageMaker JumpStart with the absolutely managed expertise of Amazon Bedrock, together with superior safety controls, scalable infrastructure, and complete monitoring capabilities.

On this publish, we present you methods to deploy FMs by way of SageMaker JumpStart, register them with Amazon Bedrock, and invoke them utilizing Amazon Bedrock APIs.

Resolution overview

The Converse API standardizes interplay with Amazon Bedrock FMs, enabling builders to write down code one time and use it throughout varied fashions while not having to regulate for model-specific variations. It helps multi-turn conversations by way of conversational historical past as a part of the API request, and builders can carry out duties that require entry to exterior APIs by way of the utilization of instruments (perform calling). Moreover, the Converse API means that you can block inappropriate inputs or generated content material by together with a guardrail in your API calls. To evaluate the entire checklist of supported fashions and mannequin options, seek advice from Supported fashions and mannequin options.

This new characteristic extends the capabilities of the Converse API right into a single interface that builders can use to name FMs deployed in SageMaker JumpStart. This permits builders to make use of the identical API to invoke fashions from Amazon Bedrock and SageMaker JumpStart, streamlining the method to combine fashions into their generative AI functions. Now you may construct on prime of a fair bigger library of world-class open supply and proprietary fashions by way of a single API. To view the complete checklist of Bedrock Prepared fashions accessible from SageMaker JumpStart, seek advice from the Bedrock Market documentation. You may as well use Amazon Bedrock Market to find and deploy these fashions to SageMaker endpoints.

On this publish, we stroll by way of the next steps:

1. Deploy the Gemma 2 9B Instruct mannequin utilizing SageMaker JumpStart.
2. Register the mannequin with Amazon Bedrock.
3. Take a look at the mannequin with pattern prompts utilizing the Amazon Bedrock playground.
4. Use the Amazon Bedrock RetrieveAndGenerate API to question the Amazon Bedrock information base.
5. Arrange Amazon Bedrock Guardrails to assist block dangerous content material and personally identifiable info (PII) knowledge.
6. Invoke fashions with Converse APIs to indicate an end-to-end Retrieval Augmented Era (RAG) pipeline.

Stipulations

You possibly can entry and deploy pretrained fashions from SageMaker JumpStart within the Amazon SageMaker Studio UI. To entry SageMaker Studio on the AWS Administration Console, you’ll want to arrange an Amazon SageMaker area. SageMaker makes use of domains to arrange consumer profiles, functions, and their related sources. To create a site and arrange a consumer profile, seek advice from Information to getting arrange with Amazon SageMaker.

You additionally want an AWS Identification and Entry Administration (IAM) function with applicable permissions. To get began with this instance, you need to use the AmazonSageMakerFullAccess, AmazonBedrockFullAccess, AmazonOpenSearchAccess managed insurance policies to supply the required permissions to SageMaker JumpStart and Amazon Bedrock. For a extra scoped down set of permissions, seek advice from the next:

{
  "Model": "2012-10-17",
  "Assertion": [
    {
      "Sid": "BedrockEndpointManagementMutatingOperations",
      "Action": [
        "sagemaker:AddTags",
        "sagemaker:CreateEndpoint",
        "sagemaker:CreateEndpointConfig",
        "sagemaker:CreateModel",
        "sagemaker:DeleteEndpoint",
        "sagemaker:UpdateEndpoint",
        "sagemaker:DeleteTags"
      ],
      "Impact": "Permit",
      "Useful resource": "arn:aws:sagemaker:*",
      "Situation": {
        "StringEquals": {
            "aws:ViaAWSService": "bedrock.amazonaws.com"
        }
       }
    },
    {
      "Sid": "BedrockEndpointManagementNonMutatingOperations",
      "Motion": [
        "sagemaker:DescribeEndpoint",
        "sagemaker:DescribeEndpointConfig",
        "sagemaker:DescribeModel",
        "sagemaker:ListEndpoints",
        "sagemaker:ListTags"
      ],
      "Impact": "Permit",
      "Useful resource": "arn:aws:sagemaker:*",
      "Situation": {
        "StringEquals": {
            "aws:ViaAWSService": "bedrock.amazonaws.com"
        }
       }
    },
    {
      "Sid": "BedrockEndpointInvokingOperations",
      "Motion": [
        "sagemaker:InvokeEndpoint",
        "sagemaker:InvokeEndpointWithResponseStream"      
      ],
      "Impact": "Permit",
      "Useful resource": "arn:aws:sagemaker:*",
      "Situation": {
        "StringEquals": {
            "aws:ViaAWSService": "bedrock.amazonaws.com"
         }
       }
    },
    {
      "Sid": "AllowDiscoveringPublicModelDetails",
      "Motion": [
        "sagemaker:DescribeHubContent"
      ],
      "Impact": "Permit",
      "Useful resource": "arn:aws:sagemaker:*:aws:hub-content/SageMakerPublicHub/Mannequin/*"
    },
    {
      "Sid": "AllowListingPublicModels",
      "Motion": [
        "sagemaker:ListHubContents"
      ],
      "Impact": "Permit",
      "Useful resource": "arn:aws:sagemaker:*:aws:hub/SageMakerPublicHub"
    },
    {
      "Sid": "RetrieveSubscribedMarketplaceLicenses",
      "Motion": [
        "license-manager:ListReceivedLicenses"
      ],
      "Impact": "Permit",
      "Useful resource": "arn:aws:license-manager:*"
    },
    {
      "Sid" : "PassRoleToSagemaker",
      "Impact" : "Permit",
      "Motion" : [
        "iam:PassRole"
      ],
      "Useful resource" : "arn:aws:iam::*:function/*AmazonSageMaker*",
      "Situation" : {
        "StringEquals" : {
        "iam:PassedToService" : [
            "sagemaker.amazonaws.com"
          ]
        }
      }
    },
    {
      "Sid" : "BedrockAll",
      "Impact" : "Permit",
      "Motion" : [ "bedrock:*" ],
      "Useful resource" : "*" 
    },
    {
      "Sid" : "AmazonOpenSearchAccess",
      "Impact" : "Permit",
      "Motion" : [ "aoss:*" ],
      "Useful resource" : "*",
      "Situation": {
                "StringEquals": {
                    "aws:ResourceAccount": "${aws:PrincipalAccount}"
                }
      }
    },
  ]
}

After making use of the related permissions, organising a SageMaker area, and creating consumer profiles, you might be able to deploy your SageMaker JumpStart mannequin and register it with Amazon Bedrock.

Deploy a mannequin with SageMaker JumpStart and register it with Amazon Bedrock

This part gives a walkthrough of deploying a mannequin utilizing SageMaker JumpStart and registering it with Amazon Bedrock. On this walkthrough, you’ll deploy and register the Gemma 2 9B Instruct mannequin supplied by way of Hugging Face in SageMaker JumpStart. Full the next steps:

1. On the SageMaker console, select Studio within the navigation pane.
2. Select the related consumer profile on the dropdown menu and select Open Studio.

3. In SageMaker Studio, select JumpStart within the navigation pane.

Right here, you will notice a listing of the accessible SageMaker JumpStart fashions. Fashions that may be registered to Amazon Bedrock after they’ve been deployed by way of SageMaker JumpStart have a Bedrock prepared tag.

4. The Gemma 2 9B Instruct mannequin for this instance is offered by Hugging Face, so select the Hugging Face mannequin card.

5. To filter the checklist of fashions to view which fashions are supported by Amazon Bedrock, choose Bedrock Prepared below Motion.
6. Seek for Gemma 2 9B Instruct and select the mannequin card for Gemma 2 9B Instruct.

You possibly can evaluate the mannequin card for Gemma 2 9B Instruct to be taught extra in regards to the mannequin.

7. To deploy the mannequin, select Deploy.
8. Evaluation the Finish Person License Settlement for Gemma 2 9B Instruct and choose I settle for the Finish Person License Settlement (EULA) and browse the phrases and circumstances.
9. Depart the endpoint settings with their default values and select Deploy.

The endpoint deployment course of will take a couple of minutes.

10. Beneath Deployments within the navigation pane, select Endpoints to view your accessible endpoints.

After a couple of minutes, the mannequin shall be deployed to the endpoint and its standing will change to In service, indicating that the endpoint is able to serve visitors. You should utilize the Refresh icon on the backside of the endpoint display to get the most recent info.

11. When your endpoint is in service, select it to go to the endpoint particulars web page.

12. Select Use with Bedrock to begin the registration course of.

You’ll be redirected to the Amazon Bedrock console.

13. On the Register endpoint web page, the SageMaker endpoint Amazon Useful resource Identify (ARN) and mannequin ARN have already been prepopulated. Evaluation these values and select Register.

Your SageMaker endpoint shall be registered with Amazon Bedrock in a couple of minutes.

After your SageMaker endpoint is registered with Amazon Bedrock, you may invoke it utilizing the Converse API. Then you may take a look at your endpoint within the Amazon Bedrock playground.

14. Within the navigation pane on the Amazon Bedrock console, select Market deployments below Basis fashions.
15. From the checklist of managed deployments, choose your registered mannequin, then select Open in playground.

You’ll now be within the Amazon Bedrock playground for Chat/textual content. The Chat/textual content playground permits to you take a look at your mannequin with a single immediate, or gives chat functionality for conversational use circumstances. As a result of this instance shall be an interactive chat session, depart the Mode because the default Chat. The chat functionality within the playground ought to be set to check your Gemma 2 9B Instruct mannequin.

Now you may take a look at your SageMaker endpoint by way of Amazon Bedrock! Use the next immediate to check summarizing a gathering transcript, and evaluate the outcomes:

Assembly transcript:
Miguel: Hello Brant, I wish to talk about the workstream for our new product launch
Brant: Positive Miguel, is there something specifically you wish to talk about?
Miguel: Sure, I wish to discuss how customers enter into the product.
Brant: Okay, in that case let me add in Namita.
Namita: Hey everybody
Brant: Hello Namita, Miguel desires to debate how customers enter into the product.
Miguel: its too sophisticated and we should always take away friction.  for instance, why do I have to fill out extra types?  I additionally discover it tough to search out the place to entry the product once I first land on the touchdown web page.
Brant: I might additionally add that I believe there are too many steps.
Namita: Okay, I can work on the touchdown web page to make the product extra discoverable however brant can you're employed on the additonal types?
Brant: Sure however I would wish to work with James from one other workforce as he must unblock the enroll workflow.  Miguel are you able to doc another issues in order that I can talk about with James solely as soon as?
Miguel: Positive.

From the assembly transcript above, Create a listing of motion objects for every particular person.

16. Enter the immediate into the playground, then select Run.

You possibly can view the response within the chat generated by your deployed SageMaker JumpStart mannequin by way of Amazon Bedrock:

Here is a breakdown of motion objects from the assembly transcript:

**Miguel:**

* **Doc:** Record out any extra issues relating to consumer entry into the product. Share these with Brant for his dialogue with James.

**Brant:**

* **Collaborate with James:**  Work with James from one other workforce to simplify the extra types concerned within the consumer sign-up workflow.
* **Evaluation Documentation:** Evaluation Miguel's documented issues about consumer entry to arrange for the dialogue with James.

**Namita:**

* **Touchdown Web page Redesign:**  Enhance the touchdown web page to make the product extra discoverable for brand spanking new customers.

Let me know if you would like me to elaborate on any of those motion objects!

You may as well take a look at the mannequin with your personal prompts and use circumstances.

Use Amazon Bedrock APIs with the deployed mannequin

This part demonstrates utilizing the AWS SDK for Python (Boto3) and Converse APIs to invoke the Gemma 2 9B Instruct mannequin you deployed earlier by way of SageMaker and registered with Amazon Bedrock. The complete supply code related to this publish is obtainable within the accompanying GitHub repo. For added Converse API examples, seek advice from Converse API examples.

On this code pattern, we additionally implement a RAG structure along with the deployed mannequin. RAG is the method of optimizing the output of a giant language mannequin (LLM) so it references an authoritative information base exterior of its coaching knowledge sources earlier than producing a response.

Use the deployed SageMaker mannequin with the RetrieveAndGenerate API supplied by Amazon Bedrock to question a information base and generate responses based mostly on the retrieved outcomes. The response solely cites sources which can be related to the question. For info on making a Information Base, seek advice from Making a Information Base. For added code samples, seek advice from RetrieveAndGenerate.

The next diagram illustrates the RAG workflow.

Full the next steps:

1. To invoke the deployed mannequin, you’ll want to go the endpoint ARN of the deployed mannequin within the modelId parameter of the Converse API.

To acquire the ARN of the deployed mannequin, navigate to the Amazon Bedrock console. Within the navigation pane, select Market deployments below Basis fashions. From the checklist of managed deployments, select your registered mannequin to view extra particulars.

You’ll be directed to the mannequin abstract on the Mannequin catalog web page below Basis fashions. Right here, you can find the main points related along with your deployed mannequin. Copy the mannequin ARN to make use of within the following code pattern.

import boto3

bedrock_runtime = boto3.shopper("bedrock-runtime")

# Add your bedrock endpoint arn right here.
endpoint_arn = "arn:aws:sagemaker:<:region>:<:accountid>:endpoint/"

# Base inference parameters to make use of.
inference_config = {
        "maxTokens": 256,
        "temperature": 0.1,
        "topP": 0.999,
}

# Extra inference parameters to make use of.
additional_model_fields = {"parameters": {"repetition_penalty": 0.9, "top_k": 250, "do_sample": True}}


response = bedrock_runtime.converse(
    modelId=endpoint_arn,
    messages=[
        {
            "role": "user",
            "content": [
                {
                    "text": "What is Amazon doing in the field of generative AI?",
                },
            ]
        },
    ],
    inferenceConfig=inference_config,
    additionalModelRequestFields=additional_model_fields,
)

2. Invoke the SageMaker JumpStart mannequin with the RetrieveAndGenerate API. The generation_template and orchestration_template parameters within the retrieve_and_generate API are mannequin particular. These templates outline the prompts and directions for the language mannequin, guiding the era course of and the mixing with the information retrieval element.

import boto3

bedrock_agent_runtime_client = boto3.shopper("bedrock-agent-runtime")

# Present your Information Base Id 
kb_id = "" 

response = bedrock_agent_runtime_client.retrieve_and_generate(
    enter={
        "textual content": "What's Amazon doing within the subject of generative AI?"
    },
    retrieveAndGenerateConfiguration={
        "sort": "KNOWLEDGE_BASE",
        "knowledgeBaseConfiguration": {
            "generationConfiguration": {
                "inferenceConfig": {
                    "textInferenceConfig": {
                        "maxTokens": 512,
                        "temperature": 0.1,
                        "topP": 0.9
                    }
                },
                "promptTemplate": {
                    "textPromptTemplate": generation_template
                }
            },
            "knowledgeBaseId": kb_id,
            "orchestrationConfiguration": {
                "inferenceConfig": {
                    "textInferenceConfig": {
                        "maxTokens": 512,
                        "temperature": 0.1,
                        "topP": 0.9
                    }
                },
                "promptTemplate": {
                    "textPromptTemplate": orchestration_template
                },
            },
            "modelArn": endpoint_arn,
            "retrievalConfiguration": {
                "vectorSearchConfiguration": {
                    "numberOfResults":5
                } 
            }
        }
    }
)

Now you may implement guardrails with the Converse API on your SageMaker JumpStart mannequin. Amazon Bedrock Guardrails allows you to implement safeguards on your generative AI functions based mostly in your use circumstances and accountable AI insurance policies. For info on creating guardrails, seek advice from Create a Guardrail. For added code samples to implement guardrails, seek advice from Embody a guardrail with Converse API.

3. Within the following code pattern, you embrace a guardrail in a Converse API request invoking a SageMaker JumpStart mannequin:

import boto3

bedrock_agent_runtime_client = boto3.shopper("bedrock-agent-runtime")

# Present your Information Base Id
kb_id = "" 

relevant_documents = bedrock_agent_runtime_client.retrieve(
    retrievalQuery= {
        "textual content": "What's Amazon doing within the subject of generative AI?"
    },
    knowledgeBaseId=kb_id,
    retrievalConfiguration= {
        "vectorSearchConfiguration": {
            "numberOfResults": 1
        }
    }
)

def invoke_model(immediate, supply, inference_config=None, additional_model_field=None):
    messages = [
        {
            "role": "user",
            "content": [
                {
                    "guardContent": {
                        "text": {
                            "text": source,
                            "qualifiers": ["grounding_source"],
                        }
                    }
                },
                {
                    "guardContent": {
                        "textual content": {
                            "textual content": immediate,
                            "qualifiers": ["query"],
                        }
                    }
                },
            ],
        }
    ]
    if not inference_config:
        # Base inference parameters to make use of.
        inference_config = {
                "maxTokens": 256,
                "temperature": 0.1,
                "topP": 0.999,
        }
    
    if not additional_model_field:
        # Extra inference parameters to make use of.
        additional_model_fields = {"parameters": {"repetition_penalty": 0.9, "top_k": 250, "do_sample": True}}


    response = bedrock_runtime.converse(
        modelId=endpoint_arn,
        messages=messages,
        inferenceConfig=inference_config,
        additionalModelRequestFields=additional_model_fields,
        guardrailConfig={
            'guardrailIdentifier': guardrail_identifier,
            'guardrailVersion': guardrail_version
        },
    )
    
    return response["output"]["message"]["content"][0]["text"]

invoke_model(immediate="What's Amazon doing within the subject of generative AI?", supply=relevant_documents["retrievalResults"][0]["content"]["text"]) 
# Content material is Blocked 
invoke_model(immediate="Ought to I purchase bitcoin?", supply=relevant_documents["retrievalResults"][0]["content"]["text"])

Clear up

To scrub up your sources, use the next code:

import boto3

from knowledge_base import KnowledgeBasesForAmazonBedrock

kb = KnowledgeBasesForAmazonBedrock()
kb.delete_kb(knowledge_base_name, delete_s3_bucket=True, delete_iam_roles_and_policies=True)

bedrock.delete_guardrail(guardrailIdentifier=guardrail_identifier)

The SageMaker JumpStart mannequin you deployed will incur price should you depart it operating. Delete the endpoint if you wish to cease incurring prices. Deleting the endpoint may also de-register the mannequin from Amazon Bedrock. For extra particulars, see Delete Endpoints and Assets.

Conclusion

On this publish, you realized methods to deploy FMs by way of SageMaker JumpStart, register them with Amazon Bedrock, and invoke them utilizing Amazon Bedrock APIs. With this new functionality, organizations can entry main proprietary and open-weight fashions utilizing a single API, decreasing the complexity of constructing generative AI functions with quite a lot of fashions. This integration between SageMaker JumpStart and Amazon Bedrock is mostly accessible in all AWS Areas the place Amazon Bedrock is obtainable. Do this code to make use of ConverseAPIs, Information bases and Guardrails with SageMaker.

Concerning the Creator

Vivek Gangasani is a Senior GenAI Specialist Options Architect at AWS. He helps rising GenAI firms construct modern options utilizing AWS providers and accelerated compute. At the moment, he’s centered on growing methods for fine-tuning and optimizing the inference efficiency of Giant Language Fashions. In his free time, Vivek enjoys mountaineering, watching motion pictures and making an attempt totally different cuisines.

Abhishek Doppalapudi is a Options Architect at Amazon Net Companies (AWS), the place he assists startups in constructing and scaling their merchandise utilizing AWS providers. At the moment, he’s centered on serving to AWS clients undertake Generative AI options. In his free time, Abhishek enjoys enjoying soccer, watching Premier League matches, and studying.

June Received  is a product supervisor with Amazon SageMaker JumpStart. He focuses on making basis fashions simply discoverable and usable to assist clients construct generative AI functions. His expertise at Amazon additionally consists of cellular buying functions and final mile supply.

Eashan Kaushik is an Affiliate Options Architect at Amazon Net Companies. He’s pushed by creating cutting-edge generative AI options whereas prioritizing a customer-centric strategy to his work. Earlier than this function, he obtained an MS in Laptop Science from NYU Tandon College of Engineering. Exterior of labor, he enjoys sports activities, lifting, and operating marathons.

Giuseppe Zappia is a Principal AI/ML Specialist Options Architect at AWS, centered on serving to massive enterprises design and deploy ML options on AWS. He has over 20 years of expertise as a full stack software program engineer, and has spent the previous 5 years at AWS centered on the sector of machine studying.

Bhaskar Pratap is a Senior Software program Engineer with the Amazon SageMaker workforce. He’s enthusiastic about designing and constructing elegant programs that convey machine studying to individuals’s fingertips. Moreover, he has intensive expertise with constructing scalable cloud storage providers.