DeepSeek-R1 mannequin now out there in Amazon Bedrock Market and Amazon SageMaker JumpStart

At this time, we’re asserting that DeepSeek AI’s first-generation frontier mannequin, DeepSeek-R1, is offered via Amazon SageMaker JumpStart and Amazon Bedrock Market to deploy for inference. Now you can use DeepSeek-R1 to construct, experiment, and responsibly scale your generative AI concepts on AWS.

On this submit, we display how one can get began with DeepSeek-R1 on Amazon Bedrock and SageMaker JumpStart.

Overview of DeepSeek-R1

DeepSeek-R1 is a big language mannequin (LLM) developed by DeepSeek-AI that makes use of reinforcement studying to reinforce reasoning capabilities via a multi-stage coaching course of from a DeepSeek-V3-Base basis. A key distinguishing function is its reinforcement studying (RL) step, which was used to refine the mannequin’s responses past the usual pre-training and fine-tuning course of. By incorporating RL, DeepSeek-R1 can adapt extra successfully to person suggestions and aims, finally enhancing each relevance and readability. As well as, DeepSeek-R1 employs a chain-of-thought (CoT) method, that means it’s geared up to interrupt down advanced queries and purpose via them in a step-by-step method. This guided reasoning course of permits the mannequin to supply extra correct, clear, and detailed solutions. This mannequin combines RL-based fine-tuning with CoT capabilities, aiming to generate structured responses whereas specializing in interpretability and person interplay. With its wide-ranging capabilities DeepSeek-R1 has captured the trade’s consideration as a flexible text-generation mannequin that may be built-in into numerous workflows akin to brokers, logical reasoning and information interpretation duties

DeepSeek-R1 makes use of a Combination of Specialists (MoE) structure and is 671 billion parameters in dimension. The MoE structure permits activation of 37 billion parameters, enabling environment friendly inference by routing queries to essentially the most related skilled “clusters.” This method permits the mannequin to focus on totally different downside domains whereas sustaining general effectivity. DeepSeek-R1 requires not less than 800 GB of HBM reminiscence in FP8 format for inference. On this submit, we’ll use an ml.p5e.48xlarge occasion to deploy the mannequin. ml.p5e.48xlarge comes with 8 Nvidia H200 GPUs offering 1128 GB of GPU reminiscence.

You may deploy DeepSeek-R1 mannequin both via SageMaker JumpStart or Bedrock Market. As a result of DeepSeek-R1 is an rising mannequin, we advocate deploying this mannequin with guardrails in place. On this weblog, we’ll use Amazon Bedrock Guardrails to introduce safeguards, forestall dangerous content material, and consider fashions towards key security standards. On the time of penning this weblog, for DeepSeek-R1 deployments on SageMaker JumpStart and Bedrock Market, Bedrock Guardrails helps solely the ApplyGuardrail API. You may create a number of guardrails tailor-made to totally different use circumstances and apply them to the DeepSeek-R1 mannequin, bettering person experiences and standardizing security controls throughout your generative AI purposes.

Stipulations

To deploy the DeepSeek-R1 mannequin, you want entry to an ml.p5e occasion. To test in case you have quotas for P5e, open the Service Quotas console and beneath AWS Companies, select Amazon SageMaker, and make sure you’re utilizing ml.p5e.48xlarge for endpoint utilization. Just be sure you have not less than one ml.P5e.48xlarge occasion within the AWS Area you’re deploying. To request a restrict improve, create a restrict improve request and attain out to your account crew.

As a result of you’ll be deploying this mannequin with Amazon Bedrock Guardrails, ensure you have the right AWS Identification and Entry Administration (IAM) permissions to make use of Amazon Bedrock Guardrails. For directions, see Arrange permissions to make use of guardrails for content material filtering.

Implementing guardrails with the ApplyGuardrail API

Amazon Bedrock Guardrails lets you introduce safeguards, forestall dangerous content material, and consider fashions towards key security standards. You may implement security measures for the DeepSeek-R1 mannequin utilizing the Amazon Bedrock ApplyGuardrail API. This lets you apply guardrails to judge person inputs and mannequin responses deployed on Amazon Bedrock Market and SageMaker JumpStart. You may create a guardrail utilizing the Amazon Bedrock console or the API. For the instance code to create the guardrail, see the GitHub repo.

The final circulation entails the next steps: First, the system receives an enter for the mannequin. This enter is then processed via the ApplyGuardrail API. If the enter passes the guardrail test, it’s despatched to the mannequin for inference. After receiving the mannequin’s output, one other guardrail test is utilized. If the output passes this closing test, it’s returned as the ultimate end result. Nevertheless, if both the enter or output is intervened by the guardrail, a message is returned indicating the character of the intervention and whether or not it occurred on the enter or output stage. The examples showcased within the following sections display inference utilizing this API.

Deploy DeepSeek-R1 in Amazon Bedrock Market

Amazon Bedrock Market provides you entry to over 100 widespread, rising, and specialised basis fashions (FMs) via Amazon Bedrock. To entry DeepSeek-R1 in Amazon Bedrock, full the next steps:

1. On the Amazon Bedrock console, select Mannequin catalog beneath Basis fashions within the navigation pane.
   On the time of penning this submit, you need to use the InvokeModel API to invoke the mannequin. It doesn’t help Converse APIs and different Amazon Bedrock tooling.
2. Filter for DeepSeek as a supplier and select the DeepSeek-R1 mannequin.
   
   The mannequin element web page gives important details about the mannequin’s capabilities, pricing construction, and implementation pointers. You’ll find detailed utilization directions, together with pattern API calls and code snippets for integration. The mannequin helps numerous textual content era duties, together with content material creation, code era, and query answering, utilizing its reinforcement studying optimization and CoT reasoning capabilities.
   The web page additionally consists of deployment choices and licensing info that can assist you get began with DeepSeek-R1 in your purposes.
3. To start utilizing DeepSeek-R1, select Deploy.
   
   You may be prompted to configure the deployment particulars for DeepSeek-R1. The mannequin ID can be pre-populated.
4. For Endpoint title, enter an endpoint title (between 1–50 alphanumeric characters).
5. For Variety of situations, enter various situations (between 1–100).
6. For Occasion sort, select your occasion sort. For optimum efficiency with DeepSeek-R1, a GPU-based occasion sort like ml.p5e.48xlarge is advisable.
   Optionally, you may configure superior safety and infrastructure settings, together with digital personal cloud (VPC) networking, service function permissions, and encryption settings. For many use circumstances, the default settings will work properly. Nevertheless, for manufacturing deployments, you would possibly need to assessment these settings to align together with your group’s safety and compliance necessities.
7. Select Deploy to start utilizing the mannequin.
   
   When the deployment is full, you may take a look at DeepSeek-R1’s capabilities straight within the Amazon Bedrock playground.
8. Select Open in playground to entry an interactive interface the place you may experiment with totally different prompts and modify mannequin parameters like temperature and most size.
   When utilizing R1 with Bedrock’s InvokeModel and Playground Console, use DeepSeek’s chat template for optimum outcomes. For instance, <｜start▁of▁sentence｜><｜Person｜>content material for inference<｜Assistant｜> .

This is a superb option to discover the mannequin’s reasoning and textual content era talents earlier than integrating it into your purposes. The playground gives rapid suggestions, serving to you perceive how the mannequin responds to numerous inputs and letting you fine-tune your prompts for optimum outcomes.

You may rapidly take a look at the mannequin within the playground via the UI. Nevertheless, to invoke the deployed mannequin programmatically with any Amazon Bedrock APIs, you want to get the endpoint ARN.

Run inference utilizing guardrails with the deployed DeepSeek-R1 endpoint

The next code instance demonstrates how one can carry out inference utilizing a deployed DeepSeek-R1 mannequin via Amazon Bedrock utilizing the invoke_model and ApplyGuardrail API. You may create a guardrail utilizing the Amazon Bedrock console or the API. For the instance code to create the guardrail, see the GitHub repo. After you could have created the guardrail, use the next code to implement guardrails. The script initializes the bedrock_runtime shopper, configures inference parameters, and sends a request to generate textual content based mostly on a person immediate.

import boto3
import json

# Initialize Bedrock shopper
bedrock_runtime = boto3.shopper("bedrock-runtime")

# Configuration
MODEL_ID = "your-model-id"  # Bedrock mannequin ID
GUARDRAIL_ID = "your-guardrail-id"
GUARDRAIL_VERSION = "your-guardrail-version"

def invoke_with_guardrails(immediate, max_tokens=1000, temperature=0.6, top_p=0.9):
    """
    Invoke Bedrock mannequin with enter and output guardrails
    """
    # Apply enter guardrails
    input_guardrail = bedrock_runtime.apply_guardrail(
        guardrailIdentifier=GUARDRAIL_ID,
        guardrailVersion=GUARDRAIL_VERSION,
        supply="INPUT",
        content material=[{"text": {"text": prompt}}]
    )
    
    if input_guardrail['action'] == 'GUARDRAIL_INTERVENED':
        return f"Enter blocked: {input_guardrail['outputs'][0]['text']}"

    # Put together mannequin enter
    request_body = {
        "inputs": f"""You're an AI assistant. Do because the person asks.
### Instruction: {immediate}
### Response: """,
        "parameters": {
            "max_new_tokens": max_tokens,
            "top_p": top_p,
            "temperature": temperature
        }
    }

    # Invoke mannequin
    response = bedrock_runtime.invoke_model(
        modelId=MODEL_ID,
        physique=json.dumps(request_body)
    )
    
    # Parse mannequin response
    model_output = json.hundreds(response['body'].learn())['generated_text']

    # Apply output guardrails
    output_guardrail = bedrock_runtime.apply_guardrail(
        guardrailIdentifier=GUARDRAIL_ID,
        guardrailVersion=GUARDRAIL_VERSION,
        supply="OUTPUT",
        content material=[{"text": {"text": model_output}}]
    )

    if output_guardrail['action'] == 'GUARDRAIL_INTERVENED':
        return f"Output blocked: {output_guardrail['outputs'][0]['text']}"
    
    return model_output

# Instance utilization
if __name__ == "__main__":
    immediate = "What's 1+1?"
    end result = invoke_with_guardrails(immediate)
    print(end result)

Deploy DeepSeek-R1 with SageMaker JumpStart

SageMaker JumpStart is a machine studying (ML) hub with FMs, built-in algorithms, and prebuilt ML options that you would be able to deploy with only a few clicks. With SageMaker JumpStart, you may customise pre-trained fashions to your use case, together with your information, and deploy them into manufacturing utilizing both the UI or SDK.

Deploying DeepSeek-R1 mannequin via SageMaker JumpStart gives two handy approaches: utilizing the intuitive SageMaker JumpStart UI or implementing programmatically via the SageMaker Python SDK. Let’s discover each strategies that can assist you select the method that most accurately fits your wants.

Deploy DeepSeek-R1 via SageMaker JumpStart UI

Full the next steps to deploy DeepSeek-R1 utilizing SageMaker JumpStart:

1. On the SageMaker console, select Studio within the navigation pane.
2. First-time customers can be prompted to create a website.
3. On the SageMaker Studio console, select JumpStart within the navigation pane.
   
   The mannequin browser shows out there fashions, with particulars just like the supplier title and mannequin capabilities.
   
4. Seek for DeepSeek-R1 to view the DeepSeek-R1 mannequin card.
   Every mannequin card reveals key info, together with:
   • Mannequin title
   • Supplier title
   • Job class (for instance, Textual content Era)
   • Bedrock Prepared badge (if relevant), indicating that this mannequin might be registered with Amazon Bedrock, permitting you to make use of Amazon Bedrock APIs to invoke the mannequin
5. Select the mannequin card to view the mannequin particulars web page.
   
   The mannequin particulars web page consists of the next info:
   • The mannequin title and supplier info
   • Deploy button to deploy the mannequin
   • About and Notebooks tabs with detailed info

   The About tab consists of necessary particulars, akin to:

   • Mannequin description
   • License info
   • Technical specs
   • Utilization pointers

   Earlier than you deploy the mannequin, it’s advisable to assessment the mannequin particulars and license phrases to substantiate compatibility together with your use case.

6. Select Deploy to proceed with deployment.
   
7. For Endpoint title, use the mechanically generated title or create a customized one.
8. For Occasion sort¸ select an occasion sort (default: ml.p5e.48xlarge).
9. For Preliminary occasion depend, enter the variety of situations (default: 1).
   Deciding on acceptable occasion sorts and counts is essential for value and efficiency optimization. Monitor your deployment to regulate these settings as wanted.Underneath Inference sort, Actual-time inference is chosen by default. That is optimized for sustained visitors and low latency.
10. Assessment all configurations for accuracy. For this mannequin, we strongly advocate adhering to SageMaker JumpStart default settings and ensuring that community isolation stays in place.
11. Select Deploy to deploy the mannequin.

The deployment course of can take a number of minutes to finish.

When deployment is full, your endpoint standing will change to InService. At this level, the mannequin is able to settle for inference requests via the endpoint. You may monitor the deployment progress on the SageMaker console Endpoints web page, which can show related metrics and standing info. When the deployment is full, you may invoke the mannequin utilizing a SageMaker runtime shopper and combine it together with your purposes.

Deploy DeepSeek-R1 utilizing the SageMaker Python SDK

To get began with DeepSeek-R1 utilizing the SageMaker Python SDK, you will want to put in the SageMaker Python SDK and ensure you have the mandatory AWS permissions and atmosphere setup. The next is a step-by-step code instance that demonstrates how one can deploy and use DeepSeek-R1 for inference programmatically. The code for deploying the mannequin is offered within the Github right here . You may clone the pocket book and run from SageMaker Studio.

!pip set up --force-reinstall --no-cache-dir sagemaker==2.235.2

from sagemaker.serve.builder.model_builder import ModelBuilder 
from sagemaker.serve.builder.schema_builder import SchemaBuilder 
from sagemaker.jumpstart.mannequin import ModelAccessConfig 
from sagemaker.session import Session 
import logging 

sagemaker_session = Session()
 
artifacts_bucket_name = sagemaker_session.default_bucket() 
execution_role_arn = sagemaker_session.get_caller_identity_arn()
 
js_model_id = "deepseek-llm-r1"

gpu_instance_type = "ml.p5e.48xlarge"
 
response = "Hey, I am a language mannequin, and I am right here that can assist you together with your English."

 sample_input = {
 "inputs": "Hey, I am a language mannequin,",
 "parameters": {"max_new_tokens": 128, "top_p": 0.9, "temperature": 0.6},
 }
  
 sample_output = [{"generated_text": response}]
  
 schema_builder = SchemaBuilder(sample_input, sample_output)
  
 model_builder = ModelBuilder( 
 mannequin=js_model_id, 
 schema_builder=schema_builder, 
 sagemaker_session=sagemaker_session, 
 role_arn=execution_role_arn, 
 log_level=logging.ERROR ) 
 
 mannequin= model_builder.construct() 
 predictor = mannequin.deploy(model_access_configs={js_model_id:ModelAccessConfig(accept_eula=True)}, accept_eula=True) 
 
 
 predictor.predict(sample_input)

You may run further requests towards the predictor:

new_input = {
    "inputs": "What's Amazon doing in Generative AI?",
    "parameters": {"max_new_tokens": 64, "top_p": 0.8, "temperature": 0.7},
}

prediction = predictor.predict(new_input)
print(prediction)

Implement guardrails and run inference together with your SageMaker JumpStart predictor

Just like Amazon Bedrock, you may also use the ApplyGuardrail API together with your SageMaker JumpStart predictor. You may create a guardrail utilizing the Amazon Bedrock console or the API, and implement it as proven within the following code:

import boto3
import json
bedrock_runtime = boto3.shopper('bedrock-runtime')
sagemaker_runtime = boto3.shopper('sagemaker-runtime')

# Add your guardrail identifier and model created from Bedrock Console or AWSCLI
guardrail_id = "" # Your Guardrail ID
guardrail_version = "" # Your Guardrail Model
endpoint_name = "" # Endpoint Title

immediate = "What's 1+1 equal?"

# Apply guardrail to enter earlier than sending to mannequin
input_guardrail_response = bedrock_runtime.apply_guardrail(
    guardrailIdentifier=guardrail_id,
    guardrailVersion=guardrail_version,
    supply="INPUT",
    content material=[{ "text": { "text": prompt }}]
)

# If enter guardrail passes, proceed with mannequin inference
if input_guardrail_response['action'] != 'GUARDRAIL_INTERVENED':
    # Put together the enter for the SageMaker endpoint
    template = f"""You're an AI assistant. Do because the person asks.
### Instruction: {immediate}
### Response: """
    
    input_payload = {
        "inputs": template,
        "parameters": {
            "max_new_tokens": 1000,
            "top_p": 0.9,
            "temperature": 0.6
        }
    }
    
    # Convert the payload to JSON string
    input_payload_json = json.dumps(input_payload)
    
    # Invoke the SageMaker endpoint
    response = sagemaker_runtime.invoke_endpoint(
        EndpointName=endpoint_name,
        ContentType="software/json",
        Physique=input_payload_json
    )
    
    # Get the response from the mannequin
    model_response = json.hundreds(response['Body'].learn().decode())
    
    # Apply guardrail to output
    output_guardrail_response = bedrock_runtime.apply_guardrail(
        guardrailIdentifier=guardrail_id,
        guardrailVersion=guardrail_version,
        supply="OUTPUT",
        content material=[{ "text": { "text": model_response['generated_text'] }}]
    )
    
    # Verify if output passes guardrails
    if output_guardrail_response['action'] != 'GUARDRAIL_INTERVENED':
        print(model_response['generated_text'])
    else:
        print("Output blocked: ", output_guardrail_response['outputs'][0]['text'])
else:
    print("Enter blocked: ", input_guardrail_response['outputs'][0]['text'])

Clear up

To keep away from undesirable costs, full the steps on this part to wash up your sources.

Delete the Amazon Bedrock Market deployment

In the event you deployed the mannequin utilizing Amazon Bedrock Market, full the next steps:

1. On the Amazon Bedrock console, beneath Basis fashions within the navigation pane, select Market deployments.
2. Within the Managed deployments part, find the endpoint you need to delete.
3. Choose the endpoint, and on the Actions menu, select Delete.
4. Confirm the endpoint particulars to ensure you’re deleting the right deployment:
   1. Endpoint title
   2. Mannequin title
   3. Endpoint standing
5. Select Delete to delete the endpoint.
6. Within the deletion affirmation dialog, assessment the warning message, enter affirm, and select Delete to completely take away the endpoint.
   

Delete the SageMaker JumpStart predictor

The SageMaker JumpStart mannequin you deployed will incur prices in case you go away it operating. Use the next code to delete the endpoint if you wish to cease incurring costs. For extra particulars, see Delete Endpoints and Assets.

predictor.delete_model()
predictor.delete_endpoint()

Conclusion

On this submit, we explored how one can entry and deploy the DeepSeek-R1 mannequin utilizing Bedrock Market and SageMaker JumpStart. Go to SageMaker JumpStart in SageMaker Studio or Amazon Bedrock Market now to get began. For extra info, check with Use Amazon Bedrock tooling with Amazon SageMaker JumpStart fashions, SageMaker JumpStart pretrained fashions, Amazon SageMaker JumpStart Basis Fashions, Amazon Bedrock Market, and Getting began with Amazon SageMaker JumpStart.

In regards to the Authors

Vivek Gangasani is a Lead Specialist Options Architect for Inference at AWS. He helps rising generative AI corporations construct modern options utilizing AWS providers and accelerated compute. Presently, he’s centered on creating methods for fine-tuning and optimizing the inference efficiency of huge language fashions. In his free time, Vivek enjoys mountain climbing, watching films, and making an attempt totally different cuisines.

Niithiyn Vijeaswaran is a Generative AI Specialist Options Architect with the Third-Occasion Mannequin Science crew at AWS. His space of focus is AWS AI accelerators (AWS Neuron). He holds a Bachelor’s diploma in Pc Science and Bioinformatics.

Jonathan Evans is a Specialist Options Architect engaged on generative AI with the Third-Occasion Mannequin Science crew at AWS.

Banu Nagasundaram leads product, engineering, and strategic partnerships for Amazon SageMaker JumpStart, SageMaker’s machine studying and generative AI hub. She is obsessed with constructing options that assist clients speed up their AI journey and unlock enterprise worth.