Use the ApplyGuardrail API with long-context inputs and streaming outputs in Amazon Bedrock

As generative synthetic intelligence (AI) purposes turn into extra prevalent, sustaining accountable AI ideas turns into important. With out correct safeguards, giant language fashions (LLMs) can doubtlessly generate dangerous, biased, or inappropriate content material, posing dangers to people and organizations. Making use of guardrails helps mitigate these dangers by imposing insurance policies and pointers that align with moral ideas and authorized necessities. Guardrails for Amazon Bedrock evaluates person inputs and mannequin responses based mostly on use case-specific insurance policies, and offers an extra layer of safeguards whatever the underlying basis mannequin (FM). Guardrails could be utilized throughout all LLMs on Amazon Bedrock, together with fine-tuned fashions and even generative AI purposes exterior of Amazon Bedrock. You possibly can create a number of guardrails, every configured with a distinct mixture of controls, and use these guardrails throughout completely different purposes and use instances. You possibly can configure guardrails in a number of methods, together with to disclaim matters, filter dangerous content material, take away delicate info, and detect contextual grounding.

The brand new ApplyGuardrail API lets you assess any textual content utilizing your preconfigured guardrails in Amazon Bedrock, with out invoking the FMs. On this publish, we exhibit methods to use the ApplyGuardrail API with long-context inputs and streaming outputs.

ApplyGuardrail API overview

The ApplyGuardrail API presents a number of key options:

• Ease of use – You possibly can combine the API anyplace in your utility stream to validate information earlier than processing or serving outcomes to customers. For instance, in a Retrieval Augmented Technology (RAG) utility, now you can consider the person enter previous to performing the retrieval as a substitute of ready till the ultimate response era.
• Decoupled from FMs – The API is decoupled from FMs, permitting you to make use of guardrails with out invoking FMs from Amazon Bedrock. For instance, now you can use the API with fashions hosted on Amazon SageMaker. Alternatively, you might use it self-hosted or with fashions from third-party mannequin suppliers. All that’s wanted is taking the enter or output and request evaluation utilizing the API.

You should use the evaluation outcomes from the ApplyGuardrail API to design the expertise in your generative AI utility, ensuring it adheres to your outlined insurance policies and pointers.

The ApplyGuardrail API request permits you to move all of your content material that ought to be guarded utilizing your outlined guardrails. The supply subject ought to be set to INPUT when the content material to evaluated is from a person, usually the LLM immediate. The supply ought to be set to OUTPUT when the mannequin output guardrails ought to be enforced, usually an LLM response. An instance request seems like the next code:

{
    "supply": "INPUT" | "OUTPUT",
    "content material": [{
        "text": {
            "text": "This is a sample text snippet...",
        }
    }]
}

For extra details about the API construction, consult with Guardrails for Amazon Bedrock.

Streaming output

LLMs can generate textual content in a streaming method, the place the output is produced token by token or phrase by phrase, fairly than producing your entire output without delay. This streaming output functionality is especially helpful in eventualities the place real-time interplay or steady era is required, reminiscent of conversational AI assistants or dwell captioning. Incrementally displaying the output permits for a extra pure and responsive person expertise. Though it’s advantageous by way of responsiveness, streaming output introduces challenges in the case of making use of guardrails in actual time because the output is generated. In contrast to the enter situation, the place your entire textual content is obtainable upfront, the output is generated incrementally, making it tough to evaluate the whole context and potential violations.

One of many principal challenges is the necessity to consider the output because it’s being generated, with out ready for your entire output to be full. This requires a mechanism to repeatedly monitor the streaming output and apply guardrails in actual time, whereas additionally contemplating the context and coherence of the generated textual content. Moreover, the choice to halt or proceed the era course of based mostly on the guardrail evaluation must be made in actual time, which might affect the responsiveness and person expertise of the appliance.

Resolution overview: Use guardrails on streaming output

To handle the challenges of making use of guardrails on streaming output from LLMs, a method that mixes batching and real-time evaluation is required. This technique includes gathering the streaming output into smaller batches or chunks, evaluating every batch utilizing the ApplyGuardrail API, after which taking applicable actions based mostly on the evaluation outcomes.

Step one on this technique is to batch the streaming output chunks into smaller batches which are nearer to a textual content unit, which is roughly 1,000 characters. If a batch is smaller, reminiscent of 600 characters, you’re nonetheless charged for a complete textual content unit (1,000 characters). For an economical utilization of the API, it’s really useful that the batches of chunks are so as of textual content models, reminiscent of 1,000 characters, 2,000, and so forth. This fashion, you decrease the chance of incurring pointless prices.

By batching the output into smaller batches, you possibly can invoke the ApplyGuardrail API extra steadily, permitting for real-time evaluation and decision-making. The batching course of ought to be designed to keep up the context and coherence of the generated textual content. This may be achieved by ensuring the batches don’t cut up phrases or sentences, and by carrying over any essential context from the earlier batch. Although the chunking varies between use instances, for the sake of simplicity, this publish showcases easy character-level chunking, however it’s really useful to discover choices reminiscent of semantic chunking or hierarchical chunking whereas nonetheless adhering to the rules talked about on this publish.

After the streaming output has been batched into smaller chunks, every chunk could be handed to the API for analysis. The API will assess the content material of every chunk towards the outlined insurance policies and pointers, figuring out any potential violations or delicate info.

The evaluation outcomes from the API can then be used to find out the suitable motion for the present batch. If a extreme violation is detected, the API evaluation suggests halting the era course of, and as a substitute a preset message or response could be exhibited to the person. Nevertheless, in some instances, no extreme violation is detected, however the guardrail was configured to move by way of the request, for instance within the case of sensitiveInformationPolicyConfig to anonymize the detected entities as a substitute of blocking. If such an intervention happens, the output will probably be masked or modified accordingly earlier than being exhibited to the person. For latency-sensitive purposes, it’s also possible to think about creating a number of buffers and a number of guardrails, every with completely different insurance policies, after which processing them with the ApplyGuardrail API in parallel. This fashion, you possibly can decrease the time it takes to make assessments for one guardrail at a time, however maximize getting the assessments from a number of guardrails and a number of batches, although this method hasn’t been carried out on this instance.

Instance use case: Apply guardrails to streaming output

On this part, we offer an instance of how such a method might be carried out. Let’s start with making a guardrail. You should use the next code pattern to create a guardrail in Amazon Bedrock:

import boto3
REGION_NAME = "us-east-1"

bedrock_client = boto3.consumer("bedrock", region_name=REGION_NAME)
bedrock_runtime = boto3.consumer("bedrock-runtime", region_name=REGION_NAME)

response = bedrock_client.create_guardrail(
    identify="",
    description="",
    ...
)
# alternatively present the id and model to your personal guardrail
guardrail_id = response['guardrailId'] 
guardrail_version = response['version']

Correct evaluation of the insurance policies should be performed to confirm if the enter ought to be later despatched to an LLM or whether or not the output generated by the LLM ought to be exhibited to the person. Within the following code, we look at the assessments, that are a part of the response from the ApplyGuardrail API, for potential extreme violation resulting in BLOCKED intervention by the guardrail:

from typing import Checklist, Dict
def check_severe_violations(violations: Checklist[Dict]) -> int:
    """
    When guardrail intervenes both the motion on the request is BLOCKED or NONE.
    This technique checks the variety of the violations resulting in blocking the request.

    Args:
        violations (Checklist[Dict]): A listing of violation dictionaries, the place every dictionary has an 'motion' key.

    Returns:
        int: The variety of extreme violations (the place the 'motion' is 'BLOCKED').
    """
    severe_violations = [violation['action']=='BLOCKED' for violation in violations]
    return sum(severe_violations)

def is_policy_assessement_blocked(assessments: Checklist[Dict]) -> bool:
    """
    Whereas creating the guardrail you might specify a number of kinds of insurance policies.
    On the time of evaluation all of the insurance policies ought to be checked for potential violations
    If there may be even 1 violation that blocks the request, your entire request is blocked
    This technique checks if the coverage evaluation is blocked based mostly on the given assessments.

    Args:
        assessments (listing[dict]): A listing of evaluation dictionaries, the place every dictionary could include 'topicPolicy', 'wordPolicy', 'sensitiveInformationPolicy', and 'contentPolicy' keys.

    Returns:
        bool: True if the coverage evaluation is blocked, False in any other case.
    """
    blocked = []
    for evaluation in assessments:
        if 'topicPolicy' in evaluation:
            blocked.append(check_severe_violations(evaluation['topicPolicy']['topics']))
        if 'wordPolicy' in evaluation:
            if 'customWords' in evaluation['wordPolicy']:
                blocked.append(check_severe_violations(evaluation['wordPolicy']['customWords']))
            if 'managedWordLists' in evaluation['wordPolicy']:
                blocked.append(check_severe_violations(evaluation['wordPolicy']['managedWordLists']))
        if 'sensitiveInformationPolicy' in evaluation:
            if 'piiEntities' in evaluation['sensitiveInformationPolicy']:
                blocked.append(check_severe_violations(evaluation['sensitiveInformationPolicy']['piiEntities']))
            if 'regexes' in evaluation['sensitiveInformationPolicy']:
                blocked.append(check_severe_violations(evaluation['sensitiveInformationPolicy']['regexes']))
        if 'contentPolicy' in evaluation:
            blocked.append(check_severe_violations(evaluation['contentPolicy']['filters']))
    severe_violation_count = sum(blocked)
    print(f'::Guardrail:: {severe_violation_count} extreme violations detected')
    return severe_violation_count>0

We will then outline methods to apply guardrail. If the response from the API results in an motion == 'GUARDRAIL_INTERVENED', it implies that the guardrail has detected a possible violation. We now have to examine if the violation was extreme sufficient to dam the request or move it by way of with both the identical textual content as enter or an alternate textual content through which modifications are made in line with the outlined insurance policies:

def apply_guardrail(textual content, supply, guardrail_id, guardrail_version):
    response = bedrock_runtime.apply_guardrail(
        guardrailIdentifier=guardrail_id,
        guardrailVersion=guardrail_version, 
        supply=supply,
        content material=[{"text": {"text": text}}]
    )
    if response['action'] == 'GUARDRAIL_INTERVENED':
        is_blocked = is_policy_assessement_blocked(response['assessments'])
        alternate_text=" ".be a part of([output['text'] for output in response['output']])
        return is_blocked, alternate_text, response
    else:
        # Return the default response in case of no guardrail intervention
        return False, textual content, response

Let’s now apply our technique for streaming output from an LLM. We will preserve a buffer_text, which creates a batch of chunks acquired from the stream. As quickly as len(buffer_text + new_text) > TEXT_UNIT, that means if the batch is near a textual content unit (1,000 characters), it’s able to be despatched to the ApplyGuardrail API. With this mechanism, we are able to make sure that we don’t incur the pointless value of invoking the API on smaller chunks and likewise that sufficient context is obtainable inside every batch for the guardrail to make significant assessments. Moreover, when the era is full from the LLM, the ultimate batch should even be examined for potential violations. If at any level the API detects extreme violations, additional consumption of the stream is halted and the person is displayed the preset message on the time of creation of the guardrail.

Within the following instance, we ask the LLM to generate three names and inform us what’s a financial institution. This era will result in GUARDRAIL_INTERVENED however not block the era, and as a substitute anonymize the textual content (masking the names) and proceed with era.

input_message = "Checklist 3 names of outstanding CEOs and later inform me what's a financial institution and what are the advantages of opening a financial savings account?"

model_id = "anthropic.claude-3-haiku-20240307-v1:0"
text_unit= 1000 # characters

response = bedrock_runtime.converse_stream(
    modelId=model_id,
    messages=[{
        "role": "user",
        "content": [{"text": input_message}]
    system=[{"text" : "You are an assistant that helps with tasks from users. Be as elaborate as possible"}],
)

stream = response.get('stream')
buffer_text = ""
if stream:
    for occasion in stream:
        if 'contentBlockDelta' in occasion:
            new_text = occasion['contentBlockDelta']['delta']['text']
            if len(buffer_text + new_text) > text_unit:
                is_blocked, alt_text, guardrail_response = apply_guardrail(buffer_text, "OUTPUT", guardrail_id, guardrail_version)
                # print(alt_text, finish="")
                if is_blocked:
                    break
                buffer_text = new_text
            else: 
                buffer_text += new_text

        if 'messageStop' in occasion:
            # print(f"nStop cause: {occasion['messageStop']['stopReason']}")
            is_blocked, alt_text, guardrail_response = apply_guardrail(buffer_text, "OUTPUT", guardrail_id, guardrail_version)
            # print(alt_text)

After operating the previous code, we obtain an instance output with masked names:

Actually! Listed here are three names of outstanding CEOs:

1. {NAME} - CEO of Apple Inc.
2. {NAME} - CEO of Microsoft Company
3. {NAME} - CEO of Amazon

Now, let's talk about what a financial institution is and the advantages of opening a financial savings account.

A financial institution is a monetary establishment that accepts deposits, offers loans, and presents numerous different monetary providers to its clients. Banks play a vital function within the financial system by facilitating the stream of cash and enabling monetary transactions.

Lengthy-context inputs

RAG is a way that enhances LLMs by incorporating exterior information sources. It permits LLMs to reference authoritative information bases earlier than producing responses, producing output tailor-made to particular contexts whereas offering relevance, accuracy, and effectivity. The enter to the LLM in a RAG situation could be fairly lengthy, as a result of it consists of the person’s question concatenated with the retrieved info from the information base. This long-context enter poses challenges when making use of guardrails, as a result of the enter could exceed the character limits imposed by the ApplyGuardrail API. To be taught extra concerning the quotas utilized to Guardrails for Amazon Bedrock, consult with Guardrails quotas.

We evaluated the technique to keep away from the chance from mannequin response within the earlier part. Within the case of inputs, the chance might be each on the question degree or along with the question and the retrieved context for the question.

The retrieved info from the information base could include delicate or doubtlessly dangerous content material, which must be recognized and dealt with appropriately, for instance masking delicate info, earlier than being handed to the LLM for era. Subsequently, guardrails should be utilized to your entire enter to verify it adheres to the outlined insurance policies and constraints.

Resolution overview: Use guardrails on long-context inputs

The ApplyGuardrail API has a default restrict of 25 textual content models (roughly 25,000 characters) per second. If the enter exceeds this restrict, it must be chunked and processed sequentially to keep away from throttling. Subsequently, the technique turns into comparatively simple: if the size of enter textual content is lower than 25 textual content models (25,000 characters), then it may be evaluated in a single request, in any other case it must be damaged down into smaller items. The chunk dimension can range relying on utility habits and the kind of context within the utility; you can begin with 12 textual content models and iterate to search out one of the best appropriate chunk dimension. This fashion, we maximize the allowed default restrict whereas protecting many of the context intact in a single request. Even when the guardrail motion is GUARDRAIL_INTERVENED, it doesn’t imply the enter is BLOCKED. It may be true that the enter is processed and delicate info is masked; on this case, the enter textual content should be recompiled with any processed response from the utilized guardrail.

text_unit = 1000 # characters
limit_text_unit = 25
max_text_units_in_chunk = 12
def apply_guardrail_with_chunking(textual content, guardrail_id, guardrail_version="DRAFT"):
    text_length = len(textual content)
    filtered_text=""
    if text_length <= limit_text_unit * text_unit:
        return apply_guardrail(textual content, "INPUT", guardrail_id, guardrail_version)
    else:
        # If the textual content size is larger than the default textual content unit limits then it is higher to chunk the textual content to keep away from throttling.
        for i, chunk in enumerate(wrap(textual content, max_text_units_in_chunk * text_unit)):
            print(f'::Guardrail::Making use of guardrails at chunk {i+1}')
            is_blocked, alternate_text, response = apply_guardrail(chunk, "INPUT", guardrail_id, guardrail_version)
            if is_blocked:
                filtered_text = alternate_text
                break
            # It might be the case that guardrails intervened and anonymized PII within the enter textual content,
            # we are able to then take the output from guardrails to create filtered textual content response.
            filtered_text += alternate_text
        return is_blocked, filtered_text, response

Run the full pocket book to check this technique with long-context enter.

Finest practices and concerns

When making use of guardrails, it’s important to comply with greatest practices to keep up environment friendly and efficient content material moderation:

• Optimize chunking technique – Fastidiously think about the chunking technique. The chunk dimension ought to steadiness the trade-off between minimizing the variety of API calls and ensuring the context isn’t misplaced as a consequence of overly small chunks. Equally, the chunking technique ought to take note of the context cut up; a crucial piece of textual content may span two (or extra) chunks if not rigorously divided.
• Asynchronous processing – Implement asynchronous processing for RAG contexts. This can assist decouple the guardrail utility course of from the principle utility stream, bettering responsiveness and total efficiency. For steadily retrieved context from vector databases, ApplyGuardrail might be utilized one time and outcomes saved in metadata. This might keep away from redundant API calls for a similar content material. This could considerably enhance efficiency and cut back prices.
• Develop complete take a look at suites – Create a complete take a look at suite that covers a variety of eventualities, together with edge instances and nook instances, to validate the effectiveness of your guardrail implementation.
• Implement fallback mechanisms – There might be eventualities the place the guardrail created doesn’t cowl all of the doable vulnerabilities and is unable to catch edge instances. For such eventualities, it’s clever to have a fallback mechanism. One such possibility might be to convey human within the loop, or use an LLM as a decide to guage each the enter and output.

Along with the aforementioned concerns, it’s additionally good apply to recurrently audit your guardrail implementation, repeatedly refine and adapt your guardrail implementation, and implement logging and monitoring mechanisms to seize and analyze the efficiency and effectiveness of your guardrails.

Clear up

The one useful resource we created on this instance is a guardrail. To delete the guardrail, full the next steps:

1. On the Amazon Bedrock console, underneath Safeguards within the navigation pane, select Guardrails.
2. Choose the guardrail you created and select Delete.

Alternatively, you need to use the SDK:

bedrock_client.delete_guardrail(guardrailIdentifier = "")

Key takeaways

Making use of guardrails is essential for sustaining accountable and secure content material era. With the ApplyGuardrail API from Amazon Bedrock, you possibly can successfully reasonable each inputs and outputs, defending your generative AI utility towards violations and sustaining compliance together with your content material insurance policies.

Key takeaways from this publish embody:

• Perceive the significance of making use of guardrails in generative AI purposes to mitigate dangers and preserve content material moderation requirements
• Use the ApplyGuardrail API from Amazon Bedrock to validate inputs and outputs towards outlined insurance policies and guidelines
• Implement chunking methods for long-context inputs and batching methods for streaming outputs to effectively make the most of the ApplyGuardrail API
• Comply with greatest practices, optimize efficiency, and repeatedly monitor and refine your guardrail implementation to keep up effectiveness and alignment with evolving content material moderation wants

Advantages

By incorporating the ApplyGuardrail API into your generative AI utility, you possibly can unlock a number of advantages:

• Content material moderation at scale – The API permits you to reasonable content material at scale, so your utility stays compliant with content material insurance policies and pointers, even when coping with giant volumes of information
• Customizable insurance policies – You possibly can outline and customise content material moderation insurance policies tailor-made to your particular use case and necessities, ensuring your utility adheres to your group’s requirements and values
• Actual-time moderation – The API permits real-time content material moderation, permitting you to detect and mitigate potential violations as they happen, offering a secure and accountable person expertise
• Integration with any LLM – ApplyGuardrail is an unbiased API, so it may be built-in with any of your LLMs of alternative, so you possibly can reap the benefits of the ability of generative AI whereas sustaining management over the content material being generated
• Price-effective resolution – With its pay-per-use pricing mannequin and environment friendly textual content unit-based billing, the API offers an economical resolution for content material moderation, particularly when coping with giant volumes of information

Conclusion

By utilizing the ApplyGuardrail API from Amazon Bedrock and following one of the best practices outlined on this publish, you may make positive your generative AI utility stays secure, accountable, and compliant with content material moderation requirements, even with long-context inputs and streaming outputs.

To additional discover the capabilities of the ApplyGuardrail API and its integration together with your generative AI utility, think about experimenting with the API utilizing the next sources:

• Confer with Guardrails for Amazon Bedrock for detailed info on the ApplyGuardrail API, its utilization, and integration examples
• Take a look at the AWS samples GitHub repository for pattern code and reference architectures demonstrating the combination of the ApplyGuardrail API with numerous generative AI purposes
• Take part in AWS-hosted workshops and tutorials centered on accountable AI and content material moderation, the place you possibly can be taught from specialists and acquire hands-on expertise with the ApplyGuardrail API

Assets

The next sources clarify each sensible and moral facets of making use of Guardrails for Amazon Bedrock:

Concerning the Writer

Talha Chattha is a Generative AI Specialist Options Architect at Amazon Internet Companies, based mostly in Stockholm. Talha helps set up practices to ease the trail to manufacturing for Gen AI workloads. Talha is an skilled in Amazon Bedrock and helps clients throughout whole EMEA. He holds ardour about meta-agents, scalable on-demand inference, superior RAG options and price optimized immediate engineering with LLMs. When not shaping the way forward for AI, he explores the scenic European landscapes and scrumptious cuisines. Join with Talha at LinkedIn utilizing /in/talha-chattha/.