Safe distributed logging in scalable multi-account deployments utilizing Amazon Bedrock and LangChain

Information privateness is a vital difficulty for software program corporations that present providers within the knowledge administration house. If they need prospects to belief them with their knowledge, software program corporations want to indicate and show that their prospects’ knowledge will stay confidential and inside managed environments. Some corporations go to nice lengths to take care of confidentiality, generally adopting multi-account architectures, the place every buyer has their knowledge in a separate AWS account. By isolating knowledge on the account degree, software program corporations can implement strict safety boundaries, assist stop cross-customer knowledge leaks, and assist adherence with trade laws resembling HIPAA or GDPR with minimal danger.

Multi-account deployment represents the gold normal for cloud knowledge privateness, permitting software program corporations to verify buyer knowledge stays segregated even at huge scale, with AWS accounts offering safety isolation boundaries as highlighted within the AWS Effectively-Architected Framework. Software program corporations more and more undertake generative AI capabilities like Amazon Bedrock, which gives absolutely managed basis fashions with complete safety features. Nevertheless, managing a multi-account deployment powered by Amazon Bedrock introduces distinctive challenges round entry management, quota administration, and operational visibility that might complicate its implementation at scale. Continually requesting and monitoring quota for invoking basis fashions on Amazon Bedrock turns into a problem when the variety of AWS accounts reaches double digits. One strategy to simplify operations is to configure a devoted operations account to centralize administration whereas knowledge from prospects transits by way of managed providers and is saved at relaxation solely of their respective buyer accounts. By centralizing operations in a single account whereas protecting knowledge in several accounts, software program corporations can simplify the administration of mannequin entry and quotas whereas sustaining strict knowledge boundaries and safety isolation.

On this publish, we current an answer for securing distributed logging multi-account deployments utilizing Amazon Bedrock and LangChain.

Challenges in logging with Amazon Bedrock

Observability is essential for efficient AI implementations—organizations can’t optimize what they don’t measure. Observability might help with efficiency optimization, value administration, and mannequin high quality assurance. Amazon Bedrock presents built-in invocation logging to Amazon CloudWatch or Amazon Easy Storage Service (Amazon S3) by way of a configuration on the AWS Administration Console, and particular person logs may be routed to completely different CloudWatch accounts with cross-account sharing, as illustrated within the following diagram.

Routing logs to every buyer account presents two challenges: logs containing buyer knowledge can be saved within the operations account for the user-defined retention interval (no less than 1 day), which could not adjust to strict privateness necessities, and CloudWatch has a restrict of 5 monitoring accounts (buyer accounts). With these limitations, how can organizations construct a safe logging answer that scales throughout a number of tenants and prospects?

On this publish, we current an answer for enabling distributed logging for Amazon Bedrock in multi-account deployments. The target of this design is to offer strong AI observability whereas sustaining strict privateness boundaries for knowledge at relaxation by protecting logs solely inside the buyer accounts. That is achieved by transferring logging to the client accounts moderately than invoking it from the operations account. By configuring the logging directions in every buyer’s account, software program corporations can centralize AI operations whereas imposing knowledge privateness, by protecting buyer knowledge and logs inside strict knowledge boundaries in every buyer’s account. This structure makes use of AWS Safety Token Service (AWS STS) to permit buyer accounts to imagine devoted roles in AWS Identification and Entry Administration (IAM) within the operations account whereas invoking Amazon Bedrock. For logging, this answer makes use of LangChain callbacks to seize invocation metadata instantly in every buyer’s account, making all the course of within the operations account memoryless. Callbacks can be utilized to log token utilization, efficiency metrics, and the general high quality of the mannequin in response to buyer queries. The proposed answer balances centralized AI service administration with robust knowledge privateness, ensuring buyer interactions stay inside their devoted environments.

Answer overview

The whole stream of mannequin invocations on Amazon Bedrock is illustrated within the following determine. The operations account is the account the place the Amazon Bedrock permissions might be managed utilizing an identity-based coverage, the place the Amazon Bedrock shopper might be created, and the place the IAM function with the proper permissions will exist. Each buyer account will assume a special IAM function within the operations account. The client accounts are the place prospects will entry the software program or software. This account will comprise an IAM function that may assume the corresponding function within the operations account, to permit Amazon Bedrock invocations. You will need to be aware that it’s not essential for these two accounts to exist in the identical AWS group. On this answer, we use an AWS Lambda operate to invoke fashions from Amazon Bedrock, and use LangChain callbacks to jot down invocation knowledge to CloudWatch. With out lack of generality, the identical precept may be utilized to different types of compute resembling servers in Amazon Elastic Compute Cloud (Amazon EC2) cases or managed containers on Amazon Elastic Container Service (Amazon ECS).

The sequence of steps in a mannequin invocation are:

1. The method begins when the IAM function within the buyer account assumes the function within the operations account, permitting it to entry the Amazon Bedrock service. That is achieved by way of the AWS STS AssumeRole API operation, which establishes the mandatory cross-account relationship.
2. The operations account verifies that the requesting principal (IAM function) from the client account is allowed to imagine the function it’s focusing on. This verification is predicated on the belief coverage hooked up to the IAM function within the operations account. This step makes certain that solely approved buyer accounts and roles can entry the centralized Amazon Bedrock sources.
3. After belief relationship verification, momentary credentials (entry key ID, secret entry key, and session token) with specified permissions are returned to the client account’s IAM execution function.
4. The Lambda operate within the buyer account invokes the Amazon Bedrock shopper within the operations account. Utilizing momentary credentials, the client account’s IAM function sends prompts to Amazon Bedrock by way of the operations account, consuming the operations account’s mannequin quota.
5. After the Amazon Bedrock shopper response returns to the client account, LangChain callbacks log the response metrics instantly into CloudWatch within the buyer account.

Enabling cross-account entry with IAM roles

The important thing concept on this answer is that there might be an IAM function per buyer within the operations account. The software program firm will handle this function and assign permissions to outline features resembling which fashions may be invoked, during which AWS Areas, and what quotas they’re topic to. This centralized strategy considerably simplifies the administration of mannequin entry and permissions, particularly when scaling to lots of or hundreds of consumers. For enterprise prospects with a number of AWS accounts, this sample is especially precious as a result of it permits the software program firm to configure a single function that may be assumed by a lot of the client’s accounts, offering constant entry insurance policies and simplifying each permission administration and value monitoring. By rigorously crafted belief relationships, the operations account maintains management over who can entry what, whereas nonetheless enabling the flexibleness wanted in advanced multi-account environments.

The IAM function can have assigned a number of insurance policies. For instance, the next coverage permits a sure buyer to invoke some fashions:

{
    "Model": "2012-10-17",
    "Assertion": {
        "Sid": "AllowInference",
        "Impact": "Enable",
        "Motion": [
            "bedrock:Converse",
            "bedrock:ConverseStream",
            "bedrock:GetAsyncInvoke",
            "bedrock:InvokeModel",
            "bedrock:InvokeModelWithResponseStream",
            "bedrock:StartAsyncInvoke"
        ],
        "Useful resource": "arn:aws:bedrock:*::foundation-model/"
    }
}

The management can be carried out on the belief relationship degree, the place we’d solely permit some accounts to imagine that function. For instance, within the following script, the belief relationship permits the function for buyer 1 to solely be assumed by the allowed AWS account when the ExternalId matches a specified worth, with the aim of stopping the confused deputy drawback:

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Sid": "AmazonBedrockModelInvocationCustomer1",
            "Effect": "Allow",
            "Principal": {
                "Service": "bedrock.amazonaws.com"
                },
            "Action": "sts:AssumeRole",
            "Condition": {
                "StringEquals": {
                    "aws:SourceAccount": "",
                    "sts:ExternalId": ""
                },
                "ArnLike": {
                    "aws:SourceArn": "arn:aws:bedrock:::*"
                }
            }
        }
    ]
}

AWS STS AssumeRole operations represent the cornerstone of safe cross-account entry inside multi-tenant AWS environments. By implementing this authentication mechanism, organizations set up a strong safety framework that allows managed interactions between the operations account and particular person buyer accounts. The operations crew grants exactly scoped entry to sources throughout the client accounts, with permissions strictly ruled by the assumed function’s belief coverage and hooked up IAM permissions. This granular management makes certain that the operational crew and prospects can carry out solely approved actions on particular sources, sustaining robust safety boundaries between tenants.

As organizations scale their multi-tenant architectures to embody hundreds of accounts, the efficiency traits and reliability of those cross-account authentication operations turn into more and more vital issues. Engineering groups should rigorously design their cross-account entry patterns to optimize for each safety and operational effectivity, ensuring that authentication processes stay responsive and reliable even because the atmosphere grows in complexity and scale.

When contemplating the service quotas that govern these operations, it’s essential to notice that AWS STS requests made utilizing AWS credentials are topic to a default quota of 600 requests per second, per account, per Area—together with AssumeRole operations. A key architectural benefit emerges in cross-account situations: solely the account initiating the AssumeRole request (buyer account) counts towards its AWS STS quota; the goal account’s (operations account) quota stays unaffected. This uneven quota consumption signifies that the operations account doesn’t deplete their AWS STS service quotas when responding to API requests from buyer accounts. For many multi-tenant implementations, the usual quota of 600 requests per second gives ample capability, although AWS presents quota adjustment choices for environments with distinctive necessities. This quota design allows scalable operational fashions the place a single operations account can effectively service hundreds of tenant accounts with out encountering service limits.

Writing non-public logs utilizing LangChain callbacks

LangChain is a well-liked open supply orchestration framework that allows builders to construct highly effective functions by connecting varied parts by way of chains, that are sequential sequence of operations that course of and rework knowledge. On the core of LangChain’s extensibility is the BaseCallbackHandler class, a basic abstraction that gives hooks into the execution lifecycle of chains, permitting builders to implement customized logic at completely different levels of processing. This class may be prolonged to exactly outline behaviors that ought to happen upon completion of a sequence’s invocation, enabling refined monitoring, logging, or triggering of downstream processes. By implementing customized callback handlers, builders can seize metrics, persist outcomes to exterior methods, or dynamically alter the execution stream based mostly on intermediate outputs, making LangChain each versatile and highly effective for production-grade language mannequin functions.

Implementing a customized CloudWatch logging callback in LangChain gives a strong answer for sustaining knowledge privateness in multi-account deployments. By extending the BaseCallbackHandler class, we are able to create a specialised handler that establishes a direct connection to the client account’s CloudWatch logs, ensuring mannequin interplay knowledge stays inside the account boundaries. The implementation begins by initializing a Boto3 CloudWatch Logs shopper utilizing the client account’s credentials, moderately than the operations account’s credentials. This shopper is configured with the suitable log group and stream names, which may be dynamically generated based mostly on buyer identifiers or software contexts. Throughout mannequin invocations, the callback captures vital metrics resembling token utilization, latency, immediate particulars, and response traits. The next Python script serves for instance of this implementation:

class CustomCallbackHandler(BaseCallbackHandler):

    def log_to_cloudwatch(self, message: str):
        """Operate to jot down extracted metrics to CloudWatch"""

    def on_llm_end(self, response, **kwargs):
        print("nChat mannequin completed processing.")
        # Extract model_id and token utilization from the response
        input_token_count = response.llm_output.get("utilization", {}).get("prompt_tokens", None)
        output_token_count = response.llm_output.get("utilization", {}).get("completion_tokens", None)
        model_id=response.llm_output.get("model_id", None)

        # Right here we invoke the callback
        self.log_to_cloudwatch(
              f"Consumer ID: {self.user_id}nApplication ID: {self.application_id}n Enter tokens: {input_token_count}n Output tokens: {output_token_count}n Invoked mannequin: {model_id}"
             )

    def on_llm_error(self, error: Exception, **kwargs):
        print(f"Chat mannequin encountered an error: {error}")

The on_llm_start, on_llm_end, and on_llm_error strategies are overridden to intercept these lifecycle occasions and persist the related knowledge. For instance, the on_llm_end methodology can extract token counts, execution time, and model-specific metadata, formatting this info into structured log entries earlier than writing them to CloudWatch. By implementing correct error dealing with and retry logic inside the callback, we offer dependable logging even throughout intermittent connectivity points. This strategy creates a complete audit path of AI interactions whereas sustaining strict knowledge isolation within the buyer account, as a result of the logs don’t transit by way of or relaxation within the operations account.

The AWS Shared Accountability Mannequin in multi-account logging

When implementing distributed logging for Amazon Bedrock in multi-account architectures, understanding the AWS Shared Accountability Mannequin turns into paramount. Though AWS secures the underlying infrastructure and providers like Amazon Bedrock and CloudWatch, prospects stay chargeable for securing their knowledge, configuring entry controls, and implementing applicable logging methods. As demonstrated in our IAM function configurations, prospects should rigorously craft belief relationships and permission boundaries to assist stop unauthorized cross-account entry. The LangChain callback implementation outlined locations the duty on prospects to implement correct encryption of logs at relaxation, outline applicable retention intervals that align with compliance necessities, and implement entry controls for who can view delicate AI interplay knowledge. This aligns with the multi-account design precept the place buyer knowledge stays remoted inside their respective accounts. By respecting these safety boundaries whereas sustaining operational effectivity, software program corporations can uphold their obligations inside the shared safety mannequin whereas delivering scalable AI capabilities throughout their buyer base.

Conclusion

Implementing a safe, scalable multi-tenant structure with Amazon Bedrock requires cautious planning round account construction, entry patterns, and operational administration. The distributed logging strategy we’ve outlined demonstrates how organizations can keep strict knowledge isolation whereas nonetheless benefiting from centralized AI operations. Through the use of IAM roles with exact belief relationships, AWS STS for safe cross-account authentication, and LangChain callbacks for personal logging, corporations can create a strong basis that scales to hundreds of consumers with out compromising on safety or operational effectivity.

This structure addresses the vital problem of sustaining knowledge privateness in multi-account deployments whereas nonetheless enabling complete observability. Organizations ought to prioritize automation, monitoring, and governance from the start to keep away from technical debt as their system scales. Implementing infrastructure as code for function administration, automated monitoring of cross-account entry patterns, and common safety opinions will ensure the structure stays resilient and can assist keep adherence with compliance requirements as enterprise necessities evolve. As generative AI turns into more and more central to software program supplier choices, these architectural patterns present a blueprint for sustaining the very best requirements of information privateness whereas delivering revolutionary AI capabilities to prospects throughout numerous regulatory environments and safety necessities.

To study extra, discover the great Generative AI Safety Scoping Matrix by way of Securing generative AI: An introduction to the Generative AI Safety Scoping Matrix, which gives important frameworks for securing AI implementations. Constructing on these safety foundations, strengthen Amazon Bedrock deployments by getting aware of IAM authentication and authorization mechanisms that set up correct entry controls. As organizations develop to require multi-account constructions, these IAM practices join seamlessly with AWS STS, which delivers momentary safety credentials enabling safe cross-account entry patterns. To finish this built-in safety strategy, delve into LangChain and LangChain on AWS capabilities, providing highly effective instruments that construct upon these foundational safety providers to create safe, context-aware AI functions, whereas sustaining applicable safety boundaries throughout your complete generative AI workflow.

Concerning the Authors

Mohammad Tahsin is an AI/ML Specialist Options Architect at AWS. He lives for staying up-to-date with the newest applied sciences in AI/ML and serving to prospects deploy bespoke options on AWS. Exterior of labor, he loves all issues gaming, digital artwork, and cooking.

Felipe Lopez is a Senior AI/ML Specialist Options Architect at AWS. Previous to becoming a member of AWS, Felipe labored with GE Digital and SLB, the place he targeted on modeling and optimization merchandise for industrial functions.

Aswin Vasudevan is a Senior Options Architect for Safety, ISV at AWS. He’s a giant fan of generative AI and serverless structure and enjoys collaborating and dealing with prospects to construct options that drive enterprise worth.