Accessing OpenAI GPT fashions on Amazon Bedrock with international cross-Area inference in Thailand, Malaysia, Singapore, Indonesia, and Taiwan

Single-Area inference can throttle when demand exceeds native capability. Groups in Thailand, Malaysia, Singapore, Indonesia, and Taiwan want entry to the latest OpenAI fashions past that capability. Amazon Bedrock affords OpenAI GPT-6 Sol and GPT-6 Luna via international cross-Area inference from all 5 AWS Areas, and GPT-6 Astra from Asia Pacific (Singapore). This put up reveals you how one can configure the decision, set permissions, and monitor inference out of your supply Area.

GPT-6 Sol, GPT-6 Luna, and GPT-6 Astra are all obtainable via international cross-Area inference from the previous Areas, on the bedrock-runtime endpoint. See Cross-Area inference within the Amazon Bedrock Consumer Information. For the complete record of supply and vacation spot Areas in every profile, see Supported Areas and fashions for inference profiles.

On this put up, we introduce the worldwide cross-Area inference profiles and invoke them from Asia Pacific (Singapore) utilizing the Converse API, the OpenAI-compatible APIs, and coding purchasers. We then cowl the AWS Identification and Entry Administration (IAM) and repair management coverage (SCP) configuration, quota administration, and monitoring.

International cross-Area inference profiles for OpenAI GPT-6

The GPT-6 household affords three fashions, every obtainable via international cross-Area inference.

• GPT-6 Sol brings robust reasoning and coding capabilities to advanced duties carried out all through the week.
• GPT-6 Luna makes centered, repeatable duties sensible at excessive quantity.
• GPT-6 Astra handles advanced reasoning, coding, and long-document work.

All three settle for textual content and picture enter, client-side instrument use, and immediate caching. They differ on reasoning effort. GPT-6 Sol and GPT-6 Luna settle for none, low, medium, excessive, xhigh, and max. GPT-6 Astra accepts the identical ranges besides none. For GPT-6 Sol and GPT-6 Luna, see Convey extra intelligence to on a regular basis work with GPT-6 Sol and GPT-6 Luna on Amazon Bedrock. For GPT-6 Astra, see Take in your most bold work with GPT-6 Astra on Amazon Bedrock.

International cross-Area inference routes requests throughout supported business AWS Areas the place the mannequin is deployed, providing you with broad entry to Amazon Bedrock capability. You invoke it by passing a world cross-Area inference profile ID instead of a mannequin ID. Your supply Area is the place you make the API name. Amazon Bedrock selects the vacation spot Area for processing. Billing, quota monitoring, AWS CloudTrail, and Amazon CloudWatch all keep in your supply Area.

The next desk lists the worldwide cross-Area inference profile IDs and the supply Areas coated on this put up.

Inference profile availability and Area assist change as AWS provides Areas and profiles, so verify each on the mannequin card earlier than you construct: GPT-6 Sol, GPT-6 Luna, and GPT-6 Astra. For per-token charges, see Amazon Bedrock pricing.

Getting began with OpenAI fashions on Amazon Bedrock

Begin with the stipulations, then invoke any of the three profiles via the Converse API, the OpenAI-compatible APIs, or your current coding shopper.

Stipulations

1. An IAM identification with permission to invoke the profile. The IAM coverage within the Safety and permissions part grants solely what these examples want.
2. Your supply Area activated. Solely the Area you name from wants activating. You don’t activate vacation spot Areas, which run on Amazon Bedrock managed capability. Singapore (ap-southeast-1) is a Area that’s enabled by default. Jakarta (ap-southeast-3), Malaysia (ap-southeast-5), Thailand (ap-southeast-7), and Taipei (ap-east-2) are opt-in Areas it’s essential to activate first. For extra data, see Allow or disable AWS Areas in your account.
3. A present AWS SDK or CLI. The OpenAI-compatible examples additionally require the OpenAI SDK and the Amazon Bedrock token generator. Set up them with pip set up boto3 openai aws-bedrock-token-generator.

The Amazon Bedrock console lists all three international cross-Area inference profiles beneath system-defined inference profiles. See Determine 1.

Determine 1: The three international OpenAI GPT-6 inference profiles among the many system-defined profiles within the Amazon Bedrock console

Invoke GPT-6 fashions via Amazon Bedrock Runtime

Every mannequin helps three entry paths on this endpoint: the Amazon Bedrock Converse API, the OpenAI Responses API, and the OpenAI Chat Completions API. The OpenAI-compatible APIs are known as on the /openai/v1 paths relatively than via the AWS SDKs, and the endpoint accepts both Signature Model 4 (SigV4) or an Amazon Bedrock API key. Cross the worldwide cross-Area inference profile ID because the mannequin in each case.

Converse API. This instance calls GPT-6 Sol from Asia Pacific (Singapore) and logs the response. Set BEDROCK_PROFILE_ID to international.openai.gpt-6-luna or international.openai.gpt-6-astra to modify fashions, or name converse_stream for streaming.

import logging
import os
import boto3

# AWS_REGION overrides this default, so set it to your supply Area or unset it.
REGION = os.environ.get("AWS_REGION", "ap-southeast-1")
PROFILE_ID = os.environ.get("BEDROCK_PROFILE_ID", "international.openai.gpt-6-sol")
PROMPT = (
    "In three bullet factors, clarify when a workload ought to use "
    "cross-Area inference as an alternative of calling a mannequin in a single Area."
)

logging.basicConfig(format="%(levelname)s %(message)s")
log = logging.getLogger("bedrock-cris")
log.setLevel(logging.INFO)

shopper = boto3.shopper("bedrock-runtime", region_name=REGION)

# AccessDeniedException factors on the IAM coverage. ValidationException normally
# means the mannequin ID shouldn't be a world cross-Area inference profile.
response = shopper.converse(
    modelId=PROFILE_ID,
    messages=[{"role": "user", "content": [{"text": PROMPT}]}],
    inferenceConfig={"maxTokens": 1024},
    # GPT-6 Sol and Luna settle for none via max. GPT-6 Astra rejects none.
    additionalModelRequestFields={"reasoning": {"effort": "low"}},
)

# Any effort above none can prepend a reasoningContent block, so decide the textual content
# block by key relatively than by place.
textual content = "".be part of(b["text"] for b in response["output"]["message"]["content"] if "textual content" in b)

log.information("Referred to as %s from %s", PROFILE_ID, REGION)
log.information("Cease cause: %s", response["stopReason"])
log.information("Response:npercents", textual content)

OpenAI Chat Completions API. Level the OpenAI SDK on the Regional endpoint and cross the identical profile ID.

import logging
import os
from aws_bedrock_token_generator import provide_token
from openai import OpenAI

REGION = os.environ.get("AWS_REGION", "ap-southeast-1")
PROFILE_ID = os.environ.get("BEDROCK_PROFILE_ID", "international.openai.gpt-6-sol")
PROMPT = "Record three makes use of for a message queue."

logging.basicConfig(format="%(levelname)s %(message)s")
log = logging.getLogger("bedrock-cris")
log.setLevel(logging.INFO)

# provide_token mints a short-term key out of your AWS credentials, legitimate as much as 12 hours.
shopper = OpenAI(
    base_url=f"https://bedrock-runtime.{REGION}.amazonaws.com/openai/v1",
    api_key=provide_token(area=REGION),
)

response = shopper.chat.completions.create(
    mannequin=PROFILE_ID,
    messages=[{"role": "user", "content": PROMPT}],
    max_completion_tokens=256,
)

log.information("Referred to as %s from %s", PROFILE_ID, REGION)
log.information("Response:npercents", response.decisions[0].message.content material)

When migrating an current OpenAI shopper:

• Set the mannequin parameter on each request, together with these referencing a earlier response.
• Use supported instrument varieties: operate, mcp, customized, namespace, and tool_search.
• Configure the inference profile ID straight in your shopper relatively than querying the fashions endpoint.

Arrange the ChatGPT desktop app and Codex CLI

The ChatGPT desktop app and the Codex CLI share ~/.codex/config.toml, so one change covers each. Of the 2 built-in Amazon Bedrock suppliers, solely amazon-bedrock-runtime accepts an inference profile.

mannequin = "international.openai.gpt-6-astra"
model_provider = "amazon-bedrock-runtime"
[model_providers.amazon-bedrock-runtime.aws]
area = "ap-southeast-1"

SigV4 signing means there is no such thing as a key to rotate. Add profile = "" for a non-default profile, and set area explicitly, as a result of the shopper in any other case falls again to AWS_REGION. Substitute one other profile ID for mannequin to alter fashions.

Codex studies a 272,000-token window for these profiles, which is its personal bundled determine relatively than a service restrict. Increase it with model_context_window. This configuration wants Codex CLI 0.148 or later.

Determine 2 reveals the end result within the app.

Determine 2: The ChatGPT desktop app operating GPT-6 Astra on Amazon Bedrock

Safety and permissions

Amazon Bedrock affords a spread of controls over inference. Two of them matter right here. AWS Identification and Entry Administration (IAM) insurance policies management which roles can invoke which profiles, and repair management insurance policies (SCPs) apply organization-wide controls, together with Area scope. This part covers each, and the place request knowledge lands.

IAM permissions for international cross-Area inference

To name any of those fashions via a world cross-Area inference profile, your IAM coverage wants 4 statements:

1. Entry to the inference profile and the challenge/default useful resource in your supply Area.
2. Entry to the inspiration mannequin in your supply Area, scoped to the inference profile.
3. The Area-agnostic Amazon Useful resource Title (ARN) for the inspiration mannequin, arn:aws:bedrock:::foundation-model/openai.gpt-6-sol, with aws:RequestedRegion set to unspecified to allow cross-Area routing.
4. bedrock:CallWithBearerToken for API-key authentication.

This IAM coverage grants permission to invoke GPT-6 Sol via international cross-Area inference from ap-southeast-1. Substitute together with your AWS account ID and ap-southeast-1 together with your supply Area.

{
  "Model": "2012-10-17",
  "Assertion": [
    {
      "Sid": "GrantGlobalCrisProfileAndProjectAccess",
      "Effect": "Allow",
      "Action": ["bedrock:InvokeModel"],
      "Useful resource": [
        "arn:aws:bedrock:ap-southeast-1::inference-profile/global.openai.gpt-6-sol",
        "arn:aws:bedrock:ap-southeast-1::project/default"
      ],
      "Situation": {
        "StringEquals": { "aws:RequestedRegion": "ap-southeast-1" }
      }
    },
    {
      "Sid": "GrantGlobalCrisInRegionModelAccess",
      "Impact": "Permit",
      "Motion": ["bedrock:InvokeModel"],
      "Useful resource": ["arn:aws:bedrock:ap-southeast-1::foundation-model/openai.gpt-6-sol"],
      "Situation": {
        "StringEquals": {
          "aws:RequestedRegion": "ap-southeast-1",
          "bedrock:InferenceProfileArn": "arn:aws:bedrock:ap-southeast-1::inference-profile/international.openai.gpt-6-sol"
        }
      }
    },
    {
      "Sid": "GrantGlobalCrisGlobalModelAccess",
      "Impact": "Permit",
      "Motion": ["bedrock:InvokeModel"],
      "Useful resource": ["arn:aws:bedrock:::foundation-model/openai.gpt-6-sol"],
      "Situation": {
        "StringEquals": {
          "aws:RequestedRegion": "unspecified",
          "bedrock:InferenceProfileArn": "arn:aws:bedrock:ap-southeast-1::inference-profile/international.openai.gpt-6-sol"
        }
      }
    },
    {
      "Sid": "AllowBearerTokenAuth",
      "Impact": "Permit",
      "Motion": ["bedrock:CallWithBearerToken"],
      "Useful resource": "*"
    }
  ]
}

In the event you use streaming (converse_stream or stream=True), add bedrock:InvokeModelWithResponseStream to the Motion within the first three statements. Converse authorizes as bedrock:InvokeModel. In the event you solely name Converse with SigV4 credentials, drop the fourth assertion.

To change fashions, substitute openai.gpt-6-sol with openai.gpt-6-luna or openai.gpt-6-astra all through. For the entire reference, see IAM coverage necessities for international cross-Area inference.

Service management coverage issues for international cross-Area inference

A worldwide cross-Area inference request carries no Area identify. Amazon Bedrock units aws:RequestedRegion to unspecified, so a Area allowlist in an SCP denies the decision, and the error names solely the coverage sort, not which coverage. That may be a widespread cause a primary name fails in an enterprise account.

Slightly than widening the allowlist, scope the exception. Amazon Bedrock populates the bedrock:InferenceProfileArn situation key throughout authorization, so an SCP can enable these profiles whereas protecting Area controls for every part else. The next SCP does that.

{
  "Model": "2012-10-17",
  "Assertion": [
    {
      "Sid": "DenyOutsideApprovedRegionsExceptBedrockInvoke",
      "Effect": "Deny",
      "NotAction": [
        "iam:*", "sts:*", "organizations:*", "account:*",
        "route53:*", "cloudfront:*", "waf:*", "wafv2:*", "shield:*",
        "support:*", "trustedadvisor:*", "health:*",
        "budgets:*", "ce:*", "cur:*", "tax:*", "billing:*",
        "bedrock:Invoke*"
      ],
      "Useful resource": "*",
      "Situation": {
        "StringNotEquals": { "aws:RequestedRegion": ["ap-southeast-1"] }
      }
    },
    {
      "Sid": "DenyBedrockInferenceOutsideApprovedRegionsUnlessCris",
      "Impact": "Deny",
      "Motion": ["bedrock:Invoke*"],
      "Useful resource": "*",
      "Situation": {
        "StringNotEquals": { "aws:RequestedRegion": ["ap-southeast-1"] },
        "ArnNotLike": {
          "bedrock:InferenceProfileArn": [
            "arn:aws:bedrock:*:*:inference-profile/global.*"
          ]
        }
      }
    }
  ]
}

The primary assertion enforces your Area allowlist for each service besides Amazon Bedrock inference. The second denies that inference exterior these Areas until it routes via a world cross-Area inference profile. For geographic profiles, add their prefix (for instance inference-profile/us.*) to the ArnNotLike record, as a result of geographic requests carry an actual Area identify. Maintain your supply Area within the allowlist, and add the worldwide companies your group makes use of to NotAction.

A blunter different is so as to add unspecified to your current allowlist, which works however opens the identical exception to each different service. For a ready-to-use coverage file, see the aws-samples service-control-policy-examples repository.

Information retention and abuse detection

For automated offline abuse detection, classifier-flagged site visitors is retained by AWS for as much as 30 days and processed programmatically. Underneath cross-Area inference that retention occurs within the vacation spot Area, which for a world profile is any supported business AWS Area the place the mannequin is deployed. Eligible prospects would possibly request full zero knowledge retention via their AWS account staff. For extra data, see Amazon Bedrock abuse detection and Information retention.

In case your group has knowledge residency necessities, take this to your danger and compliance staff early. See the AWS Monetary Providers Compliance Middle nation pages for these 5 international locations. We suggest you acquire your personal recommendation on compliance with the necessities related to your enterprise.

Quota administration

GPT-6 quotas on bedrock-runtime are measured in tokens per minute (TPM), tracked in opposition to the supply Area you name from. Enter and cache-write tokens rely 1:1, and every output token consumes 10 tokens of quota. For instance, a request with 2,000 enter tokens and 1,000 output tokens depletes 12,000 tokens out of your quota.

Amazon Bedrock reserves the complete max_tokens up entrance, so set it near the completions you count on. One quota covers InvokeModel, Converse, Responses, and Chat Completions. Assessment and lift it on the Service Quotas console out of your supply Area. Earlier than manufacturing, request will increase early and alarm on EstimatedTPMQuotaUsage within the AWS/Bedrock namespace at 70–80 % of your allocation. See How tokens are counted in Amazon Bedrock and Amazon Bedrock endpoints and quotas for present values.

Monitoring and logging

Your operational floor stays in your supply Area. Amazon CloudWatch metrics, AWS CloudTrail occasions, mannequin invocation logs, quota accounting, and billing all stay within the Area you known as from. You pay that Area’s charges, not the processing Area’s.

CloudTrail data the place every request was processed. AWS paperwork a world cross-Area inference profile’s locations as its business Areas relatively than enumerating them. GetInferenceProfile returns the Area-agnostic and source-Area mannequin ARNs relatively than a vacation spot record. Your per-request file is additionalEventData.inferenceRegion in CloudTrail in your supply Area. Inference on this endpoint is a administration occasion, logged by default at no further price. CloudTrail Occasion Historical past retains 90 days, so longer horizons want a path or mannequin invocation logging. Retrieve the CloudTrail occasions with this command.

aws cloudtrail lookup-events 
  --region ap-southeast-1 
  --lookup-attributes AttributeKey=EventSource,AttributeValue=bedrock.amazonaws.com 
  --start-time  
  --end-time  
  --max-results 1 
  --query 'Occasions[0].CloudTrailEvent' --output textual content | python3 -m json.instrument

Set the window round a name you simply made, as a result of CloudTrail supply lags by a couple of minutes. If the latest occasion was a failed name, it carries an errorCode and no inferenceRegion. Increase --max-results and decide an occasion with out errorCode.

{
  "eventName": "Converse",
  "eventSource": "bedrock.amazonaws.com",
  "awsRegion": "ap-southeast-1",
  "managementEvent": true,
  "requestParameters": {
    "modelId": "international.openai.gpt-6-sol",
    "inferenceConfig": { "maxTokens": 1024 }
  },
  "additionalEventData": {
    "inferenceRegion": "us-east-1"
  }
}

Metrics and coding-agent utilization keep in CloudWatch. The place builders attain GPT-6 via a coding agent, CloudWatch Coding Agent Insights provides ready-made dashboards beneath GenAI Observability that break token use down by consumer, staff, and value heart. The brokers emit OpenTelemetry metrics natively. For Codex, see Arrange OpenTelemetry for OpenAI Codex.

Clear up

This put up provides an IAM coverage and optionally an SCP assertion and a logging vacation spot. Detach or delete the IAM coverage you connected, and revert the SCP entry should you added one. Delete the mannequin invocation logging vacation spot in Amazon Easy Storage Service (Amazon S3) or Amazon CloudWatch Logs should you created one, to keep away from storage fees. The worldwide cross-Area inference profiles are system-defined, so there’s nothing to delete.

Conclusion

This put up confirmed how one can invoke GPT-6 Sol and GPT-6 Luna from 5 Asia Pacific supply Areas, and GPT-6 Astra from Asia Pacific (Singapore), via international cross-Area inference on bedrock-runtime. Cross a world cross-Area inference profile ID because the mannequin parameter, and configure the IAM coverage and SCP exception from this put up. The identical endpoint serves the OpenAI-compatible APIs, so an current OpenAI integration strikes throughout by altering a base URL and a key.

To get began, connect the IAM coverage from this put up, and ship a primary request via international.openai.gpt-6-sol. Dimension your tokens-per-minute quota on the 10x burndown charge, and make sure in CloudTrail that additionalEventData.inferenceRegion seems in your calls.

For extra data, see the Amazon Bedrock Consumer Information, Route mannequin inference requests throughout AWS Areas with cross-Area inference, and Amazon Bedrock pricing.

In regards to the authors