Multi-Flip Reinforcement Studying on Amazon SageMaker AI: A hands-on information

Most fine-tuning teaches a mannequin to supply a greater reply to a single immediate. Agentic duties are completely different. An agent that recordsdata an expense report or triages a assist ticket takes a sequence of actions, and every step is dependent upon what got here earlier than. There’s not often one right sequence to mimic, and the one dependable suggestions is whether or not the entire sequence succeeded. Supervised fine-tuning has no labels for that, and single-turn reinforcement studying can’t credit score a call made three steps earlier than the end result.

Multi-Flip Reinforcement Studying (MTRL) on Amazon SageMaker AI trains fashions on precisely that sign. SageMaker AI sends prompts to an agent you host on Amazon Bedrock AgentCore, or by yourself infrastructure, comparable to Amazon Elastic Kubernetes Service (Amazon EKS), Amazon Elastic Compute Cloud (Amazon EC2), or AWS Fargate, by means of an AWS Lambda forwarder. The instance right here makes use of AgentCore. The agent calls the coverage mannequin being skilled, acts in your surroundings over as many turns as the duty wants, and experiences one reward when the episode ends. You don’t specify the precise trajectory, solely the way to rating the consequence.

This submit makes use of Wordle, a sport by which a participant has six tries to guess a hidden five-letter phrase, as the instance. After every guess, each letter turns inexperienced whether it is in the precise place, yellow if it’s within the phrase however in a distinct place, and grey if it isn’t within the phrase. The sport fits MTRL as a result of the foundations are few, every guess is dependent upon the suggestions from the final, and the end result is scored objectively. Wordle is a instructing instance, and frontier fashions already play it nicely. In observe, you’ll use MTRL to make a small, low-cost mannequin good at a multi-step process particular to your corporation, comparable to an inner workflow that public fashions had been by no means skilled on. The identical pipeline applies to that setting.

In our run, MTRL took gpt-oss-20b from a 61 to an 84 % remedy charge on held-out phrases at its default reasoning effort, and from 15 to 69 % on the low effort it skilled at, for about $339 in coaching tokens. On this submit, we stroll by means of the pipeline end-to-end: deploying the agent and reward to AgentCore, getting ready the dataset, coaching, monitoring in MLflow, and evaluating at serving settings. We additionally share the reward-design errors and coaching failures we hit. Your datasets, agent, and skilled mannequin keep in your AWS account and usually are not used to coach the bottom fashions. All code is within the accompanying aws-samples repository.

Answer overview

The system has two halves. SageMaker AI owns the coaching job and the coverage mannequin. AgentCore hosts the agent, the Wordle surroundings, and the reward perform. The next diagram reveals how they join.

Determine 1: MTRL on SageMaker AI, the place the coaching job invokes the AgentCore-hosted agent as soon as per rollout to pattern the coverage mannequin, play the sport, and report a reward

For each rollout, the MTRL service invokes the AgentCore runtime, a functionality of Amazon Bedrock AgentCore, with one coaching row, which holds one secret phrase. A Strands agent within the runtime calls the coverage mannequin by means of the SageMaker Job Runtime’s OpenAI-compatible endpoint. The mannequin performs as much as six guesses by means of one device, guess_word, and after every guess the surroundings returns per-letter suggestions and a abstract of what’s identified. When the sport ends, the agent returns one reward. Group Relative Coverage Optimization (GRPO) makes use of the unfold of rewards throughout a number of rollouts of the identical phrase to replace a Low-Rank Adaptation (LoRA) adapter on the coverage mannequin.

Three design factors matter for what follows:

• The agent and reward reside in a single place. A single Python file deployed to AgentCore holds the surroundings, the device, and the scoring perform.
• The dataset carries solely the key phrase. The phrase lists bundled with the agent are constructed from the open-source SCOWL phrase lists and are used just for scoring.
• The reward is outcome-dominant. A remedy pays 1.05–1.55 relying on velocity, a loss pays -0.5, and by no means making a legitimate guess pays -1.5. A small shaping time period rewards good technique inside every tier.

Supported fashions and AWS Areas

MTRL is serverless: you choose a base mannequin and the service provisions the coaching and sampling infrastructure. On the time of writing, it helps the next base fashions, every skilled with a LoRA adapter:

We selected GPT-OSS-20B as a result of it has the bottom per-token worth of the 4 and its reasoning effort is controllable per request. See the MTRL documentation for the present listing.

Pricing

You don’t select cases or measurement a cluster. You pay per million tokens in three dimensions, priced by base mannequin:

• Prefill: enter tokens the coverage mannequin processes. These accumulate in a multi-turn process, as a result of each flip re-reads the historical past.
• Pattern: tokens the coverage mannequin generates, together with reasoning tokens.
• Prepare: tokens processed within the backward cross that updates the LoRA adapter.

The 100-step run consumed 491.6 million prefill, 159.7 million pattern, and 643.3 million prepare tokens throughout 105,486 rollouts in 12 hours 40 minutes. At GPT-OSS-20B listing costs within the US East (N. Virginia) Area (us-east-1) on the time of writing, that’s about $339:

The prepare dimension dominates and scales with trajectory size, so shorter video games and fewer verbose reasoning lower the invoice. The AgentCore runtime payments individually for the vCPU and reminiscence the agent makes use of. For this run, that was about $79, from the runtime’s Amazon CloudWatch utilization metrics at us-east-1 listing costs. DescribeJob experiences the token counts below BillableTokenUsage. See Amazon SageMaker AI pricing for present charges.

Conditions

To comply with this walkthrough, you want the next. See additionally the MTRL conditions.

• Python 3.10 or later, uv, Node 20 or later, and the AgentCore CLI (npm set up -g @aws/agentcore). See Get began with Amazon Bedrock AgentCore for set up and the IAM permissions the CLI wants.
• botocore 1.43.37 or later, on the machine that launches the job. That is the primary model that features the AgentRFT CreateJob operation.
• Two AWS Id and Entry Administration (IAM) roles, described in Step 1.
• An Amazon Easy Storage Service (Amazon S3) bucket in the identical Area because the agent. The coaching job should additionally run in that Area, as a result of CreateJob rejects a cross-Area runtime ARN.

Step 1: Create the IAM roles

MTRL makes use of two roles, and one among them has a belief coverage that the overall SageMaker AI documentation doesn’t point out.

Agent runtime position. agentcore deploy creates this position for you. It trusts bedrock-agentcore.amazonaws.com and desires the AmazonSageMakerJobRuntimeAccess managed coverage, which lets the agent name the coverage mannequin and report rewards. The CDK stack within the pattern repository attaches this coverage routinely.

Coaching job position. You create this one. It should belief job.sagemaker.amazonaws.com with each sts:AssumeRole and sts:TagSession, and it wants the AmazonSageMakerJobFullAccess managed coverage.

{
    "Model": "2012-10-17",
    "Assertion": [{
        "Effect": "Allow",
        "Principal": {"Service": "job.sagemaker.amazonaws.com"},
        "Action": ["sts:AssumeRole", "sts:TagSession"]
    }]
}

Observe: The belief principal is job.sagemaker.amazonaws.com, not sagemaker.amazonaws.com. With the generic principal, CreateJob fails with “Couldn’t assume position,” and the message names the proper principal solely after retries.

Step 2: Deploy the agent to Amazon Bedrock AgentCore

Clone the pattern repository, set your account and Area in Wordle/agentcore/aws-targets.json, and deploy:

git clone https://github.com/aws-samples/sample-multi-turn-rl-wordle
cd sample-multi-turn-rl-wordle/Wordle
agentcore validate
agentcore deploy

The CLI builds the agent’s dependencies for the runtime’s Linux ARM64 platform, creates the execution position with AmazonSageMakerJobRuntimeAccess hooked up, and deploys by means of AWS CloudFormation. We advocate it over importing a code package deal on the console, the place it’s frequent to get the roughly 700 MB of compiled dependencies or the coverage attachment unsuitable. Observe the runtime ARN within the output for the coaching job.

Determine 2: The Wordle agent runtime within the Amazon Bedrock AgentCore console with standing Prepared

The runtime reads two surroundings variables from agentcore.json. REASONING_EFFORT=low units reasoning_effort on each request to the coverage mannequin. In our analysis jobs, low effort used 63 to 83 % fewer pattern tokens per sport than medium. REASONING_PROMPT_HINT=true additionally provides “Reasoning: low” to the system immediate, which is how gpt-oss reads the setting in its Concord chat format. Neither variable provides the mannequin any trace concerning the secret phrase.

Determine 3: Runtime surroundings variables displaying REASONING_EFFORT set to low

The agent is one file, essential.py. Simplified, its entrypoint seems to be like the next. In your personal process, you modify the duty parsing, the instruments, and the reward:

app = BedrockAgentCoreApp()

@app.entrypoint  # outermost: begins the HTTP server AgentCore invokes
@sagemaker_rft_handler  # experiences the returned reward to SageMaker AI
def handle_rollout(payload):
    immediate, reply = parse_task(payload["prompt"])
    sport = WordleGame(reply)

    @device
    def guess_word(guess: str) -> str:
        """Submit a 5-letter Wordle guess and get suggestions."""
        return sport.guess(guess)

    agent = Agent(mannequin=build_model(), system_prompt=SYSTEM_PROMPT, instruments=[guess_word])
    agent(immediate)
    return {"reward": sport.final_reward()}

Necessary: The sagemaker_rft_handler decorator doesn’t begin an HTTP server by itself. Wrap it with BedrockAgentCoreApp and @app.entrypoint, with the entrypoint decorator outermost. With out this, each invocation instances out after 30 seconds.

Step 3: Confirm the runtime

Earlier than spending GPU time, affirm the entire chain works with a payload formed just like the one the coaching service sends. The repository consists of one as Wordle/payload.json:

{
  "immediate": "{"immediate": "Guess the 5-letter phrase", "reply": "seashore", "id": "smoke-test"}",
  "metadata": {
    "jobArn": "arn:aws:sagemaker:us-east-1:111122223333:job/AgentRFT/smoke-test",
    "rolloutId": "r1"
  }
}

The pretend job ARN is predicted to fail, and the way in which it fails is the proof. Put your individual account ID into it first: for a job ARN in one other account, the service returns a 403 earlier than it checks whether or not the job exists, which seems to be like a damaged IAM setup. From the Wordle listing, run:

ACCOUNT=$(aws sts get-caller-identity --query Account --output textual content)
sed "s/111122223333/$ACCOUNT/" payload.json > /tmp/payload.json

aws bedrock-agentcore invoke-agent-runtime 
  --agent-runtime-arn  
  --runtime-session-id smoke-test-$(date +%s)000000000000000 
  --payload fileb:///tmp/payload.json /dev/stdout

Within the runtime’s Amazon CloudWatch logs, you see the agent parse the immediate, authenticate to the Job Runtime endpoint, and obtain a 400 error studying “Couldn’t discover job.” That failure confirms that parsing, authentication, and error reporting work. The command itself prints {“reward”: -1.5}, as a result of the agent by no means reached the coverage mannequin. A 403 studying “not approved to entry this useful resource” means the job ARN nonetheless names one other account. A 30-second timeout as an alternative means the HTTP server shouldn’t be beginning (see the Necessary notice in Step 2).

Step 4: Put together the dataset

The MTRL dataset has no goal responses, transcripts, or labels. Every row is one process for the agent to aim. Listed below are the primary three rows of the coaching file:

{"immediate": "{"immediate": "Guess the 5-letter phrase", "reply": "riser", "id": "wordle_train_0000"}"}
{"immediate": "{"immediate": "Guess the 5-letter phrase", "reply": "swoon", "id": "wordle_train_0001"}"}
{"immediate": "{"immediate": "Guess the 5-letter phrase", "reply": "soiled", "id": "wordle_train_0002"}"}

The service reads the immediate column (or the primary column) and passes the string to your agent verbatim, with out parsing it. We made the string a small JSON object with three fields:

• immediate: the instruction the agent reveals the coverage mannequin, an identical on each row.
• reply: the key phrase. Solely the surroundings and the reward perform use it. The coverage mannequin by no means sees it.
• id: a row ID for tracing a rollout within the logs again to its supply row.

The pattern repository’s make_dataset.py attracts 640 coaching phrases (5 full batches of 128) and 128 validation phrases from the reply listing with no overlap, so the validation remedy charge measures generalization. The service additionally accepts Apache Parquet, JSON arrays, and CSV. Add each recordsdata to Amazon S3:

uv run python make_dataset.py
aws s3 cp training-data.jsonl s3:///wordle-mtrl/prepare/
aws s3 cp validation-data.jsonl s3:///wordle-mtrl/validation/

As a result of the service by no means inspects the immediate string, a manufacturing process can pack a full message listing, device configuration, or reward specification into it. See the MTRL documentation for examples, together with the way to defend delicate content material within the string.

Step 5: Launch the coaching job

The launcher makes use of the MultiTurnRLTrainer class from the sagemaker-train SDK. Level it on the runtime ARN from Step 2, the position from Step 1, the datasets from Step 4, and an MLflow app for metrics. The MLflow app is required, so for those who don’t have one, create it first with aws sagemaker create-mlflow-app:

uv run python run_mtrl_training.py 
  --model openai-reasoning-gpt-oss-20b 
  --agent-runtime-arn  
  --role-arn arn:aws:iam:::position/ 
  --train-dataset s3:///wordle-mtrl/prepare/training-data.jsonl 
  --val-dataset s3:///wordle-mtrl/validation/validation-data.jsonl 
  --s3-output-path s3:///wordle-mtrl/output/ 
  --mlflow-app-arn arn:aws:sagemaker:::mlflow-app/

The launcher overrides three hyperparameters and leaves each different one on the gpt-oss-20b default. It units sampling_max_tokens to 8192, the service most, as a result of on the default of 4096 gpt-oss usually ran out of tokens whereas reasoning, earlier than its first device name. It units max_epochs to twenty and max_steps to 100: 640 rows on the default global_batch_size of 128 is 5 steps per epoch, and with the default group_size of 8, every step performs 1,024 video games. The job strikes by means of Beginning, Downloading, and Pending earlier than reaching Coaching. After it’s coaching, your AgentCore runtime begins receiving rollout invocations.

Necessary: Don’t redeploy the agent whereas a job is operating. Redeploying restarts the runtime, in-flight rollouts see no sampling requests, and the service fails your complete job.

Step 6: Monitor coaching in MLflow

The service logs reward, token, and switch statistics at each step and runs validation each val_every steps. The metric to look at is val/reward/pass_at_1, the fraction of held-out phrases solved. Each remedy scores at the very least 1.01, so it clears the service’s success_threshold of 1.0 and cross@1 counts it. The next chart is constructed from this run’s MLflow historical past:

Determine 4: Coaching curves for the 100-step run, with imply reward per sport on the left and validation remedy charge on the precise

Coaching reward climbs to about +0.57 inside 15 steps and holds there. Validation reward stays decrease, largely due to temperature 1.0 versus 0 slightly than overfitting (see Step 7). The validation remedy charge jumps from 2 to 31 % within the first 10 steps, then strikes between 35 and 48 %. With 128 phrases, these swings are largely noise. We didn’t check whether or not extra steps assist at serving settings.

The runtime’s OpenTelemetry traces in CloudWatch file each mannequin flip and gear name. Right here is one sport the skilled mannequin performed throughout an analysis job (medium effort, temperature 1.0), with the reasoning trimmed. The key phrase was PASTE:

[model] Reasoning: [...] "CRANE" has 5 distinct letters: C,R,A,N,E. [...]
We'll guess "CRANE".
Device name: guess_word(guess="CRANE")
[tool] Guess 1/6: CRANE -> C=grey, R=grey, A=yellow, N=grey, E=inexperienced
Present Data:
- Right Place (Inexperienced): _ _ _ _ E
- In Phrase, Incorrect Place (Yellow): 'A' (at the very least 1)
- Not in Phrase (Grey): C, N, R
- Guesses Remaining: 5
[model] Reasoning: [...] "ALIVE": ends with E, consists of A. Letters not C,N,R.
Device name: guess_word(guess="ALIVE")
[tool] Guess 2/6: ALIVE -> A=yellow, L=grey, I=grey, V=grey, E=inexperienced
[model] Reasoning: [...] So positions not 1 or 3 for A. Positions left: 2,4
for A. [...] We would guess "HASTE"? [...]
Device name: guess_word(guess="HASTE")
[tool] Guess 3/6: HASTE -> H=grey, A=inexperienced, S=inexperienced, T=inexperienced, E=inexperienced
[model] Reasoning: We've the phrase: _ASTE. [...] "PASTE" is a phrase. PASTE
suits. Additionally "WASTE" additionally suits [...] Select "PASTE" perhaps.
Device name: guess_word(guess="PASTE")
[tool] Guess 4/6: PASTE --- all inexperienced. Right, you solved it!

A trained-model rollout from the AgentCore runtime traces. The mannequin retains inexperienced letters in place, by no means reuses a grey letter, and makes use of every yellow letter to rule out positions, although nothing within the coaching knowledge confirmed it how.

Step 7: Consider the skilled mannequin at serving settings

When the job completes, its web page in SageMaker AI Studio reveals the standing and the mannequin package deal group that holds the skilled mannequin. DescribeJob, with JobCategory set to AgentRFT, returns the output mannequin package deal ARN and the billable token counts.

Determine 5: The finished coaching job in SageMaker AI Studio on the Jobs, Coaching, Multi-turn RL tab

An analysis job (JobCategory set to AgentRFTEvaluation) performs your analysis prompts by means of the identical AgentCore runtime and reward, on the bottom mannequin or the skilled package deal, and logs eval/reward/pass_at_1 to MLflow. In contrast to coaching validation, you possibly can set the temperature. The repository’s launcher runs evaluations again to again, as a result of the default quota permits one analysis job at a time (you possibly can request a rise):

uv run python run_mtrl_eval.py --runs base:1.0 skilled:1.0 
  --model-package-arn  
  --agent-runtime-arn  
  --role-arn arn:aws:iam:::position/ 
  --dataset s3:///wordle-mtrl/validation/validation-data.jsonl 
  --s3-output-path s3:///wordle-mtrl/eval/ 
  --mlflow-app-arn arn:aws:sagemaker:::mlflow-app/

Determine 6: Reasoning effort and sampling temperature at every stage, the place the coaching job fixes validation at temperature 0 and the deployed effort whereas analysis jobs allow you to select

Every 128-word analysis took 4 to seven minutes. Reasoning effort comes from the agent’s surroundings variables, so to vary it you edit REASONING_EFFORT, redeploy, and consider. We evaluated each fashions at low and medium effort and at temperature 0 and 1.0:

Determine 7: Base mannequin versus skilled mannequin below 4 decoding settings, with remedy charge on the left and imply reward per sport on the precise

The grey phase of every bar is the bottom mannequin and the blue phase is the achieve from coaching. The low-effort, temperature-0 pair is the coaching job’s validation at steps 0 and 100. The medium-effort, temperature-1.0 skilled result’s the imply of two runs (80 and 88 %).

At temperature 0, each fashions usually finish the sport with out a remedy, at both effort. With sampling on, medium effort provides 15 factors for the skilled mannequin and 46 for the bottom mannequin. The adapter provides 14–54 factors in each setting, so what coaching taught transfers throughout settings.

The coaching job’s personal validation curve in MLflow advised a distinct story: it peaked at 48 % at step 90 and ended at 37 %, scored at temperature 0. The hyperparameters reference recommends temperature 0 for analysis as a result of grasping decoding makes curves deterministic and comparable throughout runs. We report temperature 1.0 as a result of that’s how we serve the mannequin. At medium effort and temperature 1.0, the skilled mannequin solves 84 % of held-out phrases, up from 61 % for the bottom mannequin.

Tip: Consider on the settings you’ll serve at. The coaching curve says this adapter ends at 37 %. At serving settings it solves 84 %. The analysis jobs that present this price about $1.20 in tokens and took about an hour.

Step 8: Clear up

Coaching and analysis jobs cease billing after they full, however the AgentCore runtime, its IAM position, the MLflow app, and the S3 objects persist. From the challenge listing:

# 1. Agent runtime and its execution position. take away all resets the challenge config;
# the subsequent deploy sees the empty state and tears down the deployed sources.
cd sample-multi-turn-rl-wordle/Wordle
agentcore take away all
agentcore deploy

# 2. MLflow app
aws sagemaker delete-mlflow-app --arn arn:aws:sagemaker:::mlflow-app/

# 3. Datasets, job output, and analysis output
aws s3 rm s3:///wordle-mtrl/ --recursive

As a result of agentcore take away all rewrites the recordsdata below agentcore/, run git checkout -- agentcore/ if you wish to redeploy later. Deleting the AgentCore-Wordle-default CloudFormation stack removes the identical sources and leaves the recordsdata alone. The coaching job position and the mannequin packages usually are not billed.

Deep dive: Designing the reward

Our reward wanted one revision, and the error is a standard one.

The primary design summed the per-guess scoring from the wordle-lora-rl challenge over the entire sport. That scoring works there as a result of every guess is an impartial coaching pattern. Summed into one reward per sport, it produced two inversions we discovered solely by studying traces:

• A misplaced sport paid constructive reward. Six turns of bonuses added as much as about +0.49.
• Fixing on flip 1 scored under fixing on flip 3. A primary-guess remedy earned +0.99 and a third-guess remedy earned +1.61, as a result of the additional turns collected extra bonuses.

The repair separates a dominant final result time period from a small, bounded shaping time period:

reward = final result + w * clip(shaping, -1, 1), w = 0.04 if solved, else 0.15

The result time period is about fully by how the sport ended:

The shaping time period is that per-guess scoring, summed over the sport:

• Penalties for contradicting a identified inexperienced, yellow, or grey letter, for repeating a guess, and for phrases exterior the dictionary.
• An entropy bonus for a powerful opening phrase.
• Bonuses for making an attempt new letters and for shrinking the set of remaining prospects.

The sum is clipped to [-1, 1] and scaled by 0.04 for a remedy and 0.15 for a loss, so it separates video games with the identical final result however by no means strikes a sport into one other tier. With typical shaping values:

A quicker remedy now all the time beats a slower one, each remedy beats each loss, and by no means taking part in is the worst final result. Our first repair used a single weight of 0.15, which let a guess-3 remedy with full shaping (+1.45) outscore a guess-2 remedy with the minimal (+1.25). Adjoining guess counts are 0.1 aside, so a smaller remedy weight of 0.04 removes the overlap, and the worst remedy (+1.01) nonetheless clears the 1.0 success threshold. Shaping nonetheless issues as a result of GRPO learns from reward variance inside a bunch, so two misplaced video games that differ in clue consistency nonetheless educate the mannequin one thing. Because the SageMaker AI documentation advises, begin with an outcome-based reward, add shaping sparingly, and examine the reward on actual trajectories.

Deep dive: What went unsuitable, within the order you’ll hit it

Every of those price at the very least one coaching run.

• Bearer tokens below concurrency. Minting a token with a recent botocore session per name starved the credential supplier at 32 concurrent rollouts, and caching tokens for an hour returned 403 errors as credentials rotated. The repair is one self-refreshing session, a brief cache, and a re-sign on 403.
• A rollout that by no means reaches the coverage mannequin fails the entire job. When each mannequin name in a rollout failed authentication, the service ended the job with “No sampling requests had been acquired.” Returning an error standing fails it too, so all the time return a reward. A sport the place the mannequin is sampled however by no means makes a legitimate guess is okay: it scores -1.5, and the 100-step run had 51 of them in about 105,000 rollouts.
• Device calls drift into plain textual content. At too excessive a studying charge, the mannequin wrote device calls as textual content, each rollout bought the identical flooring reward, and the gradient vanished. The agent now recovers guesses from uncooked textual content with a small formatting penalty.
• The plateau. Coaching reward stopped rising after about step 15 and validation stopped enhancing after about step 30, although at temperature 0 validation understates the mannequin (Step 7). A curriculum that seeds video games with prior guesses, or extra rollouts per phrase than the default group_size of 8, is what we’d attempt subsequent.

Conclusion

MTRL on SageMaker AI trains a mannequin on the end result of a complete episode. That matches agentic duties, the place the precise trajectory is unknown and solely the consequence might be scored. We used a single episode-level reward. When intermediate steps might be scored too, MTRL additionally accepts one reward per flip by means of the UpdateReward API. On this submit, an open-weight 20B mannequin reached an 84 % remedy charge at serving settings, up from 61 %, for about $339 in coaching tokens and about $79 in AgentCore runtime utilization. The 37 % the coaching curve experiences is an artifact of how the coaching job scores validation, so consider on the settings you’ll serve at.

The identical pipeline applies to inner procedures, instruments, and workflows that public fashions had been by no means skilled on.

To get began, clone the pattern repository, comply with the previous steps, and run the smoke check in Step 3 earlier than committing GPU time. For the underlying APIs, see the SageMaker AI MTRL documentation.

Concerning the authors