Construct environment friendly brokers with data distillation on Amazon SageMaker AI

You constructed a procuring assistant on a frontier mannequin, and it really works. A consumer asks for “a black waist bag underneath twenty {dollars} from a store with at the very least 4.5 stars,” and the agent searches the catalog, inspects candidates, checks the constraints, and commits to a suggestion. However what does that interplay truly price? An agentic assistant shouldn’t be one mannequin name, it’s a loop: search, learn outcomes, refine, confirm, resolve. A single accomplished process may be a number of calls to that frontier mannequin, and also you pay for each one, on each shopper. At manufacturing scale, that per-task price compounds, erodes the mission’s ROI, and stops the mission from delivery.

One strategy to make an agentic software extra environment friendly is to make use of a smaller mannequin, which is quicker to reply and cheaper to host. The catch is high quality: a smaller mannequin normally falls quick on the duty, which is why the bigger one was chosen within the first place. Data distillation closes that hole by transferring the habits of the bigger “trainer” mannequin right into a smaller “scholar”, coaching the coed to breed the trainer’s outputs, so it retains a lot of the standard on the smaller mannequin’s pace and value.

On this submit you’ll learn to use data distillation, and particularly sequence-level data distillation (SeqKD), to fine-tune a procuring agent on Amazon SageMaker AI. We cowl the right way to generate high-quality multi-step trajectories with a frontier trainer and maintain those that remedy the duty, the right way to encode them for supervised fine-tuning, the right way to run the coaching on serverless infrastructure, and the right way to consider the coed towards a held-out benchmark.

Determine 1: One agentic process is a loop, not a single name: the agent searches, reads outcomes, refines, verifies, and decides, and every step is a separate billed name to the frontier mannequin. A single accomplished process may be a number of calls, on each shopper

What’s sequence-level data distillation (SeqKD)?

The best relevant type of distillation is sequence-level data distillation (SeqKD). It makes use of a trainer mannequin to generate high-quality responses, then trains a scholar mannequin on these responses with supervised fine-tuning (SFT). SeqKD wants solely the trainer’s textual content, with no entry to trainer logits and no reinforcement-learning infrastructure, so it really works even with a closed or API-only trainer. In comparison with different approaches akin to logit-based or on-policy, it sits on the most cost-effective and moveable finish of the distillation spectrum. The trainer writes the solutions, and the coed is fine-tuned on them. As a result of these solutions persist as a dataset, you possibly can reuse them to coach or check completely different college students with out regenerating the information.

In precept, you’d need the coed to match the trainer’s full distribution over attainable outputs. The caveat right here is you can’t. The area of attainable sequences is astronomically giant, so matching it precisely isn’t possible. SeqKD sidesteps that by approximating it with samples the trainer truly produces.

For an agent, the sequence the trainer produces shouldn’t be a block of prose however a trajectory, an ordered run of reasoning, software calls, and the observations they return, the place every step is conditioned on every part earlier than it. Coaching the coed on that complete sequence teaches it the reasoning that results in every motion and the multi-step plan of action itself, not a single paragraph of reply textual content. That makes an agentic process one of many cleanest illustrations of sequence-level KD, as a result of the sequence being copied is the habits itself.

Determine 2: SeqKD transfers habits via the trainer’s textual content outputs alone: the trainer generates trajectories, the coed trains on them, and the outcome retains a lot of the trainer’s high quality on the small mannequin’s pace and value. No trainer weights, logits, or gradients are required throughout coaching, so it runs as extraordinary supervised fine-tuning

When SeqKD is the proper selection

SeqKD suits a selected set of conditions:

You have got the inputs however no high-quality goal outputs

That is the core case for SeqKD, which is basically knowledge technology. You provide the inputs, and the trainer writes the targets, so you possibly can fine-tune on a process or area that has no present high-quality labels. The targets may be so simple as single solutions or as concerned as multi-turn tool-use trajectories. This submit takes the demanding finish of that spectrum, agentic trajectories, that are particularly arduous to supply. A single process branches into many partially right runs, and telling a solved run from a failed one takes a dependable grader. A benchmark with a programmatic scorer provides precisely that, so you possibly can let the trainer generate trajectories and maintain solely those that remedy the duty.

Your trainer is a frontier mannequin you possibly can solely attain via an API

That is the traditional case for frontier fashions. You’ll be able to’t entry the weights to host it your self, you possibly can’t learn its logits, and you may’t fine-tune it. That guidelines out logit-based distillation and on-policy strategies, which all want that inner entry. SeqKD wants solely the trainer’s textual content responses, so it’s the one method that also works, and it frees you from paying frontier costs on each request after the coed takes over.

You desire a robust, cost-effective first step

A few of the strongest small reasoning fashions have been inbuilt important half with SeqKD (supervised fine-tuning on trainer reasoning traces), so it’s an efficient place to begin for reasoning switch when the setup is correct. The DeepSeek-R1 distilled fashions are a widely known instance: the reasoning of a big trainer was distilled into smaller Qwen and Llama college students by fine-tuning them on the trainer’s generated reasoning traces.

Conditions

To run this instance, you want:

Answer overview

This part walks the pipeline from finish to finish. A frontier trainer, GLM-5.3 on Amazon Bedrock, acts because the ShoppingBench agent to generate trajectories. The benchmark’s personal scorer retains solely those that remedy the duty, and people solved trajectories fine-tune a Qwen3.5-4B scholar with serverless supervised fine-tuning on SageMaker AI mannequin customization. The distilled scholar is then evaluated on a held-out ShoppingBench break up and served behind a SageMaker AI Inference real-time endpoint that runs the software loop.

Determine 3: Distilling a procuring agent on Amazon SageMaker AI: a frontier trainer on Amazon Bedrock acts because the ShoppingBench agent on the coaching issues, ShoppingBench’s personal scorer retains solely the trajectories that remedy the duty (rejection sampling), every solved trajectory is encoded per step, the coed is fine-tuned serverlessly with Amazon SageMaker AI mannequin customization, and the result’s evaluated on the held-out ShoppingBench check break up

Getting ready the dataset

Sourcing trajectories price imitating is the arduous half, and the ShoppingBench scorer is what separates the profitable runs from the failed ones. ShoppingBench publishes 750 synthesized procuring issues, 250 every for the product, store, and voucher intents, every with a hidden gold reply and the constraints that outline success. We break up them as soon as into 426 coaching issues and 324 held-out analysis issues (product 103, store 105, voucher 116), as fastened indices so the 2 units don’t overlap.

For every coaching drawback, a frontier trainer, GLM-5.3 on Amazon Bedrock with reasoning on and temperature 1.0, acts because the ShoppingBench agent. It searches the catalog of about 2.7 million merchandise, inspects candidates, and recommends those that meet the constraints. We pattern the trainer twice per drawback, which supplies 850 trajectories, then let ShoppingBench’s personal scorer maintain solely those that remedy the duty. Temperature 1.0 is deliberate. Sampling slightly than grasping decoding makes the 2 makes an attempt discover completely different queries and reasoning paths, which supplies extra issues at the very least one solved trajectory and a extra various set to be taught from. This rejection-sampling step turns a capable-but-imperfect trainer right into a clear coaching set, because the trainer solely must be proper typically sufficient that the solved runs are price imitating.

Every solved trajectory is an entire episode: the patron request, the trainer’s reasoning and gear calls at every step (search, examine, suggest, terminate), and the statement the setting returned. We maintain the reasoning alongside the actions, which the Outcomes present is what carries the coed a lot of the strategy to its trainer.

Encoding trajectories for coaching

Sequence-level distillation is supervised studying over trajectories that exist already, so nothing calls a software throughout coaching, and the applying solely ever sees textual content. As a result of these trajectories are multi-step tool-use conversations, we encode every as a messages listing of chat turns and let the coed’s personal chat template render it, so the tokens the coach sees match the tokens the agent produces at inference.

Every trajectory turns into one coaching row per step. The person flip carries the system immediate and the dialog to date. The assistant flip carries that step’s reasoning and the software name it results in, a … block adopted by a [{…}] within the harness’s native type. Loss falls on the assistant flip solely, so the mannequin learns to provide the subsequent reasoning-and-action, to not predict the observations the setting returned. A trajectory with a number of steps expands into a number of rows. Right here the solved trajectories increase to about 2,600 coaching and 130 validation rows after filtering.

A single row pairs the dialog to date (person) with the one subsequent step to be taught (assistant):

{
  "messages": [
    {"role": "user", "content": "You are a shopping assistant. ...n# TasknFind a black waist bag under $20 from a shop rated 4.5 or higher.n# Observations so farn(none yet)"},
    {"role": "assistant", "content": "Start with a broad search and let the price filter work before I narrow by shop rating.n[{"name": "find_product", "parameters": {"q": "black waist bag", "price": "0-20"}}]"}
  ]
}

Every later step turns into its personal row, with the brand new statement appended to the person content material and the subsequent reasoning-and-tool-call because the assistant goal, via to the ultimate recommend_product and terminate calls. Render each row along with your mannequin’s precise chat template.

Cut up the information by drawback, not by row. The trainer is sampled twice per drawback, and every trajectory expands into a number of steps, so many rows share one drawback, and a row-level break up would leak near-identical context throughout coaching and validation. Holding out complete issues (about 5 p.c) retains the validation set sincere.

The price of per-step encoding is repeated context, as a result of every row restates the rising dialog, and serverless fine-tuning truncates something previous dataset_max_len, so a row that crosses the restrict loses a part of what it wants. Moderately than let the restrict resolve what will get minimize, maintain rows inside price range. The simplest repair is to compress software observations, which dominate the token rely. For product search outcomes, maintain solely the fields the subsequent resolution depends upon (id, worth, ranking, and the constraints being checked), drop the uncooked itemizing textual content, and cap what number of outcomes you keep per search. If trajectories are nonetheless over price range, cap the steps per trajectory. Right here dataset_max_len is 16,384 tokens, which inserts most trajectories after observations are trimmed.

After the JSONL for every break up is in S3, register every as a customization dataset asset with DataSet.create from the SageMaker Python SDK v3, which returns the train_dataset and val_dataset the coaching job consumes.

Tremendous-tuning the coed

Coaching makes use of SageMaker AI mannequin customization with serverless SFT. You choose a base mannequin from the mannequin hub, level the job on the encoded datasets, and the managed service provisions the compute, runs the job, and returns the personalized mannequin, with metrics tracked in Amazon SageMaker with MLflow. As a result of the information is already in a supported format, there isn’t any coaching script, container picture, or occasion to handle.

The job writes its output right into a mannequin package deal group that variations each mannequin it produces, so create the group first with ModelPackageGroup.create (reusing it if the identify already exists) and go it to the coach. Then configure the customization job. Select the bottom mannequin and coaching kind, level it on the practice and validation datasets plus the mannequin package deal group, then identify an MLflow experiment so metrics stream to MLflow whereas the job runs.

from sagemaker.practice.sft_trainer import SFTTrainer
from sagemaker.practice.frequent import TrainingType

sft_trainer = SFTTrainer(
    mannequin="huggingface-vlm-qwen3-5-4b",  # Qwen3.5-4B from the JumpStart mannequin hub
    training_type=TrainingType.LORA,  # TrainingType.FULL for full fine-tuning
    training_dataset=train_dataset.arn,
    validation_dataset=val_dataset.arn,
    model_package_group=model_package_group,
    mlflow_experiment_name="seqkd-shopping-exp",
    s3_output_path="s3:///seqkd/output/",
    accept_eula=True,
)

The coach begins from a documented set of default hyperparameters you can examine with sft_trainer.hyperparameters.to_dict() and override earlier than launching. For this run we elevate the sequence size to suit the multi-step trajectories, use a Low-Rank Adaptation (LoRA) rank of 64, and practice for 3 epochs over the small dataset:

sft_trainer.hyperparameters.dataset_max_len = 16384
sft_trainer.hyperparameters.max_epochs = 3
sft_trainer.hyperparameters.global_batch_size = 64
sft_trainer.hyperparameters.learning_rate = 0.0001
sft_trainer.hyperparameters.lora_rank = 64
sft_trainer.hyperparameters.lora_alpha = 256  # alpha / rank = 4
sft_trainer.hyperparameters.lr_warmup_steps_ratio = 0.1
sft_trainer.hyperparameters.train_val_split_ratio = 0.95

Validate the configuration with a dry run, then launch the managed job and skim the registered Mannequin Package deal ARN when it finishes. That ARN is the deploy-ready artifact the analysis and serving steps devour:

from sagemaker.core.sources import TrainingJob

sft_trainer.practice(dry_run=True)  # checks the config with out launching
training_job = sft_trainer.practice(wait=True)  # launch and wait
model_package_arn = TrainingJob.get(
    training_job_name=training_job.training_job_name
).output_model_package_arn

Evaluating the coed

Consider in two levels, lowest price first. The complete end-to-end analysis is each time and useful resource intensive. It stands up a search server over tens of millions of merchandise, deploys the endpoint, and runs a multi-step agent loop on each held-out drawback. Earlier than operating the complete analysis, run a low-cost managed pre-check utilizing SageMaker AI mannequin analysis, which scores one generated step at a time towards the trainer’s ground-truth motion.

The pre-check wants a scoring operate. Mannequin analysis helps a Customized Scorer reward operate you provide that runs as soon as per pattern, evaluating the coed’s generated motion towards the trainer’s. For tool-call distillation, grade software choice (did the mannequin name the proper instruments, so as, ignoring argument formatting), as a result of that’s the habits distillation most must switch. To maintain the give attention to choice slightly than syntax, the reward operate reads the software names out of each the coed’s motion and the trainer’s reference with out requiring an actual format match.

def reward_function(pattern, index):
    pred = pattern.get("model_response", "")
    ref = pattern.get("reference_answer", "") or pattern.get("response", "")

    ref_names = tool_names(ref)  # software names from the trainer's [...]
    pred_names = tool_names(pred)  # software names the coed emitted, parsed leniently
    toolset = 1.0 if (pred_names and pred_names == ref_names) else 0.0

    return {
        "id": str(pattern.get("id", f"sample-{index:03d}")),
        "aggregate_reward_score": toolset,  # the gate metric: software choice
        "metrics_list": [
            {"name": "toolset_match", "value": toolset, "type": "Reward"},
        ],
    }

Register that operate as a versioned Evaluator with Evaluator.create(), then name consider() to run one Customized Scorer job per process, scoring the bottom mannequin alongside the coed so each outcome has a comparability.

The choice rule is simple. The scholar should beat the bottom mannequin on this low-cost pre-check. If it does, the distillation took and it’s price operating the extra complete end-to-end analysis. If it doesn’t, repair coaching first.

Serving the distilled agent

The distilled mannequin doesn’t run instruments by itself. Tremendous-tuning teaches it to emit the proper software name on the proper second as textual content, however one thing nonetheless has to execute that decision and feed again the outcome. At inference you wrap the mannequin in a small deterministic harness that prompts the mannequin, parses the software name it emits, runs the software, appends the statement, and prompts once more, looping till the mannequin produces a ultimate reply. That is extraordinary software code, or an off-the-shelf serving framework, not reinforcement studying and never an on-policy coaching loop.

The scholar is a Low-Rank Adaptation (LoRA) adapter over an open-weight base, which shapes the way you deploy it. Moderately than merge the adapter and serve a second full copy of the weights, serve the bottom mannequin as soon as and fasten the coed as an adapter on prime, as two inference parts on one real-time endpoint. Naming the inference part at invoke time selects which one solutions, so the identical endpoint serves each the bottom and the distilled scholar for a side-by-side comparability.

We serve the open-weight mannequin with the Deep Java Library Giant Mannequin Inference (DJL LMI) container, which bundles the vLLM engine for high-throughput technology, pinned to a selected model slightly than newest, so redeploys are reproducible. The accompanying pocket book wraps the inference-component calls in a small kd_serve helper, so deploying the bottom part with LoRA serving enabled after which attaching the coed adapter reduces to:

from sagemaker.core import image_uris
import kd_serve

# DJL LMI container (vLLM engine bundled in); pin the model for reproducible redeploys
image_uri = image_uris.retrieve("djl-lmi", area=area, model="0.36.0")

# 1. Deploy the bottom mannequin as an inference part, with LoRA serving enabled.
endpoint_name, base_ic = kd_serve.deploy_base_endpoint(
    endpoint_name="seqkd-shopping-agent",
    position=position,
    area=area,
    image_uri=image_uri,
    base_model_id="Qwen/Qwen3.5-4B",
    instance_types=["ml.g5.2xlarge", "ml.g6.2xlarge", "ml.g6e.2xlarge"],
    max_lora_rank=64,
    max_loras=4,
    max_model_len=131072,
)

# 2. Resolve the coed's adapter artifacts from its Mannequin Package deal and fasten them
# as a second inference part on the identical base.
adapter_uri = kd_serve.resolve_adapter_uri(model_package_arn, area)
student_ic = kd_serve.attach_adapter(
    endpoint_name=endpoint_name,
    base_ic_name=base_ic,
    adapter_component_name=f"{endpoint_name}-student",
    adapter_s3_uri=adapter_uri,
    area=area,
)

The instance_types listing is an ordered desire. SageMaker AI Inference deploys on the primary kind with accessible capability and falls again down the listing, so the coed serves on an ml.g5.2xlarge, ml.g6.2xlarge, or ml.g6e.2xlarge relying on availability. The bottom weights load as soon as, and each inference parts share them, so including the adapter prices adapter-sized reminiscence as a substitute of a second copy of the mannequin.

Outcomes

We consider the distilled scholar on the held-out ShoppingBench check break up, towards a set of reference factors that bracket what beauty like:

• Base Qwen3.5-4B: the off-the-shelf scholar with no fine-tuning. That is the ground.
• Distilled scholar (SeqKD): the identical 4B mannequin fine-tuned on the trainer trajectories, the method this submit describes. That is the outcome that issues.
• GLM-5.3 trainer: the frontier mannequin that generated the coaching knowledge, on Bedrock. We report it each sampled (temperature 1.0, the setting that produced the information) and grasping (temperature 0).
• Medium-sized frontier mannequin: a powerful proprietary mannequin run as an agent, included as a sensible ceiling.

Each mannequin runs the identical unmodified ShoppingBench agent loop over the three shopper intents (product, store, and voucher). We report Absolute Success Charge (ASR), the share of held-out issues the agent solves, averaged over the three intents, with lacking trajectories counted as failures. An issue counts as solved when the beneficial product or merchandise meet each acknowledged constraint. The next numbers come from a single run with a LoRA scholar, so anticipate just a few factors of run-to-run variation.

Process success

The headline result’s the Absolute Success Charge on the 324 held-out issues, damaged down by intent and averaged throughout the three:

Determine 4: Absolute Success Charge (ASR) on the held-out ShoppingBench check break up, by shopper intent. The distilled 4B scholar (orange) improves on the bottom mannequin by 24 factors on common, reaches the GLM-5.3 trainer that generated its knowledge and a medium-sized frontier mannequin

In abstract, a 4B mannequin recovered basically all of its frontier trainer’s process high quality, matching it inside run-to-run noise. We report the trainer two methods as a result of they reply completely different questions. Grasping decoding returns the only probably trajectory, the stronger selection when every mannequin will get one try per drawback (as on this analysis) which is why the grasping trainer scores 0.540 towards the sampled trainer’s 0.511. We generated the coaching knowledge the opposite means, sampling at temperature 1.0 so repeated makes an attempt explored completely different paths, and the scorer might maintain those that solved the duty. That filtering can be why the coed can edge out the sampled trainer it realized from, since rejection sampling trains it solely on the trainer’s solved runs, not its common output, so it imitates a best-of slice slightly than the complete unfold of makes an attempt.

Reasoning within the targets issues

The scholar learns from the trainer’s reasoning in addition to its software calls, and that selection is liable for a big a part of the outcome. Coaching the identical scholar on the identical trajectories with the reasoning stripped, leaving solely the naked software calls, drops common ASR from 0.541 to 0.415:

Determine 5: Absolute Success Charge for the bottom mannequin, the coed skilled on software calls solely, and the coed skilled on the trainer’s reasoning plus software calls. Preserving the reasoning provides about 13 factors of common ASR, and the achieve holds on each intent

Each college students clear the bottom mannequin, so even the tool-calls-only model is an actual enchancment, however preserving the reasoning is what carries the coed the remainder of the way in which to its trainer. The reasoning between the actions isn’t narration to discard, it’s a part of the habits that solves the duty.

Preserving normal capacity

ASR measures the ability we distilled for. The counterweight is whether or not the coed stored its normal instruction-following capacity or overfit to the procuring process. We verify with IFEval, a normal instruction-following benchmark that scores verifiable format directions akin to “reply in precisely three bullet factors.” Deal with it as a regression verify, not a leaderboard. The scholar’s prompt-strict rating falls from 0.819 for the bottom mannequin to 0.741, and its instruction-strict rating from 0.872 to 0.813, a drop of roughly 6 to eight factors (Determine 6). That’s the anticipated price of specialization. The scholar traded some normal instruction-following for a big achieve on the procuring process. A a lot bigger drop can be the warning signal that distillation narrowed the mannequin too far.

Determine 6: IFEval normal instruction-following for the bottom mannequin and the distilled scholar, on the prompt-strict and instruction-strict metrics. The small drop is the anticipated price of specializing on the procuring process

Advantages and tradeoffs

Successful fee by itself doesn’t justify the change. The rationale to distill is that you just get a lot of the stronger mannequin’s high quality on the small mannequin’s price and pace. The clearest proof right here is that the 4B scholar reaches the duty high quality of its frontier trainer and a medium-sized frontier mannequin, whereas being sufficiently small to serve on a single G household occasion (NVIDIA A10G/L40/L40S on this instance).

The fee distinction is as a lot about form as measurement. A frontier mannequin reached via an API is billed per token, on each step of each process, so the fee grows with utilization and doesn’t cease, and the agentic loop from the introduction pays that toll each time. The distilled scholar runs on an endpoint you host, which payments for the occasion by the hour whether or not it serves one process or hundreds, so added visitors prices nothing extra till you scale out. You’re buying and selling a variable per-request cost for a set hourly one, and the smaller the mannequin you may get away with, the smaller that fastened price is.

We don’t publish a per-task price or latency quantity right here as a result of it depends upon your visitors: the throughput a G household occasion sustains, and subsequently the fee amortized per process, is a operate of your request sample, sequence lengths, and concurrency. Measure it by yourself workload towards the present SageMaker AI pricing and your frontier supplier’s token pricing, slightly than counting on a single headline determine.

Limitations and greatest practices

Will the duty match the coed (the capability wall)

Distillation transfers a distribution the coed can already characterize, not uncooked functionality it lacks. Throughout too giant a spot, the coed copies the trainer’s format with out studying to resolve. A small mannequin skilled on a big trainer’s chains of thought produces lengthy, well-structured reasoning whose solutions keep mistaken, and an agent skilled the identical means emits well-formed software calls that make no progress, re-searches with reworded queries, and runs to its step restrict with out assembly the constraints. The symptom is similar in each, traces lengthen whereas accuracy stays flat. If you see it, cease including knowledge and both develop the coed or decrease process problem.

The hole that decides that is between the duty and the coed, not between the coed and the trainer. A small scholar can recuperate a a lot bigger trainer’s high quality so long as the duty itself is inside its attain. That’s the reason the procuring agent right here works. The trainer is a mixture-of-experts mannequin with roughly 40 billion energetic parameters, about ten occasions the coed’s 4 billion, however the procuring process is effectively inside a 4B mannequin’s capability, so the coed recovers the trainer’s high quality as a substitute of solely lengthening its trajectories.

The place SeqKD stops serving to

SeqKD lifts straightforward and mid-difficulty issues however plateaus on the toughest ones, the place the coed should recuperate from its personal errors. That is sharper for an agent than a single-step process. In a multi-step loop an early error compounds, as a result of the coed doesn’t see the states its personal errors result in. Closing that hole is what on-policy strategies goal, which is why SeqKD is normally the primary stage slightly than the end line. SeqKD additionally asks the least of your infrastructure, because it makes use of solely the trainer’s textual content, whereas logit distillation wants the trainer’s per-token distribution and on-policy strategies rating the coed’s personal rollouts. A text-only SeqKD heat begin adopted by an on-policy stage is a typical recipe, so this submit covers the primary stage and pushing previous the plateau is the pure subsequent step.

Two decisions that carry the outcome

First, match the coaching targets to the habits you need. Right here that meant preserving the trainer’s reasoning with every software name slightly than compressing to ultimate solutions, because the reasoning is a part of what solves the duty (the Outcomes quantify the hole), and rendering each goal with the chat template the agent makes use of at serving time. Second, consider truthfully. Measure on knowledge the fashions haven’t seen, since leaked benchmarks inflate scores, and examine towards greater than the untrained base mannequin the coed is certain to beat, setting it beside the trainer and an impartial frontier mannequin so the quantity exhibits how shut the coed got here to what it was distilled from.

Cleansing up

An actual-time inference endpoint payments for so long as it runs, so tear down the endpoint that you just stood up for serving after you will have your outcomes. As a result of the coed is served as an adapter inference part on prime of a base inference part, the items should come down in dependency order: the adapter part first, then the bottom part, then the endpoint and its configuration. The accompanying kd_serve helper does this for you, discovering the adapter structurally so it cleans up whatever the precise part names:

import kd_serve

# Deletes the adapter IC, then the bottom IC, then the endpoint and its configuration
kd_serve.teardown_endpoint(endpoint_name, area=area)

The coaching job itself wants no cleanup. Serverless coaching provisions compute, runs, and releases it robotically, so a accomplished job isn’t an ongoing cost. The registered mannequin package deal and its group carry no compute price both, so you possibly can maintain them for lineage and redeploy later. The one factor that does maintain costing you is your knowledge in S3 (the encoded trajectories and output artifacts), which incurs storage price for so long as you retain it, so delete what you now not want or transfer it to a lower-cost storage class.

Conclusion

On this submit, we confirmed the right way to distill an agentic procuring assistant right into a smaller, lower-cost scholar with sequence-level data distillation (SeqKD). We generated high-quality single-turn, multi-step trajectories with a frontier trainer on Bedrock and stored solely those that solved the duty utilizing ShoppingBench’s personal scorer, encoded every step as chat-formatted messages that protect the trainer’s reasoning alongside its software calls, fine-tuned the coed on serverless infrastructure with SageMaker AI mannequin customization, tracked coaching with MLflow, ran a top quality gate verify with SageMaker AI mannequin analysis, and served the outcome behind an inference endpoint that runs the software loop. The payoff was a 4B scholar that recovers basically all of its frontier trainer’s process high quality. As a result of SeqKD wants solely the trainer’s textual content outputs, it really works even when the trainer is a closed frontier mannequin you attain via an API, and you may cease paying frontier costs on each step after the coed takes over.

SeqKD is a powerful first step with actual limits. Match the coed measurement to the duty, consider on a held-out suite towards robust reference factors slightly than solely the untrained base mannequin, and when the toughest tail of issues resists imitation, comply with SeqKD with on-policy strategies not coated right here that allow the coed be taught from its personal rollouts.

To do this your self, see the accompanying code repository and the SageMaker AI documentation.

In regards to the authors