Massive language fashions (LLMs) carry out nicely on basic duties however battle with specialised work that requires understanding proprietary information, inner processes, and industry-specific terminology. Supervised fine-tuning (SFT) adapts LLMs to those organizational contexts. SFT may be applied by way of two distinct methodologies: Parameter-Environment friendly Effective-Tuning (PEFT), which updates solely a subset of mannequin parameters, providing sooner coaching and decrease computational prices whereas sustaining cheap efficiency enhancements; Full-rank SFT, which updates all mannequin parameters quite than a subset and incorporates extra area data than PEFT.
Full-rank SFT usually faces a problem: catastrophic forgetting. As fashions be taught domain-specific patterns, they lose basic capabilities together with instruction-following, reasoning, and broad data. Organizations should select between area experience and basic intelligence, which limits mannequin utility throughout enterprise use instances.
Amazon Nova Forge addresses the issue. Nova Forge is a brand new service that you should utilize to construct your individual frontier fashions utilizing Nova. Nova Forge clients can begin their improvement from early mannequin checkpoints, mix proprietary information with Amazon Nova-curated coaching information, and host their customized fashions securely on AWS.
On this put up, we share outcomes from the AWS China Utilized Science workforce’s complete analysis of Nova Forge utilizing a difficult Voice of Buyer (VOC) classification activity, benchmarked in opposition to open-source fashions. Working with over 16,000 buyer remark samples throughout a fancy four-level label hierarchy containing 1,420 leaf classes, we reveal how Nova Forge’s information mixing strategy supplies two benefits:
- In-domain activity efficiency beneficial properties: reaching 17% F1 rating enhancements
- Preserved basic capabilities: sustaining near-baseline MMLU (Huge Multitask Language Understanding) scores and instruction-following talents post-finetuning
The problem: real-world buyer suggestions classification
Take into account a typical situation at a big ecommerce firm. The client expertise workforce receives hundreds of buyer feedback each day with detailed suggestions spanning product high quality, supply experiences, fee points, web site usability, and customer support interactions. To function effectively, they want an LLM that may robotically classify every remark into actionable classes with excessive precision. Every classification have to be particular sufficient to route the problem to the suitable workforce: logistics, finance, improvement, or customer support, and set off the suitable workflow. This requires area specialization.
Nonetheless, this similar LLM doesn’t function in isolation. Throughout your group, groups want the mannequin to:
- Generate customer-facing responses that require basic communication expertise
- Carry out information evaluation requiring mathematical and logical reasoning
- Draft documentation following particular formatting pointers
This requires broad basic capabilities—instruction-following, reasoning, data throughout domains, and conversational fluency.
Analysis methodology
Check overview
To check whether or not Nova Forge can ship each area specialization and basic capabilities, we designed a dual-evaluation framework measuring efficiency throughout two dimensions.
For domain-specific efficiency, we use a real-world Voice of Buyer (VOC) dataset derived from precise buyer critiques. The dataset accommodates 14,511 coaching samples and 861 take a look at samples, reflecting production-scale enterprise information. The dataset employs a four-level taxonomy the place Degree 4 represents the leaf classes (ultimate classification targets). Every class features a descriptive rationalization of its scope. Instance classes:
| Degree 1 | Degree 2 | Degree 3 | Degree 4 (leaf class) |
| Set up – app configuration | Preliminary setup steering | Setup course of | Simple setup expertise: Set up course of traits and complexity degree |
| Utilization – {hardware} expertise | Night time imaginative and prescient efficiency | Low-light Picture high quality | Night time imaginative and prescient readability: Night time imaginative and prescient mode produces photographs in low-light or darkish circumstances |
| Utilization – {hardware} expertise | Pan-tilt-zoom performance | Rotation functionality | 360-degree rotation: The digital camera can rotate a full 360 levels, offering full panoramic protection |
| After-sales coverage and value | Return and change coverage | Return course of execution | Product return accomplished: Buyer initiated and accomplished product return on account of performance points |
The dataset displays excessive class imbalance typical of real-world buyer suggestions environments. The next picture shows the category distribution:
Consequently, the dataset locations a big problem on classification accuracy.
For evaluating general-purpose capabilities, we use the general public take a look at set break up of the MMLU (Huge Multitask Language Understanding) benchmark (all subsets). The take a look at spans topics within the humanities, social sciences, laborious sciences, and different areas which might be vital for some individuals to be taught. On this put up, MMLU serves as a proxy for basic functionality retention. We use it to measure whether or not supervised fine-tuning improves area efficiency at the price of degrading foundational mannequin behaviors, and to evaluate the effectiveness of Nova information mixing in mitigating catastrophic forgetting.
| Merchandise | Description |
| Complete samples | 15,372 buyer critiques |
| Label hierarchy | 4-level classification, 1,420 classes in whole |
| Coaching set | 14,511 samples |
| Check set | 861 samples |
| MMLU Benchmark all (take a look at break up) | 14,000 samples |
In-domain activity analysis: voice of buyer classification
To know how Nova Forge performs in actual enterprise situations, we first consider mannequin accuracy on the VOC classification activity earlier than and after supervised fine-tuning. With this strategy, we will quantify area adaptation beneficial properties whereas establishing a baseline for subsequent robustness evaluation.
Base mannequin analysis
We start with a base mannequin analysis to evaluate out-of-the-box efficiency on the VOC classification activity with none task-specific fine-tuning. This setup establishes every mannequin’s inherent functionality to deal with extremely granular classification underneath strict output format constraints. The next immediate is used for the VOC classification activity:
# Position Definition
You're a rigorous buyer expertise classification system. Your sole accountability is to map person suggestions to the prevailing label taxonomy at Degree 1 by way of Degree 4 (L1–L4). You will need to strictly comply with the predefined taxonomy construction and should not create, modify, or infer any new labels.
## Working Rules
### 1. Strict taxonomy alignment
All classifications have to be totally grounded within the offered label taxonomy and strictly adhere to its hierarchical construction.
### 2. Suggestions decomposition utilizing MECE ideas
A single piece of person suggestions might include one or a number of points. You will need to rigorously analyze all points described and decompose the suggestions into a number of non-overlapping segments, following the MECE (Mutually Unique, Collectively Exhaustive) precept:
- **Semantic singularity**: Every section describes just one difficulty, perform, service, or touchpoint (for instance, pricing, efficiency, or UI).
- **Independence**: Segments should not overlap in that means.
- **Full protection**: All info within the authentic suggestions have to be preserved with out omission.
### 3. No taxonomy growth
You will need to not invent, infer, or modify any labels or taxonomy ranges.
## Label Taxonomy
The next part supplies the label taxonomy: {tag class}. Use this taxonomy to carry out L1–L4 classification for the unique VOC suggestions. No taxonomy growth is allowed.
## Activity Directions
You can be given a chunk of person suggestions: {person remark}. Customers might come from totally different areas and use totally different languages. You will need to precisely perceive the person's language and intent earlier than assigning labels.
Consult with the offered examples for the anticipated labeling format.
## Output Format
Return the classification ends in JSON format solely. For every suggestions section, output the unique textual content together with the corresponding L1–L4 labels and sentiment. Don't generate or rewrite content material.
```json
[
{
"content": "
"L1": "
"L2": "
"L3": "
"L4": "
"emotion": "
}
]
```
For base mannequin analysis, we chosen:
| Mannequin | Precision | Recall | F1-Rating |
| Nova 2 Lite | 0.4596 | 0.3627 | 0.387 |
| Qwen3-30B-A3B | 0.4567 | 0.3864 | 0.394 |
The F1-scores reveal that Nova 2 Lite and Qwen3-30B-A3B reveal comparable efficiency on this domain-specific activity, with each fashions reaching F1-scores close to 0.39. These outcomes additionally spotlight the inherent issue of the duty: even sturdy basis fashions battle with fine-grained label classification when no domain-specific information is offered.
Supervised fine-tuning
We then apply full-parameter supervised fine-tuning (SFT) utilizing buyer VOC information. All fashions are fine-tuned utilizing the identical dataset and comparable coaching configurations for a good comparability.
Coaching infrastructure:
In area activity efficiency comparability
| Mannequin | Coaching Knowledge | Precision | Recall | F1-Rating |
| Nova 2 Lite | None (baseline) | 0.4596 | 0.3627 | 0.387 |
| Nova 2 Lite | Buyer information solely | 0.6048 | 0.5266 | 0.5537 |
| Qwen3-30B | Buyer information solely | 0.5933 | 0.5333 | 0.5552 |
After fine-tuning on buyer information alone, Nova 2 Lite achieves a considerable efficiency enchancment, with F1 growing from 0.387 to 0.5537—an absolute acquire of 17 factors. This end result locations the Nova mannequin within the high tier for this activity and makes its efficiency corresponding to that of the fine-tuned Qwen3-30B open-source mannequin. These outcomes verify the effectiveness of Nova full-parameter SFT for complicated enterprise classification workloads.
Normal capabilities analysis: MMLU benchmark
Fashions fine-tuned for VOC classification are sometimes deployed past a single activity and built-in into broader enterprise workflows. Preserving general-purpose capabilities is vital. Business-standard benchmarks comparable to MMLU present an efficient mechanism for evaluating general-purpose capabilities and detecting catastrophic forgetting in fine-tuned fashions.
For the fine-tuned Nova mannequin, Amazon SageMaker HyperPod presents out-of-the-box analysis recipes that streamline MMLU analysis with minimal configuration.
| Mannequin | Coaching information | VOC F1-Rating | MMLU accuracy |
| Nova 2 Lite | None (baseline) | 0.38 | 0.75 |
| Nova 2 Lite | Buyer information solely | 0.55 | 0.47 |
| Nova 2 Lite | 75% buyer + 25% Nova information | 0.5 | 0.74 |
| Qwen3-30B | Buyer information solely | 0.55 | 0.0038 |
When Nova 2 Lite is fine-tuned utilizing buyer information solely, we observe a important drop in MMLU accuracy from 0.75 to 0.47, indicating the lack of general-purpose capabilities. The degradation is much more pronounced for the Qwen mannequin, which largely loses instruction-following capability after fine-tuning. An instance of Qwen mannequin degraded output:
{
"prediction": "[n {n "content": "x^5 + 3x^3 + x^2 + 2x in Z_5",n "A": "0",n "B": "1",n "C": "0,1",n "D": "0,4",n "emotion": "neutral"n }n]"
}This habits can also be associated to the VOC immediate design, the place class data is internalized by way of supervised fine-tuning—a standard strategy in large-scale classification programs.
Notably, when Nova information mixing is utilized throughout fine-tuning, Nova 2 Lite retains near-baseline basic efficiency. MMLU accuracy stays at 0.74, solely 0.01 under the unique baseline, whereas VOC F1 nonetheless improves by 12 factors (0.38 → 0.50). This validates that Nova information mixing is a sensible and efficient mechanism for mitigating catastrophic forgetting whereas preserving area efficiency.
Key findings and sensible suggestions
This analysis exhibits that when the bottom mannequin supplies a robust basis, full-parameter supervised fine-tuning on Amazon Nova Forge can ship substantial beneficial properties for complicated enterprise classification duties. On the similar time, the outcomes verify that catastrophic forgetting is an actual concern in manufacturing fine-tuning workflows. Effective-tuning on buyer information alone can degrade general-purpose capabilities comparable to instruction following and reasoning, limiting a mannequin’s usability throughout broader enterprise situations.
The information mixing functionality of Nova Forge supplies an efficient mitigation technique. By mixing buyer information with Nova-curated datasets throughout fine-tuning, groups can protect near-baseline basic capabilities whereas persevering with to realize sturdy domain-specific efficiency.
Primarily based on these findings, we suggest the next practices when utilizing Nova Forge:
- Use supervised fine-tuning to maximise in-domain efficiency for complicated or extremely custom-made duties.
- Apply Nova information mixing when fashions are anticipated to assist a number of general-purpose workflows in manufacturing, to cut back the chance of catastrophic forgetting.
Collectively, these practices assist steadiness mannequin customization with manufacturing robustness, enabling extra dependable deployment of fine-tuned fashions in enterprise environments.
Conclusion
On this put up, we demonstrated how organizations can construct specialised AI fashions with out sacrificing basic intelligence with Nova Forge information mixing capabilities. Relying in your use instances and enterprise targets, Nova Forge can ship different advantages, together with entry checkpoints throughout all phases of mannequin improvement and performing reinforcement studying with reward features in your atmosphere. To get began along with your experiments, see the Nova Forge Developer Information for detailed documentation.
Concerning the authors
Yuan Wei is an Utilized Scientist at Amazon Internet Companies, working with enterprise clients on proof-of-concepts and technical advisory. She makes a speciality of giant language fashions and vision-language fashions, with a deal with evaluating rising methods underneath real-world information, value, and system constraints.
Xin Hao is a Senior AI/ML Go-to-Market Specialist at AWS, serving to clients obtain success with Amazon Nova fashions and associated Generative AI options. He has intensive hands-on expertise in cloud computing, AI/ML, and Generative AI. Previous to becoming a member of AWS, Xin spent over 10 years within the industrial manufacturing sector, together with industrial automation and CNC machining.
Sharon Li is an AI/ML Specialist Options Architect at Amazon Internet Companies (AWS) primarily based in Boston, Massachusetts. With a ardour for leveraging cutting-edge know-how, Sharon is on the forefront of creating and deploying progressive generative AI options on the AWS cloud platform.
