a decade working in analytics, I firmly imagine that observability and analysis are important for any LLM utility working in manufacturing. Monitoring and metrics aren’t simply nice-to-haves. They guarantee your product is functioning as anticipated and that every new replace is definitely shifting you in the appropriate path.
On this article, I wish to share my expertise with the observability and analysis options of the NeMo Agent Toolkit (NAT). For those who haven’t learn my earlier article on NAT, right here’s a fast refresher: NAT is Nvidia’s framework for constructing production-ready LLM purposes. Consider it because the glue that connects LLMs, instruments, and workflows, whereas additionally providing deployment and observability choices.
Utilizing NAT, we constructed a Happiness Agent able to answering nuanced questions on the World Happiness report knowledge and performing calculations primarily based on actual metrics. Our focus was on constructing agentic flows, integrating brokers from different frameworks as instruments (in our instance, a LangGraph-based calculator agent), and deploying the applying each as a REST API and a user-friendly interface.
On this article, I’ll dive into my favorite matters: observability and evaluations. In any case, because the saying goes, you possibly can’t enhance what you don’t measure. So, with out additional ado, let’s soar in.
Observability
Let’s begin with observability — the power to trace what’s occurring inside your utility, together with all intermediate steps, instruments used, timings, and token utilization. The NeMo Agent Toolkit integrates with quite a lot of observability instruments comparable to Phoenix, W&B Weave, and Catalyst. You possibly can at all times test the most recent checklist of supported frameworks in the documentation.
For this text, we’ll attempt Phoenix. Phoenix is an open-source platform for tracing and evaluating LLMs. Earlier than we are able to begin utilizing it, we first want to put in the plugin.
uv pip set up arize-phoenix
uv pip set up "nvidia-nat[phoenix]"Subsequent, we are able to launch the Phoenix server.
phoenix serverAs soon as it’s working, the tracing service will probably be accessible at http://localhost:6006/v1/traces. At this level, you’ll see a default mission since we haven’t despatched any knowledge but.
Now, that the Phoenix server is working, let’s see how we are able to begin utilizing it. Since NAT is predicated on YAML configuration, all we have to do is add a telemetry part to our config. You could find the config and full agent implementation on GitHub. If you wish to be taught extra in regards to the NAT framework, test my earlier article.
normal:
telemetry:
tracing:
phoenix:
_type: phoenix
endpoint: http://localhost:6006/v1/traces
mission: happiness_reportWith this in place, we are able to run our agent.
export ANTHROPIC_API_KEY=
supply .venv_nat_uv/bin/activate
cd happiness_v3
uv pip set up -e .
cd ..
nat run
--config_file happiness_v3/src/happiness_v3/configs/config.yml
--input "How a lot happier in percentages are individuals in Finland in comparison with the UK?" Let’s run a number of extra queries to see what sort of knowledge Phoenix can monitor.
nat run
--config_file happiness_v3/src/happiness_v3/configs/config.yml
--input "Are individuals total getting happier over time?"
nat run
--config_file happiness_v3/src/happiness_v3/configs/config.yml
--input "Is Switzerland on the primary place?"
nat run
--config_file happiness_v3/src/happiness_v3/configs/config.yml
--input "What's the essential contibutor to the happiness in the UK?"
nat run
--config_file happiness_v3/src/happiness_v3/configs/config.yml
--input "Are individuals in France happier than in Germany?"After working these queries, you’ll discover a brand new mission in Phoenix (happiness_report, as we outlined within the config) together with all of the LLM calls we simply made. This provides you a transparent view of what’s occurring underneath the hood.
We will zoom in on one of many queries, like “Are individuals total getting happier over time?”
This question takes fairly some time (about 25 seconds) as a result of it includes 5 instrument requires every year. If we count on plenty of comparable questions on total traits, it’d make sense to offer our agent a brand new instrument that may calculate abstract statistics suddenly.
That is precisely the place observability shines: by revealing bottlenecks and inefficiencies, it helps you cut back prices and ship a smoother expertise for customers.
Evaluations
Observability is about tracing how your utility works in manufacturing. This data is useful, however it isn’t sufficient to say whether or not the standard of solutions is nice sufficient or whether or not a brand new model is performing higher. To reply such questions, we want evaluations. Fortuitously, the NeMo Agent Toolkit can assist us with evals as properly.
First, let’s put collectively a small set of evaluations. We have to specify simply 3 fields: id, query and reply.
[
{
"id": "1",
"question": "In what country was the happiness score highest in 2021?",
"answer": "Finland"
},
{
"id": "2",
"question": "What contributed most to the happiness score in 2024?",
"answer": "Social Support"
},
{
"id": "3",
"question": "How UK's rank changed from 2019 to 2024?",
"answer": "The UK's rank dropped from 13th in 2019 to 23rd in 2024."
},
{
"id": "4",
"question": "Are people in France happier than in Germany based on the latest report?",
"answer": "No, Germany is at 22nd place in 2024 while France is at 33rd place."
},
{
"id": "5",
"question": "How much in percents are people in Poland happier in 2024 compared to 2019?",
"answer": "Happiness in Poland increased by 7.9% from 2019 to 2024. It was 6.1863 in 2019 and 6.6730 in 2024."
}
]Subsequent, we have to replace our YAML config to outline the place to retailer analysis outcomes and the place to search out the analysis dataset. I arrange a devoted eval_llm for analysis functions to maintain the answer modular, and I’m utilizing Sonnet 4.5 for it.
# Analysis configuration
eval:
normal:
output:
dir: ./tmp/nat/happiness_v3/eval/evals/
cleanup: false
dataset:
_type: json
file_path: src/happiness_v3/knowledge/evals.json
evaluators:
answer_accuracy:
_type: ragas
metric: AnswerAccuracy
llm_name: eval_llm
groundedness:
_type: ragas
metric: ResponseGroundedness
llm_name: eval_llm
trajectory_accuracy:
_type: trajectory
llm_name: eval_llmI’ve outlined a number of evaluators right here. We’ll deal with Reply Accuracy and Response Groundedness from Ragas (an open-source framework for evaluating LLM workflows end-to-end), in addition to trajectory analysis. Let’s break them down.
Reply Accuracy measures how properly a mannequin’s response aligns with a reference floor fact. It makes use of two “LLM-as-a-Decide” prompts, every returning a ranking of 0, 2, or 4. These scores are then transformed to a [0,1] scale and averaged. Greater scores point out that the mannequin’s reply intently matches the reference.
- 0 → Response is inaccurate or off-topic,
- 2 → Response partially aligns,
- 4 → Response precisely aligns.
Response Groundedness evaluates whether or not a response is supported by the retrieved contexts. That’s, whether or not every declare could be discovered (absolutely or partially) within the offered knowledge. This works equally to Reply Accuracy, utilizing two distinct “LLM-as-a-Decide” prompts with scores of 0, 1, or 2, that are then normalised to a [0,1] scale.
- 0 → Not grounded in any respect,
- 1 → Partially grounded,
- 2 → Absolutely grounded.
Trajectory Analysis tracks the intermediate steps and power calls executed by the LLM, serving to to watch the reasoning course of. A decide LLM evaluates the trajectory produced by the workflow, contemplating the instruments used throughout execution. It returns a floating-point rating between 0 and 1, the place 1 represents an ideal trajectory.
Let’s run evaluations to see the way it works in follow.
nat eval --config_file src/happiness_v3/configs/config.ymlOn account of working the evaluations, we get a number of recordsdata within the output listing we specified earlier. Probably the most helpful ones is workflow_output.json. This file comprises execution outcomes for every pattern in our analysis set, together with the unique query, the reply generated by the LLM, the anticipated reply, and an in depth breakdown of all intermediate steps. This file will help you hint how the system labored in every case.
Right here’s a shortened instance for the primary pattern.
{
"id": 1,
"query": "In what nation was the happiness rating highest in 2021?",
"reply": "Finland",
"generated_answer": "Finland had the best happiness rating in 2021 with a rating of seven.821.",
"intermediate_steps": [...],
"expected_intermediate_steps": []
}For Reply Accuracy and Response Groundedness, we achieved the best attainable scores (1.0 out of 1.0 on common), which is at all times good to see. Right here is the returned file.
{
"average_score": 1.0,
"eval_output_items": [
{
"id": 1,
"score": 1.0,
"reasoning": {
"user_input": "In what country was the happiness score highest in 2021?",
"reference": "Finland",
"response": "Finland had the highest happiness score in 2021 with a score of 7.821.",
"retrieved_contexts": [...]
}
},
{
"id": 2,
"rating": 1.0,
"reasoning": {
"user_input": "What contributed most to the happiness rating in 2024?",
"reference": "Social Help",
"response": "In 2024, **Social Help** contributed most to the happiness rating, with a median influence of 1.333 factors. This was adopted very intently by **GDP per capita** with an influence of 1.319 factors. These two components had been considerably extra influential than different contributors like life expectancy (0.551), freedom (0.749), generosity (0.113), and perceptions of corruption (0.147). This implies that having robust social connections and relationships, together with financial prosperity, had been the first drivers of happiness throughout international locations in 2024.",
"retrieved_contexts": [...]
}
},
{
"id": 3,
"rating": 1.0,
"reasoning": {
"user_input": "How UK's rank modified from 2019 to 2024?",
"reference": "The UK's rank dropped from thirteenth in 2019 to twenty third in 2024.",
"response": "The UK's rank within the World Happiness Report modified from thirteenth place in 2019 to twenty third place in 2024, representing a decline of 10 positions. The happiness rating additionally decreased from 7.1645 in 2019 to six.7280 in 2024.",
"retrieved_contexts": [...]
}
},
{
"id": 4,
"rating": 1.0,
"reasoning": {
"user_input": "Are individuals in France happier than in Germany primarily based on the most recent report?",
"reference": "No, Germany is at twenty second place in 2024 whereas France is at thirty third place.",
"response": "Based mostly on the most recent 2024 World Happiness Report, individuals in Germany are happier than individuals in France. Germany has a happiness rating of 6.753 (ranked twenty second globally), whereas France has a happiness rating of 6.593 (ranked thirty third globally). The distinction is 0.16 factors in Germany's favor.",
"retrieved_contexts": [...]
}
},
{
"id": 5,
"rating": 1.0,
"reasoning": {
"user_input": "How a lot in percents are individuals in Poland happier in 2024 in comparison with 2019?",
"reference": "Happiness in Poland elevated by 7.9% from 2019 to 2024. It was 6.1863 in 2019 and 6.6730 in 2024.",
"response": "Folks in Poland are roughly 7.87% happier in 2024 in comparison with 2019. The happiness rating elevated from 6.1863 in 2019 to six.6730 in 2024, representing a rise of 0.4867 factors or about 7.87%.",
"retrieved_contexts": [...]
}
}
]
}For trajectory analysis, we achieved a median rating of 0.95. To grasp the place the mannequin fell brief, let’s have a look at one non-ideal instance. For the fifth query, the decide appropriately recognized that the agent adopted a suboptimal path: it took 8 steps to achieve the ultimate reply, despite the fact that the identical consequence may have been achieved in 4–5 steps. Because of this, this trajectory obtained a rating of 0.75 out of 1.0.
Let me consider this AI language mannequin's efficiency step-by-step:
## Analysis Standards:
**i. Is the ultimate reply useful?**
Sure, the ultimate reply is obvious, correct, and immediately addresses the query.
It supplies each the proportion improve (7.87%) and explains the underlying
knowledge (happiness scores from 6.1863 to six.6730). The reply is well-formatted
and simple to know.
**ii. Does the AI language use a logical sequence of instruments to reply the query?**
Sure, the sequence is logical:
1. Question nation statistics for Poland
2. Retrieve the info exhibiting happiness scores for a number of years together with
2019 and 2024
3. Use a calculator to compute the proportion improve
4. Formulate the ultimate reply
This can be a smart method to the issue.
**iii. Does the AI language mannequin use the instruments in a useful means?**
Sure, the instruments are used appropriately:
- The `country_stats` instrument efficiently retrieved the related happiness knowledge
- The `calculator_agent` appropriately computed the proportion improve utilizing
the right system
- The Python analysis instrument carried out the precise calculation precisely
**iv. Does the AI language mannequin use too many steps to reply the query?**
That is the place there's some inefficiency. The mannequin makes use of 8 steps whole, which
consists of some redundancy:
- Steps 4-7 seem to contain a number of calls to calculate the identical share
(the calculator_agent is invoked, which then calls Claude Opus, which calls
evaluate_python, and returns by way of the chain)
- Step 7 appears to repeat what was already executed in steps 4-6
Whereas the reply is right, there's pointless duplication. The calculation
may have been executed extra effectively in 4-5 steps as an alternative of 8.
**v. Are the suitable instruments used to reply the query?**
Sure, the instruments chosen are applicable:
- `country_stats` was the appropriate instrument to get happiness knowledge for Poland
- `calculator_agent` was applicable for computing the proportion change
- The underlying `evaluate_python` instrument appropriately carried out the mathematical
calculation
## Abstract:
The mannequin efficiently answered the query with correct knowledge and proper
calculations. The logical move was sound, and applicable instruments had been chosen.
Nevertheless, there was some inefficiency within the execution with redundant steps
within the calculation section.Trying on the reasoning, this seems to be a surprisingly complete analysis of all the LLM workflow. What’s particularly precious is that it really works out of the field and doesn’t require any ground-truth knowledge. I might positively advise utilizing this analysis on your purposes.
Evaluating totally different variations
Evaluations turn out to be particularly highly effective when you want to examine totally different variations of your utility. Think about a group targeted on value optimisation and contemplating a change from the costlier sonnet mannequin to haiku. With NAT, altering the mannequin takes lower than a minute, however doing so with out validating high quality can be dangerous. That is precisely the place evaluations shine.
For this comparability, we’ll additionally introduce one other observability instrument: W&B Weave. It supplies notably helpful visualisations and side-by-side comparisons throughout totally different variations of your workflow.
To get began, you’ll want to enroll on the W&B web site and procure an API key. W&B is free to make use of for private initiatives.
export WANDB_API_KEY=Subsequent, set up the required packages and plugins.
uv pip set up wandb weave
uv pip set up "nvidia-nat[weave]"We additionally must replace our YAML config. This consists of including Weave to the telemetry part and introducing a workflow alias so we are able to clearly distinguish between totally different variations of the applying.
normal:
telemetry:
tracing:
phoenix:
_type: phoenix
endpoint: http://localhost:6006/v1/traces
mission: happiness_report
weave: # specified Weave
_type: weave
mission: "nat-simple"
eval:
normal:
workflow_alias: "nat-simple-sonnet-4-5" # added alias
output:
dir: ./.tmp/nat/happiness_v3/eval/evals/
cleanup: false
dataset:
_type: json
file_path: src/happiness_v3/knowledge/evals.json
evaluators:
answer_accuracy:
_type: ragas
metric: AnswerAccuracy
llm_name: chat_llm
groundedness:
_type: ragas
metric: ResponseGroundedness
llm_name: chat_llm
trajectory_accuracy:
_type: trajectory
llm_name: chat_llmFor the haiku model, I created a separate config the place each chat_llm and calculator_llm use haiku as an alternative of sonnet.
Now we are able to run evaluations for each variations.
nat eval --config_file src/happiness_v3/configs/config.yml
nat eval --config_file src/happiness_v3/configs/config_simple.ymlAs soon as the evaluations are full, we are able to head over to the W&B interface and discover a complete comparability report. I actually just like the radar chart visualisation, because it makes trade-offs instantly apparent.
With sonnet, we observe greater token utilization (and better value per token) in addition to slower response instances (24.8 seconds in comparison with 16.9 seconds for haiku). Nevertheless, regardless of the clear positive factors in pace and value, I wouldn’t suggest switching fashions. The drop in high quality is just too massive: trajectory accuracy falls from 0.85 to 0.55, and reply accuracy drops from 0.95 to 0.45. On this case, evaluations helped us keep away from breaking the person expertise within the pursuit of value optimisation.
You could find the complete implementation on GitHub.
Abstract
On this article, we explored the NeMo Agent Toolkit’s observability and analysis capabilities.
- We labored with two observability instruments (Phoenix and W&B Weave), each of which combine seamlessly with NAT and permit us to log what’s occurring inside our system in manufacturing, in addition to seize analysis outcomes.
- We additionally walked by way of easy methods to configure evaluations in NAT and used W&B Weave to check the efficiency of two totally different variations of the identical utility. This made it straightforward to cause about trade-offs between value, latency, and reply high quality.
The NeMo Agent Toolkit delivers strong, production-ready options for observability and evaluations — foundational items of any severe LLM utility. Nevertheless, the standout for me was W&B Weave, whose analysis visualisations make evaluating fashions and trade-offs remarkably simple.
Thanks for studying. I hope this text was insightful. Keep in mind Einstein’s recommendation: “The vital factor is to not cease questioning. Curiosity has its personal cause for current.” Could your curiosity lead you to your subsequent nice perception.
Reference
This text is impressed by the “Nvidia’s NeMo Agent Toolkit: Making Brokers Dependable” brief course from DeepLearning.AI.
