New Relic Inc. is a San Francisco-based know-how firm that pioneered utility efficiency monitoring (APM) and gives complete observability options. Serving main clients worldwide, together with main manufacturers like Ryanair, New Relic helps organizations monitor and optimize their digital programs to ship higher buyer experiences.
New Relic confronted a problem widespread to many quickly rising enterprises. Their engineers had been spending priceless time looking by fragmented documentation throughout a number of programs, with time consuming inner system queries, in some circumstances, taking greater than a day. As a number one observability platform supporting hundreds of shoppers worldwide, New Relic knew a extra environment friendly method to entry and make the most of organizational data was wanted.
This problem led to the creation of New Relic NOVA (New Relic Omnipresence Digital Assistant): an revolutionary synthetic intelligence (AI) device constructed on Amazon Net Providers (AWS). New Relic NOVA has reworked how New Relic staff entry and work together with firm data and programs.
Working with the Generative AI Innovation Middle, New Relic NOVA developed from a data assistant right into a complete productiveness engine. New Relic NOVA is constructed on AWS companies together with Amazon Bedrock, Amazon Kendra, Amazon Easy Storage Service (Amazon S3), and Amazon DynamoDB. By Strands Brokers, New Relic NOVA gives clever code opinions, AI governance, and managed Mannequin Context Protocol (MCP) companies.
Amazon Bedrock is a completely managed service that gives entry to main basis fashions for constructing generative AI purposes, eliminating the necessity to handle infrastructure whereas enabling groups to customise fashions for his or her particular use circumstances. By a single API, builders can experiment with and consider totally different basis fashions, combine them with enterprise programs, and construct safe AI purposes at scale.
The answer has decreased info search time whereas automating complicated operational workflows. By collaboration with the Generative AI Innovation Middle, New Relic NOVA was developed into an answer that now processes over 1,000 each day queries throughout their group. New Relic NOVA integrates seamlessly with Confluence, GitHub, Salesforce, Slack, and varied inner programs, sustaining 80% accuracy in its responses for each knowledge-based queries and transactional duties.
We are going to present how New Relic NOVA is architected utilizing AWS companies to create a scalable, clever assistant that goes past doc retrieval to deal with complicated duties like automated staff permission requests and fee restrict administration. We discover the technical structure, growth journey, and key classes realized in constructing an enterprise-grade AI answer that delivers measurable productiveness positive aspects at scale.
Answer overview
In designing New Relic NOVA, New Relic established a number of essential goals past the preliminary purpose of bettering documentation search. These included sustaining information safety throughout data retrieval and attaining constant response high quality throughout totally different information sources. As proven in Determine 1, New Relic NOVA’s AWS structure permits seamless interplay between customers and varied AWS companies whereas sustaining safety and scalability. The answer required a versatile framework that would evolve with the group’s wants for each data retrieval and transactional duties. A key problem was balancing these necessities whereas protecting response occasions below 20 seconds to keep up person engagement.
Determine 1 – Answer structure of New Relic NOVA framework
The event staff recognized a number of potential dangers early within the venture. These included the opportunity of exposing delicate info by AI responses, sustaining accuracy when retrieving from a number of information sources, and making certain system reliability at enterprise scale. Determine 2 illustrates New Relic NOVA’s detailed agent workflow, demonstrating how queries are processed and routed by varied specialised brokers to handle person intentions. Moreover, the staff carried out complete safety controls which included personable identifiable info (PII) detection and masking, together with a strong analysis framework to observe and preserve response high quality.
Determine 2 – New Relic NOVA agent workflow structure
The venture additionally revealed alternatives for future optimization. These embody increasing an agent hierarchy structure to help extra automated workflows and creating extra refined analytics for monitoring person interplay patterns. The staff’s expertise means that organizations enterprise related tasks ought to give attention to establishing clear analysis metrics early and constructing versatile architectures that may accommodate evolving enterprise wants.
Answer
New Relic NOVA was developed over an eight-week interval, involving a collaborative effort between inner engineering, safety, authorized, and compliance groups and the AWS Generative AI Innovation Middle. This partnership accelerated speedy growth and iteration, leveraging AWS experience in large-scale AI implementations.
Agent structure
The New Relic NOVA structure consists of three key layers:
- Principal agent layer – This acts as a controllable orchestration for executing totally different workflows by figuring out the person intent and delegating efforts to the next downstream layers:
- Retrieval Augmented Era (RAG) with custom-made ingested data from Amazon Bedrock Data Bases or Amazon Kendra.
- Brokers for direct interplay with third-party platforms.
- Custom-made brokers for dealing with inner New Relic duties.
- Fallback dealing with if customers’ responses can’t be decided.
- Knowledge supply layers (vector DB, enrich, information sources) – These layers symbolize sources the place inner data (for instance, New Relic requirements documentation and code repository documentation) are ingested for retrieval or RAG functions. The good thing about these customized sources is to reinforce info and search efficiency to be used info requests.
- Brokers layer – Contains two distinct agent varieties:
- Strands Brokers with MCP: Deal with multi-step processes for third-party companies, leveraging MCP for standardized service interactions.
- Customized motion brokers: Execute New Relic-specific duties resembling permission requests and repair restrict modifications, offering exact management over inner programs.
A central agent acts as an orchestrator, routing queries to specialised sub-agents in a delegation mannequin the place responses movement immediately again to the person relatively than requiring inter-agent reasoning or changes. In the meantime, Strands Brokers are used to effectively handle third-party service integrations utilizing MCP. This strategy provides New Relic NOVA the perfect of each worlds: the orchestration mannequin maintains flexibility for inner processes whereas standardizing exterior companies by MCP, making a scalable basis for New Relic concerning future automation wants.
Knowledge integration technique
The ability lies within the skill of New Relic NOVA to seamlessly combine a number of information sources, offering a unified interface for data retrieval. This strategy contains:
- Amazon Bedrock Data Bases for Confluence: Confirms direct synchronization with Confluence areas and maintains up-to-date info.
- Amazon Kendra for GitHub Enterprise: Indexes and searches GitHub repositories, offering fast entry to code documentation.
- Strands Brokers for Salesforce and Jira: Customized brokers execute SOQL and JQL queries, respectively, to fetch related information from their respective platforms (Salesforce and Jira).
- Amazon Q Index for Slack: Makes use of Amazon Q Index capabilities to implement a RAG answer for Slack channel historical past, chosen for its speedy growth potential.
A singular facet of the information integration of New Relic NOVA is the customized doc enrichment course of. Throughout ingestion, paperwork are enhanced with metadata, key phrases, and summaries, considerably bettering retrieval relevance and accuracy.
Utilizing Amazon Nova fashions
Amazon Nova is AWS’s new technology of basis fashions designed to ship frontier intelligence with industry-leading worth efficiency for enterprise use circumstances. The Amazon Nova household of fashions can course of numerous inputs together with textual content, photos, and video, excelling in duties from interactive chat to doc evaluation, whereas supporting superior capabilities like RAG programs and AI agent workflows.
To optimize efficiency and cost-efficiency, New Relic NOVA makes use of Amazon Nova Lite and Professional fashions by Amazon Bedrock. These fashions had been rigorously chosen to steadiness response high quality with latency, enabling New Relic NOVA to keep up sub-20 second response occasions whereas processing complicated queries. Amazon Bedrock gives entry to numerous basis mannequin households. Its standardized framework and immediate optimization helps seamless switching between fashions with out code modifications. This permits New Relic NOVA to optimize for velocity with Amazon Nova Lite or, due to complexity, change to Amazon Nova Professional whereas sustaining constant efficiency and price effectivity.
Superior RAG implementation
New Relic NOVA employs a classy RAG strategy, using Amazon Bedrock Data Bases, Amazon Kendra, and Amazon Q Index. To maximise retrieval accuracy, New Relic NOVA implements a number of key optimization methods:
- Hierarchical chunking: Amazon Bedrock Data Bases employs hierarchical chunking, a way confirmed simplest by in depth experimentation with varied chunking methodologies.
- Context enrichment: A customized AWS Lambda perform enhances chunks throughout data base ingestion, incorporating related key phrases and contextual info. This course of is especially priceless for code-related content material, the place structural and semantic cues considerably affect retrieval efficiency.
- Metadata integration: Throughout data base doc ingestion, extra context, resembling summaries, titles, authors, creation dates, and final modified dates, is appended as doc metadata. This enriched metadata enhances the standard and relevance of retrieved info.
- Customized doc processing: For particular information sources like GitHub repositories, tailor-made doc processing methods are utilized to protect code construction and enhance search relevance.
These methods work in live performance to optimize the RAG system inside New Relic NOVA, delivering extremely correct retrieval throughout diversified doc varieties whereas minimizing growth effort by current connectors. The mixture of hierarchical chunking, context enrichment, metadata integration, and customized doc processing permits New Relic NOVA to offer exact, context-aware responses whatever the information supply or doc format.
Analysis framework
New Relic NOVA implements a complete analysis framework, leveraging Amazon Bedrock basis fashions for its giant language mannequin (LLM)-as-a-judge strategy, together with validation datasets that mix questions, floor fact solutions, and supply doc URLs. This analysis framework, which might be executed on-demand in growth environments, encompasses three essential metrics for system validation:
- Reply accuracy measurement makes use of a 1–5 discrete scale score system, the place the LLM evaluates the generated response’s factual alignment with the established floor fact information.
- Context relevance evaluation on a scale of 1–5, analyzing the retrieved context’s relevance to the person question.
- Response latency monitoring measures workflow efficiency, from preliminary question enter to last reply technology, making certain optimum person expertise by complete timing evaluation.
This triple-metric analysis strategy helps detailed efficiency optimization throughout the New Relic NOVA answer core functionalities.
Observability and steady enhancements
The answer features a complete observability framework that collects metrics and analyzes person suggestions. The metric and suggestions assortment is carried out by New Relic AI monitoring options. Suggestions is carried out by the Slack response function (emoji responses), customers can shortly present suggestions on New Relic NOVA responses. These reactions are captured by a New Relic python agent and despatched to a https://one.newrelic.com/ area. The suggestions assortment system gives priceless insights for:
- Measuring person satisfaction with responses.
- Figuring out areas the place accuracy might be improved.
- Understanding utilization patterns throughout totally different groups.
- Monitoring the effectiveness of several types of queries.
- Monitoring the efficiency of assorted information sources.
- Tracing every LLM name and latency.
The collected suggestions information might be analyzed utilizing AWS analytics companies resembling AWS Glue for ETL processing, Amazon Athena for querying, and Amazon QuickSight for visualization. This data-driven strategy permits steady enchancment of New Relic NOVA and helps prioritize future enhancements based mostly on precise person interactions.
Inner groups are already experiencing the benefits of New Relic NOVA. Determine 3 showcases a number of the responses captured by the Slack suggestions course of.
Determine 3 – Customers Slack message exchanges about New Relic NOVA expertise
Concerns and subsequent steps
The success of New Relic NOVA highlights a number of key learnings for organizations seeking to implement related options:
- Begin with a transparent understanding of person ache factors and measurable success standards.
- Implement strong information integration methods with customized doc enrichment.
- Use the generative AI companies and basis fashions that finest suit your use circumstances to attain optimum outcomes.
- Construct in suggestions mechanisms from the begin to allow steady enchancment.
- Give attention to each velocity and accuracy to make sure person adoption.
When it comes to subsequent steps, New Relic NOVA is evolving from a standalone answer right into a complete enterprise AI platform by integrating cutting-edge AWS applied sciences and open-source frameworks. Sooner or later, New Relic anticipates leveraging Amazon S3 Vectors. It gives as much as 90% value discount for vector storage and querying in comparison with standard approaches, enabling the dealing with of massive-scale AI workloads extra effectively. New Relic is seeking to discover Amazon Bedrock AgentCore for enterprise-grade safety, reminiscence administration, and scalable AI agent deployment, supporting strong manufacturing capabilities.
Moreover, New Relic is exploring Strands Agent Workflows, an open-source SDK that streamlines constructing AI brokers from easy conversational assistants to complicated autonomous workflows. This know-how stack positions New Relic NOVA to ship enterprise-ready AI options that scale seamlessly whereas sustaining value effectivity and developer productiveness.
Conclusion
The journey of making New Relic NOVA demonstrates how enterprises can use the generative AI companies of AWS to rework organizational productiveness. By the mixing of Amazon Bedrock, Amazon Kendra, and different AWS companies, New Relic created an AI assistant that transforms their inner operations. Working with the Generative AI Innovation Middle of AWS, New Relic achieved a 95% discount in info search time throughout their group whereas automating complicated operational workflows.
Study extra about remodeling your online business with generative AI by visiting the Generative AI Innovation Middle or communicate with an AWS Associate Specialist or AWS Consultant to know the way we may also help speed up your online business.
Additional studying
Concerning the authors
Yicheng Shen is a lead software program engineer for New Relic NOVA, the place he focuses on creating gen AI and agentic options that remodel how companies perceive their utility efficiency. When he’s not constructing clever programs, you’ll discover him exploring the outside along with his household and their canine.
Sarathy Varadarajan, Senior Director of Engineering at New Relic, drives AI-first transformation and developer productiveness, aiming for tenfold positive aspects by way of clever automation and enterprise AI. He scaled engineering groups from 15 to over 350 in Bangalore and Hyderabad. He enjoys household time and volleyball.
Joe King is an AWS Senior Knowledge Scientist on the Generative AI Innovation Middle, the place he helps organizations architect and implement cutting-edge generative AI options. With deep experience in science, engineering, and AI/ML structure, he makes a speciality of remodeling complicated generative AI use circumstances into scalable options on AWS.
Priyashree Roy is an AWS information scientist on the Generative AI Innovation Middle, the place she applies her deep experience in machine studying and generative AI to construct cutting-edge options for AWS strategic clients. With a PhD in experimental particle physics, she brings a rigorous scientific strategy to fixing complicated real-world issues by superior AI applied sciences.
Gene Su is an AWS Knowledge Scientist on the Generative AI Innovation Middle, specializing in generative AI options for finance, retail, and different industries. He makes use of his experience in giant language fashions (LLMs) to ship generative AI purposes on AWS.
Dipanshu Jain is a generative AI Strategist at AWS, serving to unlock the potential of gen AI by strategic advisory and tailor-made answer growth. Specialised in figuring out high-impact generative AI use circumstances, shaping execution roadmaps, and guiding cross-functional groups by proofs of idea—from discovery to manufacturing.
Ameer Hakme is an AWS Options Architect that collaborates with Impartial Software program Distributors (ISVs) within the Northeast area, aiding in designing and constructing scalable and fashionable platforms on the AWS Cloud. An knowledgeable in AI/ML and generative AI, Ameer helps clients unlock the potential of those cutting-edge applied sciences. In his leisure time, he enjoys driving his motorbike and spending high quality time along with his household.
