Use {custom} metrics to guage your generative AI software with Amazon Bedrock

With Amazon Bedrock Evaluations, you may consider basis fashions (FMs) and Retrieval Augmented Era (RAG) methods, whether or not hosted on Amazon Bedrock or one other mannequin or RAG system hosted elsewhere, together with Amazon Bedrock Data Bases or multi-cloud and on-premises deployments. We lately introduced the final availability of the big language mannequin (LLM)-as-a-judge approach in mannequin analysis and the brand new RAG analysis instrument, additionally powered by an LLM-as-a-judge behind the scenes. These instruments are already empowering organizations to systematically consider FMs and RAG methods with enterprise-grade instruments. We additionally talked about that these analysis instruments don’t should be restricted to fashions or RAG methods hosted on Amazon Bedrock; with the convey your individual inference (BYOI) responses characteristic, you may consider fashions or functions should you use the enter formatting necessities for both providing.

The LLM-as-a-judge approach powering these evaluations permits automated, human-like analysis high quality at scale, utilizing FMs to evaluate high quality and accountable AI dimensions with out handbook intervention. With built-in metrics like correctness (factual accuracy), completeness (response thoroughness), faithfulness (hallucination detection), and accountable AI metrics akin to harmfulness and reply refusal, you and your staff can consider fashions hosted on Amazon Bedrock and data bases natively, or utilizing BYOI responses out of your custom-built methods.

Amazon Bedrock Evaluations provides an intensive record of built-in metrics for each analysis instruments, however there are occasions once you may wish to outline these analysis metrics another way, or make fully new metrics which can be related to your use case. For instance, you may wish to outline a metric that evaluates an software response’s adherence to your particular model voice, or wish to classify responses based on a {custom} categorical rubric. You may wish to use numerical scoring or categorical scoring for varied functions. For these causes, you want a method to make use of {custom} metrics in your evaluations.

Now with Amazon Bedrock, you may develop {custom} analysis metrics for each mannequin and RAG evaluations. This functionality extends the LLM-as-a-judge framework that drives Amazon Bedrock Evaluations.

On this publish, we exhibit how one can use {custom} metrics in Amazon Bedrock Evaluations to measure and enhance the efficiency of your generative AI functions based on your particular enterprise necessities and analysis standards.

Overview

Customized metrics in Amazon Bedrock Evaluations supply the next options:

• Simplified getting began expertise – Pre-built starter templates can be found on the AWS Administration Console based mostly on our industry-tested built-in metrics, with choices to create from scratch for particular analysis standards.
• Versatile scoring methods – Help is obtainable for each quantitative (numerical) and qualitative (categorical) scoring to create ordinal metrics, nominal metrics, and even use analysis instruments for classification duties.
• Streamlined workflow administration – It can save you {custom} metrics for reuse throughout a number of analysis jobs or import beforehand outlined metrics from JSON recordsdata.
• Dynamic content material integration – With built-in template variables (for instance, {{immediate}}, {{prediction}}, and {{context}}), you may seamlessly inject dataset content material and mannequin outputs into analysis prompts.
• Customizable output management – You should use our advisable output schema for constant outcomes, with superior choices to outline {custom} output codecs for specialised use instances.

Customized metrics provide you with unprecedented management over the way you measure AI system efficiency, so you may align evaluations together with your particular enterprise necessities and use instances. Whether or not assessing factuality, coherence, helpfulness, or domain-specific standards, {custom} metrics in Amazon Bedrock allow extra significant and actionable analysis insights.

Within the following sections, we stroll via the steps to create a job with mannequin analysis and {custom} metrics utilizing each the Amazon Bedrock console and the Python SDK and APIs.

Supported knowledge codecs

On this part, we assessment some vital knowledge codecs.

Choose immediate importing

To add your beforehand saved {custom} metrics into an analysis job, comply with the JSON format within the following examples.

The next code illustrates a definition with numerical scale:

{
    "customMetricDefinition": {
        "metricName": "my_custom_metric",
        "directions": "Your full {custom} metric immediate together with at the very least one {{enter variable}}",
        "ratingScale": [
            {
                "definition": "first rating definition",
                "value": {
                    "floatValue": 3
                }
            },
            {
                "definition": "second rating definition",
                "value": {
                    "floatValue": 2
                }
            },
            {
                "definition": "third rating definition",
                "value": {
                    "floatValue": 1
                }
            }
        ]
    }
}

The next code illustrates a definition with string scale:

{
    "customMetricDefinition": {
        "metricName": "my_custom_metric",
        "directions": "Your full {custom} metric immediate together with at the very least one {{enter variable}}",
        "ratingScale": [
            {
                "definition": "first rating definition",
                "value": {
                    "stringValue": "first value"
                }
            },
            {
                "definition": "second rating definition",
                "value": {
                    "stringValue": "second value"
                }
            },
            {
                "definition": "third rating definition",
                "value": {
                    "stringValue": "third value"
                }
            }
        ]
    }
}

The next code illustrates a definition with no scale:

{
    "customMetricDefinition": {
        "metricName": "my_custom_metric",
        "directions": "Your full {custom} metric immediate together with at the very least one {{enter variable}}"
    }
}

For extra data on defining a decide immediate with no scale, see the very best practices part later on this publish.

Mannequin analysis dataset format

When utilizing LLM-as-a-judge, just one mannequin might be evaluated per analysis job. Consequently, you need to present a single entry within the modelResponses record for every analysis, although you may run a number of analysis jobs to match completely different fashions. The modelResponses discipline is required for BYOI jobs, however not wanted for non-BYOI jobs. The next is the enter JSONL format for LLM-as-a-judge in mannequin analysis. Fields marked with ? are elective.

{
    "immediate": string
    "referenceResponse"?: string
    "class"?: string
     "modelResponses"?: [
        {
            "response": string
            "modelIdentifier": string
        }
    ]
}

RAG analysis dataset format

We up to date the analysis job enter dataset format to be much more versatile for RAG analysis. Now, you may convey referenceContexts, that are anticipated retrieved passages, so you may evaluate your precise retrieved contexts to your anticipated retrieved contexts. You could find the brand new referenceContexts discipline within the up to date JSONL schema for RAG analysis:

{
    "conversationTurns": [{
            "prompt": {
                "content": [{
                    "text": string
                }]
            },
            "referenceResponses": [{
                "content": [{
                    "text": string
                }]
            }],
            "referenceContexts" ? : [{
                "content": [{
                    "text": string
                }]
            }],
            "output": {
                "textual content": string "modelIdentifier" ? : string "knowledgeBaseIdentifier": string "retrievedPassages": {
                    "retrievalResults": [{
                        "name" ? : string "content": {
                            "text": string
                        },
                        "metadata" ? : {
                            [key: string]: string
                        }
                    }]
                }
            }]
    }

Variables for knowledge injection into decide prompts

To make it possible for your knowledge is injected into the decide prompts in the best place, use the variables from the next desk. We have now additionally included a information to indicate you the place the analysis instrument will pull knowledge out of your enter file, if relevant. There are instances the place should you convey your individual inference responses to the analysis job, we are going to use that knowledge out of your enter file; should you don’t use convey your individual inference responses, then we are going to name the Amazon Bedrock mannequin or data base and put together the responses for you.

The next desk summarizes the variables for mannequin analysis.

The next desk summarizes the variables for RAG analysis (retrieve solely).

The next desk summarizes the variables for RAG analysis (retrieve and generate).

Conditions

To make use of the LLM-as-a-judge mannequin analysis and RAG analysis options with BYOI, you need to have the next stipulations:

Create a mannequin analysis job with {custom} metrics utilizing Amazon Bedrock Evaluations

Full the next steps to create a job with mannequin analysis and {custom} metrics utilizing Amazon Bedrock Evaluations:

1. On the Amazon Bedrock console, select Evaluations within the navigation pane and select the Fashions
2. Within the Mannequin analysis part, on the Create dropdown menu, select Computerized: mannequin as a decide.
   
3. For the Mannequin analysis particulars, enter an analysis title and elective description.
   
4. For Evaluator mannequin, select the mannequin you wish to use for computerized analysis.
5. For Inference supply, choose the supply and select the mannequin you wish to consider.

For this instance, we selected Claude 3.5 Sonnet because the evaluator mannequin, Bedrock fashions as our inference supply, and Claude 3.5 Haiku as our mannequin to guage.

5. The console will show the default metrics for the evaluator mannequin you selected. You’ll be able to choose different metrics as wanted.
   
6. Within the Customized Metrics part, we create a brand new metric known as “Comprehensiveness.” Use the template offered and modify based mostly in your metrics. You should use the next variables to outline the metric, the place solely {{prediction}} is necessary:
   1. immediate
   2. prediction
   3. ground_truth

The next is the metric we outlined in full:

Your function is to evaluate the comprehensiveness of a solution based mostly on the query and 
the prediction. Assess the standard, accuracy, and helpfulness of language mannequin response,
 and use these to evaluate how complete the response is. Award greater scores to responses
 which can be detailed and considerate.

Rigorously consider the comprehensiveness of the LLM response for the given question (immediate)
 in opposition to all specified standards. Assign a single general rating that greatest represents the 
comprehensivenss, and supply a quick clarification justifying your score, referencing 
particular strengths and weaknesses noticed.

When evaluating the response high quality, take into account the next rubrics:
- Accuracy: Factual correctness of knowledge offered
- Completeness: Protection of vital elements of the question
- Readability: Clear group and presentation of knowledge
- Helpfulness: Sensible utility of the response to the person

Consider the next:

Question:
{{immediate}}

Response to guage:
{{prediction}}

7. Create the output schema and extra metrics. Right here, we outline a scale that gives most factors (10) if the response may be very complete, and 1 if the response just isn’t complete in any respect.
   
8. For Datasets, enter your enter and output areas in Amazon S3.
9. For Amazon Bedrock IAM function – Permissions, choose Use an current service function and select a job.
   
10. Select Create and look forward to the job to finish.
    

Issues and greatest practices

When utilizing the output schema of the {custom} metrics, be aware the next:

• Should you use the built-in output schema (advisable), don’t add your grading scale into the principle decide immediate. The analysis service will mechanically concatenate your decide immediate directions together with your outlined output schema score scale and a few structured output directions (distinctive to every decide mannequin) behind the scenes. That is so the analysis service can parse the decide mannequin’s outcomes and show them on the console in graphs and calculate common values of numerical scores.
• The absolutely concatenated decide prompts are seen within the Preview window if you’re utilizing the Amazon Bedrock console to assemble your {custom} metrics. As a result of decide LLMs are inherently stochastic, there could be some responses we will’t parse and show on the console and use in your common rating calculations. Nevertheless, the uncooked decide responses are at all times loaded into your S3 output file, even when the analysis service can’t parse the response rating from the decide mannequin.
• Should you don’t use the built-in output schema characteristic (we suggest you utilize it as a substitute of ignoring it), then you’re liable for offering your score scale within the decide immediate directions physique. Nevertheless, the analysis service won’t add structured output directions and won’t parse the outcomes to indicate graphs; you will notice the complete decide output plaintext outcomes on the console with out graphs and the uncooked knowledge will nonetheless be in your S3 bucket.

Create a mannequin analysis job with {custom} metrics utilizing the Python SDK and APIs

To make use of the Python SDK to create a mannequin analysis job with {custom} metrics, comply with these steps (or check with our instance pocket book):

2. Arrange the required configurations, which ought to embrace your mannequin identifier for the default metrics and {custom} metrics evaluator, IAM function with acceptable permissions, Amazon S3 paths for enter knowledge containing your inference responses, and output location for outcomes:

   import boto3
   import time
   from datetime import datetime
   
   # Configure data base and mannequin settings
   evaluator_model = "anthropic.claude-3-5-sonnet-20240620-v1:0"
   generator_model = "amazon.nova-lite-v1:0"
   custom_metrics_evaluator_model = "anthropic.claude-3-5-sonnet-20240620-v1:0"
   role_arn = "arn:aws:iam:::function/"
   BUCKET_NAME = ""
   
   # Specify S3 areas
   input_data = f"s3://{BUCKET_NAME}/evaluation_data/enter.jsonl"
   output_path = f"s3://{BUCKET_NAME}/evaluation_output/"
   
   # Create Bedrock consumer
   # NOTE: You'll be able to change the area title to the area of your selecting.
   bedrock_client = boto3.consumer('bedrock', region_name="us-east-1") 

3. To outline a {custom} metric for mannequin analysis, create a JSON construction with a customMetricDefinition Embody your metric’s title, write detailed analysis directions incorporating template variables (akin to {{immediate}} and {{prediction}}), and outline your ratingScale array with evaluation values utilizing both numerical scores (floatValue) or categorical labels (stringValue). This correctly formatted JSON schema permits Amazon Bedrock to guage mannequin outputs constantly based on your particular standards.

   comprehensiveness_metric ={
       "customMetricDefinition": {
           "title": "comprehensiveness",
           "directions": """Your function is to evaluate the comprehensiveness of an 
   reply based mostly on the query and the prediction. Assess the standard, accuracy, 
   and helpfulness of language mannequin response, and use these to evaluate how complete
    the response is. Award greater scores to responses which can be detailed and considerate.
   
   Rigorously consider the comprehensiveness of the LLM response for the given question (immediate)
    in opposition to all specified standards. Assign a single general rating that greatest represents the 
   comprehensivenss, and supply a quick clarification justifying your score, referencing 
   particular strengths and weaknesses noticed.
   
   When evaluating the response high quality, take into account the next rubrics:
   - Accuracy: Factual correctness of knowledge offered
   - Completeness: Protection of vital elements of the question
   - Readability: Clear group and presentation of knowledge
   - Helpfulness: Sensible utility of the response to the person
   
   Consider the next:
   
   Question:
   {{immediate}}
   
   Response to guage:
   {{prediction}}""",
           "ratingScale": [
               {
                   "definition": "Very comprehensive",
                   "value": {
                       "floatValue": 10
                   }
               },
               {
                   "definition": "Mildly comprehensive",
                   "value": {
                       "floatValue": 3
                   }
               },
               {
                   "definition": "Not at all comprehensive",
                   "value": {
                       "floatValue": 1
                   }
               }
           ]
       }
   }

4. To create a mannequin analysis job with {custom} metrics, use the create_evaluation_job API and embrace your {custom} metric within the customMetricConfig part, specifying each built-in metrics (akin to Builtin.Correctness) and your {custom} metric within the metricNames array. Configure the job together with your generator mannequin, evaluator mannequin, and correct Amazon S3 paths for enter dataset and output outcomes.

   # Create the mannequin analysis job
   model_eval_job_name = f"model-evaluation-custom-metrics{datetime.now().strftime('%Y-%m-%d-%H-%M-%S')}"
   
   model_eval_job = bedrock_client.create_evaluation_job(
       jobName=model_eval_job_name,
       jobDescription="Consider mannequin efficiency with {custom} comprehensiveness metric",
       roleArn=role_arn,
       applicationType="ModelEvaluation",
       inferenceConfig={
           "fashions": [{
               "bedrockModel": {
                   "modelIdentifier": generator_model
               }
           }]
       },
       outputDataConfig={
           "s3Uri": output_path
       },
       evaluationConfig={
           "automated": {
               "datasetMetricConfigs": [{
                   "taskType": "General",
                   "dataset": {
                       "name": "ModelEvalDataset",
                       "datasetLocation": {
                           "s3Uri": input_data
                       }
                   },
                   "metricNames": [
                       "Builtin.Correctness",
                       "Builtin.Completeness",
                       "Builtin.Coherence",
                       "Builtin.Relevance",
                       "Builtin.FollowingInstructions",
                       "comprehensiveness"
                   ]
               }],
               "customMetricConfig": {
                   "customMetrics": [
                       comprehensiveness_metric
                   ],
                   "evaluatorModelConfig": {
                       "bedrockEvaluatorModels": [{
                           "modelIdentifier": custom_metrics_evaluator_model
                       }]
                   }
               },
               "evaluatorModelConfig": {
                   "bedrockEvaluatorModels": [{
                       "modelIdentifier": evaluator_model
                   }]
               }
           }
       }
   )
   
   print(f"Created mannequin analysis job: {model_eval_job_name}")
   print(f"Job ID: {model_eval_job['jobArn']}")

5. After submitting the analysis job, monitor its standing with get_evaluation_job and entry outcomes at your specified Amazon S3 location when full, together with the usual and {custom} metric efficiency knowledge.

Create a RAG system analysis with {custom} metrics utilizing Amazon Bedrock Evaluations

On this instance, we stroll via a RAG system analysis with a mix of built-in metrics and {custom} analysis metrics on the Amazon Bedrock console. Full the next steps:

1. On the Amazon Bedrock console, select Evaluations within the navigation pane.
2. On the RAG tab, select Create.
   
3. For the RAG analysis particulars, enter an analysis title and elective description.
   
4. For Evaluator mannequin, select the mannequin you wish to use for computerized analysis. The evaluator mannequin chosen right here might be used to calculate default metrics if chosen. For this instance, we selected Claude 3.5 Sonnet because the evaluator mannequin.
5. Embody any elective tags.
   
6. For Inference supply, choose the supply. Right here, you might have the choice to pick out between Bedrock Data Bases and Deliver your individual inference responses. Should you’re utilizing Amazon Bedrock Data Bases, you’ll need to decide on a beforehand created data base or create a brand new one. For BYOI responses, you may convey the immediate dataset, context, and output from a RAG system. For this instance, we selected Bedrock Data Base as our inference supply.
   
7. Specify the analysis kind, response generator mannequin, and built-in metrics. You’ll be able to select between a mixed retrieval and response analysis or a retrieval solely analysis, with choices to make use of default metrics, {custom} metrics, or each on your RAG analysis. The response generator mannequin is just required when utilizing an Amazon Bedrock data base because the inference supply. For the BYOI configuration, you may proceed and not using a response generator. For this instance, we chosen Retrieval and response era as our analysis kind and selected Nova Lite 1.0 as our response generator mannequin.
   
8. Within the Customized Metrics part, select your evaluator mannequin. We chosen Claude 3.5 Sonnet v1 as our evaluator mannequin for {custom} metrics.
9. Select Add {custom} metrics.
   
10. Create your new metric. For this instance, we create a brand new {custom} metric for our RAG analysis known as information_comprehensiveness. This metric evaluates how completely and fully the response addresses the question by utilizing the retrieved data. It measures the extent to which the response extracts and incorporates related data from the retrieved passages to supply a complete reply.
11. You’ll be able to select between importing a JSON file, utilizing a preconfigured template, or making a {custom} metric with full configuration management. For instance, you may choose the preconfigured templates for the default metrics and alter the scoring system or rubric. For our information_comprehensiveness metric, we choose the {custom} choice, which permits us to enter our evaluator immediate instantly.
    
12. For Directions, enter your immediate. For instance:

    Your function is to guage how comprehensively the response addresses the question 
    utilizing the retrieved data. Assess whether or not the response gives a radical 
    therapy of the topic by successfully using the accessible retrieved passages.
    
    Rigorously consider the comprehensiveness of the RAG response for the given question
     in opposition to all specified standards. Assign a single general rating that greatest represents
     the comprehensiveness, and supply a quick clarification justifying your score, 
    referencing particular strengths and weaknesses noticed.
    
    When evaluating response comprehensiveness, take into account the next rubrics:
    - Protection: Does the response make the most of the important thing related data from the retrieved
     passages?
    - Depth: Does the response present enough element on vital elements from the
     retrieved data?
    - Context utilization: How successfully does the response leverage the accessible
     retrieved passages?
    - Info synthesis: Does the response mix retrieved data to create
     a radical therapy?
    
    Consider the next:
    
    Question: {{immediate}}
    
    Retrieved passages: {{context}}
    
    Response to guage: {{prediction}}

13. Enter your output schema to outline how the {custom} metric outcomes might be structured, visualized, normalized (if relevant), and defined by the mannequin.

Should you use the built-in output schema (advisable), don’t add your score scale into the principle decide immediate. The analysis service will mechanically concatenate your decide immediate directions together with your outlined output schema score scale and a few structured output directions (distinctive to every decide mannequin) behind the scenes in order that your decide mannequin outcomes might be parsed. The absolutely concatenated decide prompts are seen within the Preview window if you’re utilizing the Amazon Bedrock console to assemble your {custom} metrics.

14. For Dataset and analysis outcomes S3 location, enter your enter and output areas in Amazon S3.
15. For Amazon Bedrock IAM function – Permissions, choose Use an current service function and select your function.
    
16. Select Create and look forward to the job to finish.
    

Begin a RAG analysis job with {custom} metrics utilizing the Python SDK and APIs

To make use of the Python SDK for creating an RAG analysis job with {custom} metrics, comply with these steps (or check with our instance pocket book):

1. Arrange the required configurations, which ought to embrace your mannequin identifier for the default metrics and {custom} metrics evaluator, IAM function with acceptable permissions, data base ID, Amazon S3 paths for enter knowledge containing your inference responses, and output location for outcomes:

   import boto3
   import time
   from datetime import datetime
   
   # Configure data base and mannequin settings
   knowledge_base_id = ""
   evaluator_model = "anthropic.claude-3-5-sonnet-20240620-v1:0"
   generator_model = "amazon.nova-lite-v1:0"
   custom_metrics_evaluator_model = "anthropic.claude-3-5-sonnet-20240620-v1:0"
   role_arn = "arn:aws:iam:::function/"
   BUCKET_NAME = ""
   
   # Specify S3 areas
   input_data = f"s3://{BUCKET_NAME}/evaluation_data/enter.jsonl"
   output_path = f"s3://{BUCKET_NAME}/evaluation_output/"
   
   # Configure retrieval settings
   num_results = 10
   search_type = "HYBRID"
   
   # Create Bedrock consumer
   # NOTE: You'll be able to change the area title to the area of your selecting
   bedrock_client = boto3.consumer('bedrock', region_name="us-east-1") 

2. To outline a {custom} metric for RAG analysis, create a JSON construction with a customMetricDefinition Embody your metric’s title, write detailed analysis directions incorporating template variables (akin to {{immediate}}, {{context}}, and {{prediction}}), and outline your ratingScale array with evaluation values utilizing both numerical scores (floatValue) or categorical labels (stringValue). This correctly formatted JSON schema permits Amazon Bedrock to guage responses constantly based on your particular standards.

   # Outline our {custom} information_comprehensiveness metric
   information_comprehensiveness_metric = {
       "customMetricDefinition": {
           "title": "information_comprehensiveness",
           "directions": """
           Your function is to guage how comprehensively the response addresses the 
   question utilizing the retrieved data. 
           Assess whether or not the response gives a radical therapy of the topic
   by successfully using the accessible retrieved passages.
   
   Rigorously consider the comprehensiveness of the RAG response for the given question
   in opposition to all specified standards. 
   Assign a single general rating that greatest represents the comprehensiveness, and 
   present a quick clarification justifying your score, referencing particular strengths
   and weaknesses noticed.
   
   When evaluating response comprehensiveness, take into account the next rubrics:
   - Protection: Does the response make the most of the important thing related data from the 
   retrieved passages?
   - Depth: Does the response present enough element on vital elements from 
   the retrieved data?
   - Context utilization: How successfully does the response leverage the accessible 
   retrieved passages?
   - Info synthesis: Does the response mix retrieved data to 
   create a radical therapy?
   
   Consider utilizing the next:
   
   Question: {{immediate}}
   
   Retrieved passages: {{context}}
   
   Response to guage: {{prediction}}
   """,
           "ratingScale": [
               {
                   "definition": "Very comprehensive",
                   "value": {
                       "floatValue": 3
                   }
               },
               {
                   "definition": "Moderately comprehensive",
                   "value": {
                       "floatValue": 2
                   }
               },
               {
                   "definition": "Minimally comprehensive",
                   "value": {
                       "floatValue": 1
                   }
               },
               {
                   "definition": "Not at all comprehensive",
                   "value": {
                       "floatValue": 0
                   }
               }
           ]
       }
   }

3. To create a RAG analysis job with {custom} metrics, use the create_evaluation_job API and embrace your {custom} metric within the customMetricConfig part, specifying each built-in metrics (Builtin.Correctness) and your {custom} metric within the metricNames array. Configure the job together with your data base ID, generator mannequin, evaluator mannequin, and correct Amazon S3 paths for enter dataset and output outcomes.

   # Create the analysis job
   retrieve_generate_job_name = f"rag-evaluation-generate-{datetime.now().strftime('%Y-%m-%d-%H-%M-%S')}"
   
   retrieve_generate_job = bedrock_client.create_evaluation_job(
       jobName=retrieve_generate_job_name,
       jobDescription="Consider retrieval and era with {custom} metric",
       roleArn=role_arn,
       applicationType="RagEvaluation",
       inferenceConfig={
           "ragConfigs": [{
               "knowledgeBaseConfig": {
                   "retrieveAndGenerateConfig": {
                       "type": "KNOWLEDGE_BASE",
                       "knowledgeBaseConfiguration": {
                           "knowledgeBaseId": knowledge_base_id,
                           "modelArn": generator_model,
                           "retrievalConfiguration": {
                               "vectorSearchConfiguration": {
                                   "numberOfResults": num_results
                               }
                           }
                       }
                   }
               }
           }]
       },
       outputDataConfig={
           "s3Uri": output_path
       },
       evaluationConfig={
           "automated": {
               "datasetMetricConfigs": [{
                   "taskType": "General",
                   "dataset": {
                       "name": "RagDataset",
                       "datasetLocation": {
                           "s3Uri": input_data
                       }
                   },
                   "metricNames": [
                       "Builtin.Correctness",
                       "Builtin.Completeness",
                       "Builtin.Helpfulness",
                       "information_comprehensiveness"
                   ]
               }],
               "evaluatorModelConfig": {
                   "bedrockEvaluatorModels": [{
                       "modelIdentifier": evaluator_model
                   }]
               },
               "customMetricConfig": {
                   "customMetrics": [
                       information_comprehensiveness_metric
                   ],
                   "evaluatorModelConfig": {
                       "bedrockEvaluatorModels": [{
                           "modelIdentifier": custom_metrics_evaluator_model
                       }]
                   }
               }
           }
       }
   )
   
   print(f"Created analysis job: {retrieve_generate_job_name}")
   print(f"Job ID: {retrieve_generate_job['jobArn']}")

4. After submitting the analysis job, you may test its standing utilizing the get_evaluation_job methodology and retrieve the outcomes when the job is full. The output might be saved on the Amazon S3 location specified within the output_path parameter, containing detailed metrics on how your RAG system carried out throughout the analysis dimensions together with {custom} metrics.

Customized metrics are solely accessible for LLM-as-a-judge. On the time of writing, we don’t settle for {custom} AWS Lambda features or endpoints for code-based {custom} metric evaluators. Human-based mannequin analysis has supported {custom} metric definition since its launch in November 2023.

Clear up

To keep away from incurring future prices, delete the S3 bucket, pocket book situations, and different assets that have been deployed as a part of the publish.

Conclusion

The addition of {custom} metrics to Amazon Bedrock Evaluations empowers organizations to outline their very own analysis standards for generative AI methods. By extending the LLM-as-a-judge framework with {custom} metrics, companies can now measure what issues for his or her particular use instances alongside built-in metrics. With help for each numerical and categorical scoring methods, these {custom} metrics allow constant evaluation aligned with organizational requirements and objectives.

As generative AI turns into more and more built-in into enterprise processes, the power to guage outputs in opposition to custom-defined standards is important for sustaining high quality and driving steady enchancment. We encourage you to discover these new capabilities via the Amazon Bedrock console and API examples offered, and uncover how customized analysis frameworks can improve your AI methods’ efficiency and enterprise affect.

Concerning the Authors

Shreyas Subramanian is a Principal Information Scientist and helps prospects by utilizing generative AI and deep studying to resolve their enterprise challenges utilizing AWS companies. Shreyas has a background in large-scale optimization and ML and in using ML and reinforcement studying for accelerating optimization duties.

Adewale Akinfaderin is a Sr. Information Scientist–Generative AI, Amazon Bedrock, the place he contributes to innovative improvements in foundational fashions and generative AI functions at AWS. His experience is in reproducible and end-to-end AI/ML strategies, sensible implementations, and serving to international prospects formulate and develop scalable options to interdisciplinary issues. He has two graduate levels in physics and a doctorate in engineering.

Jesse Manders is a Senior Product Supervisor on Amazon Bedrock, the AWS Generative AI developer service. He works on the intersection of AI and human interplay with the aim of making and bettering generative AI services and products to fulfill our wants. Beforehand, Jesse held engineering staff management roles at Apple and Lumileds, and was a senior scientist in a Silicon Valley startup. He has an M.S. and Ph.D. from the College of Florida, and an MBA from the College of California, Berkeley, Haas College of Enterprise.

Ishan Singh is a Sr. Generative AI Information Scientist at Amazon Net Providers, the place he helps prospects construct modern and accountable generative AI options and merchandise. With a robust background in AI/ML, Ishan focuses on constructing Generative AI options that drive enterprise worth. Exterior of labor, he enjoys enjoying volleyball, exploring native bike trails, and spending time together with his spouse and canine, Beau.