Fast ML experimentation for enterprises with Amazon SageMaker AI and Comet

This publish was written with Sarah Ostermeier from Comet.

As enterprise organizations scale their machine studying (ML) initiatives from proof of idea to manufacturing, the complexity of managing experiments, monitoring mannequin lineage, and managing reproducibility grows exponentially. That is primarily as a result of knowledge scientists and ML engineers always discover totally different combos of hyperparameters, mannequin architectures, and dataset variations, producing huge quantities of metadata that should be tracked for reproducibility and compliance. Because the ML mannequin improvement scales throughout a number of groups and regulatory necessities intensify, monitoring experiments turns into much more advanced. With rising AI rules, significantly within the EU, organizations now require detailed audit trails of mannequin coaching knowledge, efficiency expectations, and improvement processes, making experiment monitoring a enterprise necessity and never only a finest follow.

Amazon SageMaker AI gives the managed infrastructure enterprises have to scale ML workloads, dealing with compute provisioning, distributed coaching, and deployment with out infrastructure overhead. Nevertheless, groups nonetheless want strong experiment monitoring, mannequin comparability, and collaboration capabilities that transcend primary logging.

Comet is a complete ML experiment administration platform that robotically tracks, compares, and optimizes ML experiments throughout the whole mannequin lifecycle. It gives knowledge scientists and ML engineers with highly effective instruments for experiment monitoring, mannequin monitoring, hyperparameter optimization, and collaborative mannequin improvement. It additionally affords Opik, Comet’s open supply platform for LLM observability and improvement.

Comet is on the market in SageMaker AI as a Companion AI App, as a completely managed experiment administration functionality, with enterprise-grade safety, seamless workflow integration, and an easy procurement course of via AWS Market.

The mixture addresses the wants of an enterprise ML workflow end-to-end, the place SageMaker AI handles infrastructure and compute, and Comet gives the experiment administration, mannequin registry, and manufacturing monitoring capabilities that groups require for regulatory compliance and operational effectivity. On this publish, we reveal a whole fraud detection workflow utilizing SageMaker AI with Comet, showcasing reproducibility and audit-ready logging wanted by enterprises at this time.

Enterprise-ready Comet on SageMaker AI

Earlier than continuing to setup directions, organizations should establish their working mannequin and based mostly on that, resolve how Comet goes to be arrange. We advocate implementing Comet utilizing a federated working mannequin. On this structure, Comet is centrally managed and hosted in a shared companies account, and every knowledge science crew maintains absolutely autonomous environments. Every working mannequin comes with their very own units of advantages and limitations. For extra data, consult with SageMaker Studio Administration Greatest Practices.

Let’s dive into the setup of Comet in SageMaker AI. Massive enterprise typically have the next personas:

• Directors – Chargeable for establishing the frequent infrastructure companies and surroundings to be used case groups
• Customers – ML practitioners from use case groups who use the environments arrange by platform crew to unravel their enterprise issues

Within the following sections, we undergo every persona’s journey.

Comet works properly with each SageMaker AI and Amazon SageMaker. SageMaker AI gives the Amazon SageMaker Studio built-in improvement surroundings (IDE), and SageMaker gives the Amazon SageMaker Unified Studio IDE. For this publish, we use SageMaker Studio.

Administrator journey

On this state of affairs, the administrator receives a request from a crew engaged on a fraud detection use case to provision an ML surroundings with a completely managed coaching and experimentation setup. The administrator’s journey consists of the next steps:

1. Comply with the conditions to arrange Companion AI Apps. This units up permissions for directors, permitting Comet to imagine a SageMaker AI execution function on behalf of the customers and extra privileges for managing the Comet subscription via AWS Market.
2. On the SageMaker AI console, underneath Functions and IDEs within the navigation pane, select Companion AI Apps, then select View particulars for Comet.

The main points are proven, together with the contract pricing mannequin for Comet and infrastructure tier estimated prices.

Comet gives totally different subscription choices starting from a 1-month to 36-month contract. With this contract, customers can entry Comet in SageMaker. Based mostly on the variety of customers, the admin can overview and analyze the suitable occasion dimension for the Comet dashboard server. Comet helps 5–500 customers working greater than 100 experiment jobs..

3. Select Go to Market to subscribe to be redirected to the Comet itemizing on AWS Market.
4. Select View buy choices.

5. Within the subscription kind, present the required particulars.

When the subscription is full, the admin can begin configuring Comet.

6. Whereas deploying Comet, add the challenge lead of the fraud detection use case crew as an admin to handle the admin operations for the Comet dashboard.

It takes a couple of minutes for the Comet server to be deployed. For extra particulars on this step, consult with Companion AI App provisioning.

7. Arrange a SageMaker AI area following the steps in Use customized setup for Amazon SageMaker AI. As a finest follow, present a pre-signed area URL for the use case crew member to instantly entry the Comet UI with out logging in to the SageMaker console.
8. Add the crew members to this area and allow entry to Comet whereas configuring the area.

Now the SageMaker AI area is prepared for customers to log in to and begin engaged on the fraud detection use case.

Consumer journey

Now let’s discover the journey of an ML practitioner from the fraud detection use case. The consumer completes the next steps:

1. Log in to the SageMaker AI area via the pre-signed URL.

You’ll be redirected to the SageMaker Studio IDE. Your consumer title and AWS Id and Entry Administration (IAM) execution function are preconfigured by the admin.

2. Create a JupyterLab Area following the JupyterLab consumer information.
3. You can begin engaged on the fraud detection use case by spinning up a Jupyter pocket book.

The admin has additionally arrange required entry to the info via an Amazon Easy Storage Service (Amazon S3) bucket.

4. To entry Comet APIs, set up the comet_ml library and configure the required surroundings variables as described in Arrange the Amazon SageMaker Companion AI Apps SDKs.
5. To entry the Comet UI, select Companion AI Apps within the SageMaker Studio navigation pane and select Open for Comet.

Now, let’s stroll via the use case implementation.

Resolution overview

This use case highlights frequent enterprise challenges: working with imbalanced datasets (on this instance, solely 0.17% of transactions are fraudulent), requiring a number of experiment iterations, and sustaining full reproducibility for regulatory compliance. To observe alongside, consult with the Comet documentation and Quickstart information for added setup and API particulars.

For this use case, we use the Credit score Card Fraud Detection dataset. The dataset accommodates bank card transactions with binary labels representing fraudulent (1) or respectable (0) transactions. Within the following sections, we stroll via a number of the vital sections of the implementation. All the code of the implementation is on the market within the GitHub repository.

Stipulations

As a prerequisite, configure the required imports and surroundings variables for the Comet and SageMaker integration:

# Comet ML for experiment monitoring
import comet_ml
from comet_ml import Experiment, API, Artifact
from comet_ml.integration.sagemaker import log_sagemaker_training_job_v1
AWS_PARTNER_APP_AUTH=true
AWS_PARTNER_APP_ARN=
COMET_API_KEY= 	
# From Particulars Web page, click on Open Comet. Within the prime #proper nook, click on on consumer -> API # Key
# Comet ML configuration
COMET_WORKSPACE = ''
COMET_PROJECT_NAME = ''

Put together the dataset

Certainly one of Comet’s key enterprise options is computerized dataset versioning and lineage monitoring. This functionality gives full auditability of what knowledge was used to coach every mannequin, which is crucial for regulatory compliance and reproducibility. Begin by loading the dataset:

# Create a Comet Artifact to trace our uncooked dataset
dataset_artifact = Artifact(
    title="fraud-dataset",
    artifact_type="dataset",
    aliases=["raw"]
)
# Add the uncooked dataset file to the artifact
dataset_artifact.add_remote(s3_data_path, metadata={
    "dataset_stage": "uncooked", 
    "dataset_split": "not_split", 
    "preprocessing": "none"
})

Begin a Comet experiment

With the dataset artifact created, now you can begin monitoring the ML workflow. Making a Comet experiment robotically begins capturing code, put in libraries, system metadata, and different contextual data within the background. You’ll be able to log the dataset artifact created earlier within the experiment. See the next code:

# Create a brand new Comet experiment
experiment_1 = comet_ml.Experiment(
    project_name=COMET_PROJECT_NAME,
    workspace=COMET_WORKSPACE,
)
# Log the dataset artifact to this experiment for lineage monitoring
experiment_1.log_artifact(dataset_artifact)

Preprocess the info

The subsequent steps are customary preprocessing steps, together with eradicating duplicates, dropping unneeded columns, splitting into practice/validation/take a look at units, and standardizing options utilizing scikit-learn’s StandardScaler. We wrap the processing code in preprocess.py and run it as a SageMaker Processing job. See the next code:

# Run SageMaker processing job
processor = SKLearnProcessor(
    framework_version='1.0-1',
    function=sagemaker.get_execution_role(),
    instance_count=1,
    instance_type="ml.t3.medium"
)
processor.run(
    code="preprocess.py",
    inputs=[ProcessingInput(source=s3_data_path, destination='/opt/ml/processing/input')],
    outputs=[ProcessingOutput(source="/opt/ml/processing/output", destination=f's3://{bucket_name}/{processed_data_prefix}')]
)

After you submit the processing job, SageMaker AI launches the compute cases, processes and analyzes the enter knowledge, and releases the assets upon completion. The output of the processing job is saved within the S3 bucket specified.

Subsequent, create a brand new model of the dataset artifact to trace the processed knowledge. Comet robotically variations artifacts with the identical title, sustaining full lineage from uncooked to processed knowledge.

# Create an up to date model of the 'fraud-dataset' Artifact for the preprocessed knowledge
preprocessed_dataset_artifact = Artifact(
    title="fraud-dataset",
    artifact_type="dataset", 
    aliases=["preprocessed"],
    metadata={
        "description": "Bank card fraud detection dataset",
        "fraud_percentage": f"{fraud_percentage:.3f}%",
        "dataset_stage": "preprocessed",
        "preprocessing": "StandardScaler + practice/val/take a look at cut up",
    }
)
# Add our practice, validation, and take a look at dataset information as distant belongings 
preprocessed_dataset_artifact.add_remote(
    uri=f's3://{bucket_name}/{processed_data_prefix}',
    logical_path="split_data"
)
# Log the up to date dataset to the experiment to trace the updates
experiment_1.log_artifact(preprocessed_dataset_artifact)

The Comet and SageMaker AI experiment workflow

Knowledge scientists choose fast experimentation; due to this fact, we organized the workflow into reusable utility features that may be referred to as a number of instances with totally different hyperparameters whereas sustaining constant logging and analysis throughout all runs. On this part, we showcase the utility features together with a short snippet of the code contained in the perform:

    # Create SageMaker estimator
    estimator = Estimator(
        image_uri=xgboost_image,
        function=execution_role,
        instance_count=1,
        instance_type="ml.m5.giant",
        output_path=model_output_path,
        sagemaker_session=sagemaker_session_obj,
        hyperparameters=hyperparameters_dict,
        max_run=1800  # Most coaching time in seconds
    )
    # Begin coaching
    estimator.match({
        'practice': train_channel,
        'validation': val_channel
    })

• log_training_job() – Captures the coaching metadata and metrics and hyperlinks the mannequin asset to the experiment for full traceability:

# Log SageMaker coaching job to Comet 
    log_sagemaker_training_job_v1(
        estimator=training_estimator,
        experiment=api_experiment
    )

• log_model_to_comet() – Hyperlinks mannequin artifacts to Comet, captures the coaching metadata, and hyperlinks the mannequin asset to the experiment for full traceability:

experiment.log_remote_model(
        model_name=model_name,
        uri=model_artifact_path,
        metadata=metadata
    )

• deploy_and_evaluate_model() – Performs mannequin deployment and analysis, and metric logging:

# Deploy to endpoint
predictor = estimator.deploy(
initial_instance_count=1,
       instance_type="ml.m5.xlarge")
# Log metrics and visualizations to Comet 
experiment.log_metrics(metrics) experiment.log_confusion_matrix(matrix=cm,labels=['Normal', 'Fraud']) 
# Log ROC curve 
fpr, tpr, _ = roc_curve(y_test, y_pred_prob_as_np_array) experiment.log_curve("roc_curve", x=fpr, y=tpr)

The entire prediction and analysis code is on the market within the GitHub repository.

Run the experiments

Now you may run a number of experiments by calling the utility features with totally different configurations and examine experiments to seek out essentially the most optimum settings for the fraud detection use case.

For the primary experiment, we set up a baseline utilizing customary XGBoost hyperparameters:

# Outline hyperparameters for first experiment
hyperparameters_v1 = {
    'goal': 'binary:logistic',	# Binary classification
    'num_round': 100,                   # Variety of boosting rounds
    'eval_metric': 'auc',               # Analysis metric
    'learning_rate': 0.15,              # Studying fee
    'booster': 'gbtree'                 # Booster algorithm
}
# Prepare the mannequin
estimator_1 = practice(
    model_output_path=f"s3://{bucket_name}/{model_output_prefix}/1",
    execution_role=function,
    sagemaker_session_obj=sagemaker_session,
    hyperparameters_dict=hyperparameters_v1,
    train_channel_loc=train_channel_location,
    val_channel_loc=validation_channel_location
)
# log the coaching job and mannequin artifact
log_training_job(experiment_key = experiment_1.get_key(), training_estimator=estimator_1)
log_model_to_comet(experiment = experiment_1,
                   model_name="fraud-detection-xgb-v1", 
                   model_artifact_path=estimator_1.model_data, 
                   metadata=metadata)
# Deploy and consider
deploy_and_evaluate_model(experiment=experiment_1,
                          estimator=estimator_1,
                          X_test_scaled=X_test_scaled,
                          y_test=y_test
                          )

Whereas working a Comet experiment from a Jupyter pocket book, we have to finish the experiment to verify the whole lot is captured and endured within the Comet server. See the next code: experiment_1.finish()

When the baseline experiment is full, you may run extra experiments with totally different hyperparameters. Try the pocket book to see the small print of each experiments.

When the second experiment is full, navigate to the Comet UI to match these two experiment runs.

View Comet experiments within the UI

To entry the UI, you may find the URL within the SageMaker Studio IDE or by executing the code supplied within the pocket book: experiment_2.url

The next screenshot exhibits the Comet experiments UI. The experiment particulars are for illustration functions solely and don’t symbolize a real-world fraud detection experiment.

This concludes the fraud detection experiment.

Clear up

For the experimentation half, SageMaker processing and coaching infrastructure is ephemeral in nature and shuts down robotically when the job is full. Nevertheless, you will need to nonetheless manually clear up a number of assets to keep away from pointless prices:

1. Shut down the SageMaker JupyterLab Area after use. For directions, consult with Idle shutdown.
2. The Comet subscription renews based mostly on the contract chosen. Cancel the contract when there is no such thing as a additional requirement to resume the Comet subscription.

Benefits of SageMaker and Comet integration

Having demonstrated the technical workflow, let’s look at the broader benefits this integration gives.

Streamlined mannequin improvement

The Comet and SageMaker mixture reduces the handbook overhead of working ML experiments. Whereas SageMaker handles infrastructure provisioning and scaling, Comet’s computerized logging captures hyperparameters, metrics, code, put in libraries, and system efficiency out of your coaching jobs with out extra configuration. This helps groups deal with mannequin improvement quite than experiment bookkeeping.Comet’s visualization capabilities lengthen past primary metric plots. Constructed-in charts allow fast experiment comparability, and customized Python panels assist domain-specific evaluation instruments for debugging mannequin habits, optimizing hyperparameters, or creating specialised visualizations that customary instruments can’t present.

Enterprise collaboration and governance

For enterprise groups, the mix creates a mature platform for scaling ML tasks throughout regulated environments. SageMaker gives constant, safe ML environments, and Comet allows seamless collaboration with full artifact and mannequin lineage monitoring. This helps keep away from pricey errors that happen when groups can’t recreate earlier outcomes.

Full ML lifecycle integration

Not like level options that solely tackle coaching or monitoring, Comet paired with SageMaker helps your full ML lifecycle. Fashions could be registered in Comet’s mannequin registry with full model monitoring and governance. SageMaker handles mannequin deployment, and Comet maintains the lineage and approval workflows for mannequin promotion. Comet’s manufacturing monitoring capabilities monitor mannequin efficiency and knowledge drift after deployment, making a closed loop the place manufacturing insights inform your subsequent spherical of SageMaker experiments.

Conclusion

On this publish, we confirmed learn how to use SageMaker and Comet collectively to spin up absolutely managed ML environments with reproducibility and experiment monitoring capabilities.

To reinforce your SageMaker workflows with complete experiment administration, deploy Comet instantly in your SageMaker surroundings via the AWS Market, and share your suggestions within the feedback.

For extra details about the companies and options mentioned on this publish, consult with the next assets:

Concerning the authors

Vikesh Pandey is a Principal GenAI/ML Specialist Options Architect at AWS, serving to giant monetary establishments undertake and scale generative AI and ML workloads. He’s the creator of e book “Generative AI for monetary companies.” He carries greater than 15 years of expertise constructing enterprise-grade purposes on generative AI/ML and associated applied sciences. In his spare time, he performs an unnamed sport together with his son that lies someplace between soccer and rugby.

Naufal Mir is a Senior GenAI/ML Specialist Options Architect at AWS. He focuses on serving to prospects construct, practice, deploy and migrate machine studying workloads to SageMaker. He beforehand labored at monetary companies institutes growing and working techniques at scale. Outdoors of labor, he enjoys extremely endurance working and biking.

Sarah Ostermeier is a Technical Product Advertising Supervisor at Comet. She makes a speciality of bringing Comet’s GenAI and ML developer merchandise to the engineers who want them via technical content material, academic assets, and product messaging. She has beforehand labored as an ML engineer, knowledge scientist, and buyer success supervisor, serving to prospects implement and scale AI options. Outdoors of labor she enjoys touring off the crushed path, writing about AI, and studying science fiction.