Saying SOCI-indexed pictures for quicker SageMaker AI jobs

Each minute your deep studying container picture spends downloading is a minute you pay for and a minute your group waits. Container picture obtain is pure overhead: billable job time that produces no mannequin progress. For contemporary deep studying pictures, that overhead is a number of minutes on each single job. In the present day, we’re excited to announce Seekable OCI (SOCI) indexing for Amazon SageMaker AI Coaching and Amazon SageMaker AI processing jobs, which takes full container picture obtain off the crucial path of each job.

With SOCI indexing, SageMaker AI lazy-loads your container picture. It fetches solely the recordsdata wanted to start out the container and streams the remaining within the background as your code runs, so your job begins in seconds as a substitute of minutes. You undertake it by pointing to a pre-indexed AWS Deep Studying Container (DLC) or including an index to your individual picture. Neither method requires adjustments to your job definition, entry level, or information channels. The result’s quicker job begins, decrease job value, and extra time spent constructing fashions relatively than ready on infrastructure.

SOCI-indexed pictures begin Amazon SageMaker AI coaching containers in seconds, with no impression on coaching efficiency. As we present within the What we measured part, this lowered GPU idle time throughout picture pulls by as much as 90 p.c in our benchmarks.

This submit reveals the right way to undertake SOCI indexing: what it’s, the outcomes we measured, and the right way to flip it on in your personal jobs.

SageMaker AI coaching and processing jobs run inside container pictures saved in Amazon Elastic Container Registry (Amazon ECR). Earlier than a job can execute your code, the service downloads the complete picture to the occasion. It decompresses and writes each layer to disk.

For small pictures this takes seconds. However deep studying pictures routinely exceed 10 GB or extra. That obtain provides minutes to each job, and also you pay for it as job time. The impression is largest precisely the place iteration velocity issues most: hyperparameter tuning sweeps for lots of of jobs, fast experiments, and quick processing jobs, the place picture obtain can dominate complete runtime.

Seekable OCI (SOCI) is an open supply know-how that lazy hundreds container pictures. As an alternative of downloading the entire picture earlier than beginning, the runtime makes use of a SOCI index to fetch solely the recordsdata wanted to launch the container instantly. It then streams the remaining within the background as your code accesses it.

A SOCI index is metadata that maps each file inside your picture. You retailer it in Amazon ECR as an OCI-compliant artifact linked to your unique picture. As a result of the index is a separate artifact, your picture digest and any present signatures stay legitimate.

SageMaker AI checks for an index robotically

Once you create a coaching or processing job, SageMaker AI checks whether or not the picture you specified has a SOCI index and behaves accordingly:

As a result of jobs at all times fall again to a normal obtain when wanted, including a SOCI index by no means prevents your job from operating. You solely acquire quicker startup. You by no means danger breaking an present workflow.

We pulled the identical container pictures two methods on actual SageMaker AI coaching jobs: commonplace obtain and SOCI-indexed lazy loading. We then ran full coaching jobs from each to measure the impact on coaching velocity.

Setup

• Eight pictures, 1.8–35.4 GB compressed.
• GPU and CPU cases ml.g4dn, ml.g5, ml.g6, ml.g6e, ml.g7, ml.g7e, ml.p4d, ml.p5, and ml.m7i, from xlarge to 48xlarge, within the US West (Oregon) Area (us-west-2) and the US East (N. Virginia) Area (us-east-1).
• Obtain time measured from the beginning of the picture pull till the container is able to run. All figures are medians of repeated runs.

Outcome 1: Obtain time drops from minutes to seconds at each picture dimension

The next chart compares container picture obtain time for the standard Docker pull towards SOCI lazy loading throughout eight picture sizes, from 1.8 GB to 35.4 GB.

Every picture dimension reveals two bars. The Docker obtain time bar (blue) grows steadily with picture dimension, from 48 seconds at 1.8 GB to 7 minutes 17 seconds at 35.4 GB.

A speedup label above every pair reveals the ratio of Docker time to SOCI time. Speedups vary from roughly 10x for the smallest picture to over 35x for the most important.

Determine 1: Container picture obtain time, commonplace Docker pull in comparison with SOCI lazy loading, throughout eight picture sizes

Normal obtain time grows with picture dimension as a result of each file is decompressed and written. Lazy loading stayed at 3–11 seconds no matter dimension, an 8x–46x enchancment. The larger the picture, the extra minutes you save.

Outcome 2: The saving holds on each occasion sort and dimension

Operating the 13.2 GB picture throughout 13 occasion sort/dimension mixtures, commonplace obtain ranged from 3 min 16 s to 4 min 20 s, whereas lazy loading held at 6-8 s. That’s about 34x quicker on each configuration. You don’t want to alter occasion sort or dimension to learn.

Determine 2: Obtain time for a 13.2 GB picture throughout 13 occasion sort and dimension mixtures

Outcome 3: Coaching efficiency is unaffected past a first-epoch penalty

We ran two sorts of full coaching jobs from the Docker-pulled and from the SOCI-indexed picture and in contrast each epoch:

• Picture classification on ResNet-50 on Imagenette (9,469 pictures)
  • Container pictures: PyTorch (13.2 GB DLC) and TensorFlow (9.9 GB DLC)
  • Cases: ml.p5.48xlarge, ml.p4d.24xlarge, ml.g7e.48xlarge, ml.g7.48xlarge, ml.g6e.12xlarge, ml.g6e.24xlarge, and ml.g6e.48xlarge.
• Massive language mannequin (LLM) fine-tuning: full-parameter supervised fine-tuning (SFT) of Qwen2.5-0.5B on alpaca-cleaned (51,760 examples) with the Hugging Face Coach.
  • Container picture: 9.4 GB Hugging Face PyTorch DLC.
  • Cases: ml.p5.48xlarge, ml.p4d.24xlarge, ml.g6.48xlarge.

The next chart breaks job startup into picture obtain time and first coaching epoch time, evaluating a normal Docker pull with a SOCI-indexed picture throughout GPU and CPU configurations for each workloads.

Determine 3: Job startup cut up into picture obtain time and first coaching epoch time, commonplace Docker pull in comparison with a SOCI-indexed picture

• From the second epoch on, time per epoch is identical (inside run-to-run variation) on each occasion, for each workloads.
• The primary epoch takes 3–16 s longer from the lazy-loaded picture, whereas recordsdata the framework touches for the primary time are fetched. This primary-epoch value is identical on each occasion dimension.
• Internet: every job completed 1-6 min sooner. That’s the obtain saving minus the one-time first-epoch value, a set saving per job.

Take a look at this by yourself workload

These numbers are from the 2 workloads described earlier. The primary-epoch value depends upon your picture and what your code touches at startup:

• Lazy loading wins when the primary coaching step wants solely a fraction of the picture, the remaining streams in when you practice.
• In case your entry level reads a lot of the picture earlier than coaching begins (for instance, scanning, checksumming, or preloading its contents), the primary epoch waits for roughly the identical bytes a full pull would have fetched. The web enchancment can be small.
• Entrance-loaded begins nonetheless got here out forward in our exams:
  • torch.compile and a just-in-time (JIT) constructed CUDA extension added 7-15 s to the primary epoch (web saving nonetheless 3 min 25 s–3 min 56 s)
  • A 6 GB mannequin baked into the picture streamed its weights at load time, including 2-3.5 min, however the general job nonetheless completed 2-2.5 minutes sooner on the listed picture.

You’ll be able to undertake SOCI indexing two methods: use a pre-indexed AWS Deep Studying Container or create an index in your personal customized picture.

Possibility 1: Use a pre-indexed AWS Deep Studying Container

AWS publishes SOCI-indexed variants of a number of common Deep Studying Containers, recognized by the -soci suffix on the picture tag:

763104351884.dkr.ecr.us-east-1.amazonaws.com/pytorch-training:2.9-gpu-py312-cu130-ubuntu22.04-sagemaker-v1-soci

Pre-indexed variants can be found for PyTorch Coaching, TensorFlow Coaching, vLLM, and SGLang, together with arm64 variants for some frameworks. To undertake lazy loading, replace the picture tag in your job definition to the -soci variant. Browse the complete record within the AWS Deep Studying Containers Gallery.

Word: If you happen to construct your individual picture FROM a DLC base picture, it’s essential to create a SOCI index in your remaining picture. The index from the bottom picture doesn’t carry ahead to derived pictures.

Possibility 2: Create a SOCI index in your personal picture

To index a customized picture, you want a Linux host (comparable to an Amazon Elastic Compute Cloud (Amazon EC2) occasion operating Amazon Linux 2023) with containerd, the SOCI snapshotter, and both Finch or nerdctl put in.

1. Set your surroundings variables.

   ACCOUNT_ID="111122223333"
   REGION="us-east-1"
   REPOSITORY_NAME="my-training-image"
   ORIGINAL_IMAGE_TAG="v1.0"
   SOCI_IMAGE_TAG="v1.0-soci"
   REGISTRY="$ACCOUNT_ID.dkr.ecr.$REGION.amazonaws.com"

2. Set up the required instruments.

   sudo dnf set up -y containerd fuse soci-snapshotter runfinch-finch
   sudo systemctl begin soci-snapshotter
   sudo systemctl restart containerd

3. Authenticate to Amazon ECR.

   aws ecr get-login-password --region "$REGION" 
     | sudo nerdctl login -u AWS --password-stdin "$REGISTRY"

4. Pull your unique picture.

   sudo nerdctl pull "$REGISTRY/$REPOSITORY_NAME:$ORIGINAL_IMAGE_TAG"

5. Create the SOCI-indexed picture.

   sudo nerdctl picture convert --soci 
     "$REGISTRY/$REPOSITORY_NAME:$ORIGINAL_IMAGE_TAG" 
     "$REGISTRY/$REPOSITORY_NAME:$SOCI_IMAGE_TAG"

6. Push the listed picture.

   sudo nerdctl push --platform linux/amd64 
     "$REGISTRY/$REPOSITORY_NAME:$SOCI_IMAGE_TAG"

This creates two artifacts in your repository: the SOCI index and an OCI-compliant picture index manifest linking your picture to the index. Your unique picture and its digest stay unchanged.

Automate index creation in CI/CD

To generate indexes robotically on each push, deploy the CFN ECR SOCI Index Builder. It makes use of an Amazon EventBridge rule to detect Amazon ECR picture push occasions and construct the index. No adjustments to your construct pipeline are required.

Reference your SOCI-indexed picture precisely as you’ll every other picture. No API adjustments are wanted. Solely the tag differs.

Coaching instance (SageMaker Python SDK)

from sagemaker.practice import ModelTrainer
from sagemaker.practice.configs import Compute

coach = ModelTrainer(
    training_image="111122223333.dkr.ecr.us-east-1.amazonaws.com/my-training-image:v1.0-soci",
    position="arn:aws:iam::111122223333:position/SageMakerExecutionRole",
    compute=Compute(
        instance_type="ml.g5.12xlarge",
        instance_count=2,
    ),
)

coach.practice()

Confirm that your picture features a SOCI index

Earlier than launching, verify your tag resolves to an listed picture:

aws ecr batch-get-image 
  --repository-name "$REPOSITORY_NAME" 
  --image-ids imageTag="$SOCI_IMAGE_TAG" 
  --accepted-media-types 
  "software/vnd.oci.picture.index.v1+json" 
  --query 'pictures[0].imageManifest' --output textual content 
  | jq '.manifests[] | {mediaType, artifactType}'

Look for a kid manifest whose artifactType begins with software/vnd.amazon.soci.index. alongside your platform manifest. If it isn’t there, the tag doesn’t level to an listed picture.

• Amazon ECR personal repositories solely. Pictures in different registries, or pulled by means of a personal Docker registry (TrainingRepositoryAccessMode set to Vpc), should not eligible for lazy loading.
• Enchancment scales with picture dimension. Pictures smaller than roughly 4 GB see comparatively small enhancements.
• Examine for Docker-specific assumptions. SOCI-indexed pictures run on a containerd-based runtime. Assessment entry factors and setup scripts for code that calls the Docker CLI, reads /var/run/docker.sock, or derives identifiers from cgroup paths. SageMaker AI mounts a /.dockerenv marker for backward compatibility, however keep away from including new dependencies on it.
• SOCI indexing for SageMaker AI jobs is obtainable throughout all AWS Areas.

With SOCI indexing, you may reduce container picture obtain time out of your SageMaker AI coaching and processing jobs. Add a SOCI index to your pictures, or change to a pre-indexed AWS Deep Studying Container. Both manner, you chop startup time from minutes to seconds, cut back complete job value, and speed up your machine studying (ML) iteration cycle. And your present job definitions, API calls, and information pipelines keep precisely the identical.

To be taught extra about Amazon SageMaker AI and begin utilizing SOCI indexing, discover the next assets:

1. Choose a SOCI-indexed Deep Studying Container. Browse the Deep Studying Containers Gallery and append -soci to your picture tag. For framework-specific steering, see SOCI index for AWS Deep Studying Containers.
2. Index your individual customized picture. Observe the step-by-step walkthrough within the SageMaker AI Developer Information, or consult with the SOCI snapshotter getting-started information for superior choices.
3. Automate indexing in your CI/CD pipeline. Deploy the CFN ECR SOCI Index Builder to generate indexes robotically on each picture push. No pipeline adjustments are required.
4. Take a look at domestically with Finch. Use the Finch lazy-loading walkthrough to validate your picture on a containerd-based runtime earlier than you submit a job.
5. Dive deeper into the know-how. Learn Decreasing container chilly begin instances utilizing SOCI index on DLAMI and DLC on the AWS Machine Studying Weblog for benchmarks, or discover Below the hood: Lazy loading container pictures with Seekable OCI and AWS Fargate on the AWS Containers Weblog to grasp the lazy-loading mechanism intimately.

Quicker container startup is just one picture tag away. Attempt it in your subsequent SageMaker AI job and see the distinction.

In regards to the authors