Use Amazon SageMaker Studio with a customized file system in Amazon EFS

Amazon SageMaker Studio is the most recent web-based expertise for working end-to-end machine studying (ML) workflows. SageMaker Studio affords a collection of built-in growth environments (IDEs), which incorporates JupyterLab, Code Editor, in addition to RStudio. Information scientists and ML engineers can spin up SageMaker Studio non-public and shared areas, that are used to handle the storage and useful resource wants of the JupyterLab and Code Editor functions, allow stopping the functions when not in use to save lots of on compute prices, and resume the work from the place they stopped.

The storage assets for SageMaker Studio areas are Amazon Elastic Block Retailer (Amazon EBS) volumes, which provide low-latency entry to consumer information like notebooks, pattern information, or Python/Conda digital environments. Nevertheless, there are a number of situations the place utilizing a distributed file system shared throughout non-public JupyterLab and Code Editor areas is handy, which is enabled by configuring an Amazon Elastic File System (Amazon EFS) file system in SageMaker Studio. Amazon EFS gives a scalable absolutely managed elastic NFS file system for AWS compute cases.

Amazon SageMaker helps mechanically mounting a folder in an EFS quantity for every consumer in a site. Utilizing this folder, customers can share information between their very own non-public areas. Nevertheless, customers can’t share information with different customers within the area; they solely have entry to their very own folder user-default-efs within the $HOME listing of the SageMaker Studio software.

On this submit, we discover three distinct situations that reveal the flexibility of integrating customized Amazon EFS with SageMaker Studio.

For additional data on configuring Amazon EFS in SageMaker Studio, check with Attaching a customized file system to a site or consumer profile.

Answer overview

Within the first situation, an AWS infrastructure admin needs to arrange an EFS file system that may be shared throughout the non-public areas of a given consumer profile in SageMaker Studio. Because of this every consumer inside the area could have their very own non-public house on the EFS file system, permitting them to retailer and entry their very own information and recordsdata. The automation described on this submit will allow new workforce members becoming a member of the information science workforce can rapidly arrange their non-public house on the EFS file system and entry the required assets to begin contributing to the continuing challenge.

The next diagram illustrates this structure.

This situation affords the next advantages:

Particular person information storage and evaluation – Customers can retailer their private datasets, fashions, and different recordsdata of their non-public areas, permitting them to work on their very own initiatives independently. Segregation is made by their consumer profile.
Centralized information administration – The administrator can handle the EFS file system centrally, sustaining information safety, backup, and direct entry for all customers. By organising an EFS file system with a non-public house, customers can effortlessly observe and preserve their work.
Cross-instance file sharing – Customers can entry their recordsdata from a number of SageMaker Studio areas, as a result of the EFS file system gives a persistent storage resolution.

The second situation is said to the creation of a single EFS listing that’s shared throughout all of the areas of a given SageMaker Studio area. Because of this all customers inside the area can entry and use the identical shared listing on the EFS file system, permitting for higher collaboration and centralized information administration (for instance, to share widespread artifacts). This can be a extra generic use case, as a result of there isn’t a particular segregated folder for every consumer profile.

The next diagram illustrates this structure.

This situation affords the next advantages:

Shared challenge directories – Suppose the information science workforce is engaged on a large-scale challenge that requires collaboration amongst a number of workforce members. By organising a shared EFS listing at challenge degree, the workforce can collaborate on the identical initiatives by accessing and dealing on recordsdata within the shared listing. The info science workforce can, for instance, use the shared EFS listing to retailer their Jupyter notebooks, evaluation scripts, and different project-related recordsdata.
Simplified file administration – Customers don’t must handle their very own non-public file storage, as a result of they’ll depend on the shared listing for his or her file-related wants.
Improved information governance and safety – The shared EFS listing, being centrally managed by the AWS infrastructure admin, can present improved information governance and safety. The admin can implement entry controls and different information administration insurance policies to keep up the integrity and safety of the shared assets.

The third situation explores the configuration of an EFS file system that may be shared throughout a number of SageMaker Studio domains inside the similar VPC. This enables customers from completely different domains to entry and work with the identical set of recordsdata and information, enabling cross-domain collaboration and centralized information administration.

The next diagram illustrates this structure.

This situation affords the next advantages:

Enterprise-level information science collaboration – Think about a big group with a number of information science groups engaged on numerous initiatives throughout completely different departments or enterprise items. By organising a shared EFS file system accessible throughout the group’s SageMaker Studio domains, these groups can collaborate on cross-functional initiatives, share artifacts, and use a centralized information repository for his or her work.
Shared infrastructure and assets – The EFS file system can be utilized as a shared useful resource throughout a number of SageMaker Studio domains, selling effectivity and cost-effectiveness.
Scalable information storage – Because the variety of customers or domains will increase, the EFS file system mechanically scales to accommodate the rising storage and entry necessities.
Information governance – The shared EFS file system, being managed centrally, may be topic to stricter information governance insurance policies, entry controls, and compliance necessities. This may also help the group meet regulatory and safety requirements whereas nonetheless enabling cross-domain collaboration and information sharing.

Conditions

This submit gives an AWS CloudFormation template to deploy the primary assets for the answer. Along with this, the answer expects that the AWS account during which the template is deployed already has the next configuration and assets:

Confer with Attaching a customized file system to a site or consumer profile for extra conditions.

Configure an EFS listing shared throughout non-public areas of a given consumer profile

On this situation, an administrator needs to provision an EFS file system for all customers of a SageMaker Studio area, creating a non-public file system listing for every consumer. We are able to distinguish two use instances:

Create new SageMaker Studio consumer profiles – A brand new workforce member joins a preexisting SageMaker Studio area and needs to connect a customized EFS file system to the JupyterLab or Code Editor areas
Use preexisting SageMaker Studio consumer profiles – A workforce member is already engaged on a particular SageMaker Studio area and needs to connect a customized EFS file system to the JupyterLab or Code Editor areas

The answer offered on this submit focuses on the primary use case. We talk about adapt the answer for preexisting SageMaker Studio area consumer profiles later on this submit.

The next diagram illustrates the high-level structure of the answer.

On this resolution, we use CloudTrail, Amazon EventBridge, and Lambda to mechanically create a non-public EFS listing when a brand new SageMaker Studio consumer profile is created. The high-level steps to arrange this structure are as follows:

Create an EventBridge rule that invokes the Lambda operate when a brand new SageMaker consumer profile is created and logged in CloudTrail.
Create an EFS file system with an entry level for the Lambda operate and with a mount goal in each Availability Zone that the SageMaker Studio area is positioned.
Use a Lambda operate to create a non-public EFS listing with the required POSIX permissions for the profile. The operate will even replace the profile with the brand new file system configuration.

Deploy the answer utilizing AWS CloudFormation

To make use of the answer, you possibly can deploy the infrastructure utilizing the next CloudFormation template. This template deploys three predominant assets in your account: Amazon EFS assets (file system, entry factors, mount targets), an EventBridge rule, and a Lambda operate.

Confer with Create a stack from the CloudFormation console for extra data. The enter parameters for this template are:

SageMakerDomainId – The SageMaker Studio area ID that will probably be related to the EFS file system.
SageMakerStudioVpc – The VPC related to the SageMaker Studio area.
SageMakerStudioSubnetId – One or a number of subnets related to the SageMaker Studio area. The template deploys its assets in these subnets.
SageMakerStudioSecurityGroupId – The safety group related to the SageMaker Studio area. The template configures the Lambda operate with this safety group.

Amazon EFS assets

After you deploy the template, navigate to the Amazon EFS console and ensure that the EFS file system has been created. The file system has a mount goal in each Availability Zone that your SageMaker area connects to.

Notice that every mount goal makes use of the EC2 safety group that SageMaker created in your AWS account whenever you first created the area, which permits NFS visitors at port 2049. The offered template mechanically retrieves this safety group when it’s first deployed, utilizing a Lambda backed customized useful resource.

You can too observe that the file system has an EFS entry level. This entry level grants root entry on the file system for the Lambda operate that can create the directories for the SageMaker Studio consumer profiles.

EventBridge rule

The second predominant useful resource is an EventBridge rule invoked when a brand new SageMaker Studio consumer profile is created. Its goal is the Lambda operate that creates the folder within the EFS file system and updates the profile that has been simply created. The enter of the Lambda operate is the occasion matched, the place you will get the SageMaker Studio area ID and the SageMaker consumer profile title.

Lambda operate

Lastly, the template creates a Lambda operate that creates a listing within the EFS file system with the required POSIX permissions for the consumer profile and updates the consumer profile with the brand new file system configuration.

At a POSIX permissions degree, you possibly can management which customers can entry the file system and which recordsdata or information they’ll entry. The POSIX consumer and group ID for SageMaker apps are:

UID – The POSIX consumer ID. The default is 200001. A legitimate vary is a minimal worth of 10000 and most worth of 4000000.
GID – The POSIX group ID. The default is 1001. A legitimate vary is a minimal worth of 1001 and most worth of 4000000.

The Lambda operate is in the identical VPC because the EFS file system and it has connected the file system and entry level beforehand created.

Adapt the answer for preexisting SageMaker Studio area consumer profiles

We are able to reuse the earlier resolution for situations during which the area already has consumer profiles created. For that, you possibly can create a further Lambda operate in Python that lists all of the consumer profiles for the given SageMaker Studio area and creates a devoted EFS listing for every consumer profile.

The Lambda operate needs to be in the identical VPC because the EFS file system and it has connected the file system and entry level beforehand created. You’ll want to add the efs_id and domain_id values as atmosphere variables for the operate.

You may embody the next code as a part of this new Lambda operate and run it manually:

import json
import subprocess
import boto3
import os

sm_client = boto3.consumer('sagemaker')

def lambda_handler(occasion, context):
    
    # Get EFS and Area ID
    file_system=os.environ['efs_id']
    domain_id=os.environ['domain_id']    
    
    
    # Get Area consumer profiles
    list_user_profiles_response = sm_client.list_user_profiles(
        DomainIdEquals=domain_id
    )
    domain_users = list_user_profiles_response["UserProfiles"]
    
    # Create directories for every consumer
    for consumer in domain_users:

        user_profile_name = consumer["UserProfileName"]

        # Permissions
        repository=f'/mnt/efs/{user_profile_name}'
        subprocess.name(['mkdir', repository])
        subprocess.name(['chown', '200001:1001', repository])
        
        # Replace SageMaker consumer
        response = sm_client.update_user_profile(
            DomainId=domain_id,
            UserProfileName=user_profile_name,
            UserSettings={
                'CustomFileSystemConfigs': [
                    {
                        'EFSFileSystemConfig': {
                            'FileSystemId': file_system,
                            'FileSystemPath': f'/{user_profile_name}'
                        }
                    }
                ]
            }
        )

Configure an EFS listing shared throughout all areas of a given area

On this situation, an administrator needs to provision an EFS file system for all customers of a SageMaker Studio area, utilizing the identical file system listing for all of the customers.

To attain this, along with the conditions described earlier on this submit, you could full the next steps.

Create the EFS file system

The file system must be in the identical VPC because the SageMaker Studio area. Confer with Creating EFS file methods for extra data.

Add mount targets to the EFS file system

Earlier than SageMaker Studio can entry the brand new EFS file system, the file system will need to have a mount goal in every of the subnets related to the area. For extra details about assigning mount targets to subnets, see Managing mount targets. You will get the subnets related to the area on the SageMaker Studio console below Community. You’ll want to create a mount goal for every subnet.

Moreover, for every mount goal, you should add the safety group that SageMaker created in your AWS account whenever you created the SageMaker Studio area. The safety group title has the format security-group-for-inbound-nfs-domain-id.

The next screenshot reveals an instance of an EFS file system with two mount targets for a SageMaker Studio area related to 2 subnets. Notice the safety group related to each mount targets.

Create an EFS entry level

The Lambda operate accesses the EFS file system as root utilizing this entry level. See Creating entry factors for extra data.

Create a brand new Lambda operate

Outline a brand new Lambda operate with the title LambdaManageEFSUsers. This operate updates the default house settings of the SageMaker Studio area, configuring the file system settings to make use of a particular EFS file system shared repository path. This configuration is mechanically utilized to all areas inside the area.

The Lambda operate is in the identical VPC because the EFS file system and it has connected the file system and entry level beforehand created. Moreover, you could add efs_id and domain_id as atmosphere variables for the operate.

UID – The POSIX consumer ID. The default is 200001.
GID – The POSIX group ID. The default is 1001.

The operate updates the default house settings of the SageMaker Studio area, configuring the EFS file system for use by all customers. See the next code:

import json
import subprocess
import boto3
import os
import logging

logger = logging.getLogger()
logger.setLevel(logging.INFO)
sm_client = boto3.consumer('sagemaker')

def lambda_handler(occasion, context):
    
    # Atmosphere variables
    file_system=os.environ['efs_id']
    domain_id=os.environ['domain_id']
    
    # EFS listing title
    repository_name="shared_repository"
    repository=f'/mnt/efs/{repository_name}'
            
    # Add permissions to the brand new listing
    attempt:
        subprocess.name(['mkdir -p', repository])
        subprocess.name(['chown', '200001:1001', repository])
    besides:
        print("Repository already created")
    
    # Replace Sagemaker area to allow entry to the brand new listing
    response = sm_client.update_domain(
        DomainId=domain_id,
        DefaultUserSettings={
            'CustomFileSystemConfigs': [
                {
                    'EFSFileSystemConfig': {
                        'FileSystemId': file_system,
                        'FileSystemPath': f'/{repository_name}'
                    }
                }
            ]
        }
    )
    logger.information(f"Up to date Studio Area {domain_id} and EFS {file_system}")
    return {
        'statusCode': 200,
        'physique': json.dumps(f"Created dir and modified permissions for Studio Area {domain_id}")
    }

The execution position of the Lambda operate must have permissions to replace the SageMaker Studio area:

{ 
"Model": "2012-10-17",
    "Assertion": [ 
        { 
        "Effect": "Allow", 
        "Action": [
            "sagemaker:UpdateDomain"
        ],
        "Useful resource": "*" 
        } 
    ]
}

Configure an EFS listing shared throughout a number of domains below the identical VPC

On this situation, an administrator needs to provision an EFS file system for all customers of a number of SageMaker Studio domains, utilizing the identical file system listing for all of the customers. The thought on this case is to assign the identical EFS file system to all customers of all domains which might be inside the similar VPC. To check the answer, the account ought to ideally have two SageMaker Studio domains contained in the VPC and subnet.

Create the EFS file system, add mount targets, and create an entry level

Full the steps within the earlier part to arrange your file system, mount targets, and entry level.

Create a brand new Lambda operate

Outline a Lambda operate referred to as LambdaManageEFSUsers. This operate is liable for automating the configuration of SageMaker Studio domains to make use of a shared EFS file system inside a particular VPC. This may be helpful for organizations that wish to present a centralized storage resolution for his or her ML initiatives throughout a number of SageMaker Studio domains. See the next code:

import json
import subprocess
import boto3
import os
import sys

import logging
logger = logging.getLogger()
logger.setLevel(logging.INFO)

sm_client = boto3.consumer('sagemaker')

def lambda_handler(occasion, context):
    
    #Atmosphere variables
    event_domain_id =occasion["domain_id"]
    file_system=os.environ['efs_id']
    env_vpc_id =os.environ['vpc_id']
    
    #Occasion parameters 
    repository_name="shared_repository"
    repository=f'/mnt/efs/{repository_name}'
    domains =[]    

    # Checklist all SageMaker domains within the specified VPC
    response = sm_client.list_domains()
    all_domains = response['Domains']
    for area in all_domains:
        domain_id =area["DomainId"]
        information =sm_client.describe_domain(DomainId=domain_id)
        domain_vpc_id = information['VpcId']
        if domain_vpc_id ==env_vpc_id:
            domains.append(domain_id)
    
    # Create listing and add the permission
    attempt:
        subprocess.name(['mkdir -p', repository])
        subprocess.name(['chown', '200001:1001', repository])
    besides:
        print("Repository already created")
    
    #Replace Sagemaker area
    if len(domains)>0:
        for domain_id in domains: 
            response = sm_client.update_domain(
                DomainId=event_domain_id,
                DefaultUserSettings={
                    'CustomFileSystemConfigs': [
                        {
                            'EFSFileSystemConfig': {
                                'FileSystemId': file_system,
                                'FileSystemPath': f'/{repository_name}'
                            }
                        }
                    ]
                }
            )
   
        logger.information(f"Up to date Studio for Domains {domains} and EFS {file_system}")
        return {
                'statusCode': 200,
                'physique': json.dumps(f"Created dir and modified permissions for Domains {domains}")
            }
    
    else:
        return {
            'statusCode': 400,
            'physique': json.dumps(f"VPC id of all of the domains {domain_vpc} is completely different than the vpc id configured {env_vpc_id}")
        }

The execution position of the Lambda operate must have permissions to explain and replace the SageMaker Studio area:

{ 
"Model": "2012-10-17",
    "Assertion": [ 
        { 
        "Effect": "Allow", 
        "Action": [
            "sagemaker:DescribeDomain",
            "sagemaker:UpdateDomain"
        ],
        "Useful resource": "*" 
        } 
    ]
}

Clear up

To wash up the answer you applied and keep away from additional prices, delete the CloudFormation template you deployed in your AWS account. If you delete the template, you additionally delete the EFS file system and its storage. For extra data, check with Delete a stack from the CloudFormation console.

Conclusion

On this submit, now we have explored three situations demonstrating the flexibility of integrating Amazon EFS with SageMaker Studio. These situations spotlight how Amazon EFS can present a scalable, safe, and collaborative information storage resolution for information science groups.

The primary situation targeted on configuring an EFS listing with non-public areas for particular person consumer profiles, permitting customers to retailer and entry their very own information whereas the administrator manages the EFS file system centrally.

The second situation showcased a shared EFS listing throughout all areas inside a SageMaker Studio area, enabling higher collaboration and centralized information administration.

The third situation explored an EFS file system shared throughout a number of SageMaker Studio domains, empowering enterprise-level information science collaboration and selling environment friendly use of shared assets.

By implementing these Amazon EFS integration situations, organizations can unlock the complete potential of their information science groups, enhance information governance, and improve the general effectivity of their data-driven initiatives. The mixing of Amazon EFS with SageMaker Studio gives a flexible platform for information science groups to thrive within the evolving panorama of ML and AI.

Concerning the Authors

Irene Arroyo Delgado is an AI/ML and GenAI Specialist Options Architect at AWS. She focuses on bringing out the potential of generative AI for every use case and productionizing ML workloads, to attain clients’ desired enterprise outcomes by automating end-to-end ML lifecycles. In her free time, Irene enjoys touring and mountaineering.

Itziar Molina Fernandez is an AI/ML Advisor within the AWS Skilled Companies workforce. In her position, she works with clients constructing large-scale machine studying platforms and generative AI use instances on AWS. In her free time, she enjoys exploring new locations.

Matteo Amadei is a Information Scientist Advisor within the AWS Skilled Companies workforce. He makes use of his experience in synthetic intelligence and superior analytics to extract helpful insights and drive significant enterprise outcomes for patrons. He has labored on a variety of initiatives spanning NLP, laptop imaginative and prescient, and generative AI. He additionally has expertise with constructing end-to-end MLOps pipelines to productionize analytical fashions. In his free time, Matteo enjoys touring and studying.

Giuseppe Angelo Porcelli is a Principal Machine Studying Specialist Options Architect for Amazon Internet Companies. With a number of years of software program engineering and an ML background, he works with clients of any dimension to grasp their enterprise and technical wants and design AI and ML options that make the very best use of the AWS Cloud and the Amazon Machine Studying stack. He has labored on initiatives in numerous domains, together with MLOps, laptop imaginative and prescient, and NLP, involving a broad set of AWS providers. In his free time, Giuseppe enjoys enjoying soccer.

Use Amazon SageMaker Studio with a customized file system in Amazon EFS

All You Have to Know In regards to the Non-Inferiority Speculation Check | by Prateek Jain | Oct, 2024

Visualization of Information with Pie Charts in Matplotlib | by Diana Rozenshteyn | Oct, 2024

Visualization of Information with Pie Charts in Matplotlib | by Diana Rozenshteyn | Oct, 2024

Leave a Reply Cancel reply

Popular News

Unlocking Japanese LLMs with AWS Trainium: Innovators Showcase from the AWS LLM Growth Assist Program

Speed up migration portfolio evaluation utilizing Amazon Bedrock

Proton launches ‘Privacy-First’ AI Email Assistant to Compete with Google and Microsoft

Rethinking the Function of PPO in RLHF – The Berkeley Synthetic Intelligence Analysis Weblog

Streamlit fairly styled dataframes half 1: utilizing the pandas Styler

About Us

Category

Recent Posts