Construct AI workflows on Amazon EKS with Union.ai and Flyte

As synthetic intelligence and machine studying (AI/ML) workflows develop in scale and complexity, it turns into more durable for practitioners to prepare and deploy their fashions. AI tasks usually battle to maneuver from pilot to manufacturing. AI tasks usually fail not as a result of fashions are dangerous, however as a result of infrastructure and processes are fragmented and brittle, and the unique pilot code base is usually compelled to bloat by these extra necessities. This makes it tough for knowledge scientists and engineers to rapidly transfer from laptop computer to cluster (native growth to manufacturing deployment) and reproduce the precise outcomes they’d seen through the pilot.

On this put up, we clarify how you should utilize the Flyte Python SDK to orchestrate and scale AI/ML workflows. We discover how the Union.ai 2.0 system permits deployment of Flyte on Amazon Elastic Kubernetes Service (Amazon EKS), integrating seamlessly with AWS companies like Amazon Easy Storage Service (Amazon S3), Amazon Aurora, AWS Identification and Entry Administration (IAM), and Amazon CloudWatch. We discover the answer by means of an AI workflow instance, utilizing the brand new Amazon S3 Vectors service.

Frequent challenges working AI/ML workflows on Kubernetes

AI/ML workflows working on Kubernetes current a number of orchestration challenges:

• Infrastructure complexity – Provisioning the suitable compute sources (CPUs, GPUs, reminiscence) dynamically throughout Kubernetes clusters
• Experiment-to-production hole – Transferring from experimentation to manufacturing usually requires rebuilding pipelines in several environments
• Reproducibility – Monitoring knowledge lineage, mannequin variations, and experiment parameters to facilitate dependable outcomes
• Value administration – Effectively using spot cases, automated scaling, and avoiding over-provisioning
• Reliability – Dealing with failures gracefully with automated retries, checkpointing, and restoration mechanisms

Goal-built AI/ML tooling is important for orchestrating complicated workflows, providing specialised capabilities like clever caching, automated versioning, and dynamic useful resource allocation that streamline growth and deployment cycles.

Why Flyte/Union for Amazon EKS

The Flyte on Amazon EKS Python workflows scale from laptop-to-cluster with dynamic execution, reproducibility, and compute-aware orchestration. These workflows, together with Union.ai’s managed deployment, facilitate seamless, crash-proof operations that absolutely make the most of Amazon EKS with out the infrastructure overhead. Flyte transforms how one can orchestrate AI/ML workloads on Amazon EKS, making workflows easy to construct. Some key components embrace:

• Pure Python workflows – Write orchestration logic in Python with 66% much less code than conventional orchestrators, assuaging the necessity to study domain-specific languages and eradicating limitations for ML engineers and AI builders migrating current code
• Dynamic execution – Make real-time choices at runtime with versatile branching, loops, and conditional logic, which is important for agentic AI methods
• Reproducibility by default – Each execution is versioned, cached, and tracked with full knowledge lineage
• Compute-aware orchestration – Dynamically provision the suitable compute sources for every activity, from CPUs for knowledge processing to GPUs for mannequin coaching
• Robustness – Pipelines can rapidly get well from failures, isolate errors, and handle checkpoints with out guide intervention

Union.ai 2.0 is constructed on Flyte, the open supply, Kubernetes-based workflow orchestration system initially developed at Lyft to energy mission-critical ML methods like ETA prediction, pricing, and mapping. After Flyte was open sourced in 2020 and have become a Linux Basis AI & Knowledge venture, the core engineering crew based Union.ai 2.0 to ship an enterprise-grade service purposed-built for groups working AI/ML workloads on Amazon EKS. Union.ai 2.0 reduces the complexity of managing Kubernetes infrastructure by means of managed operations, a multi-cloud management aircraft, and abstracted infrastructure administration, whereas offering ML-based capabilities that assist knowledge scientists and engineers concentrate on constructing fashions with enhanced scale, velocity, safety, and reliability.

Further advantages of utilizing Union.ai 2.0 embrace:

• Enhanced scalability – Workflows reply at runtime with versatile branching, activity fanout, and real-time infrastructure scaling.
• Crash-proof reliability – Computerized retries, checkpointing, and failure restoration permit workflows to remain resilient with out guide intervention.
• Agentic AI runtime – Union.ai is designed for long-lived agentic AI methods, supporting stateful brokers and really sturdy orchestration.
• Compliance – For regulated industries, built-in lineage, auditability, and safe execution (SOC2, RBAC, SSO) are crucial. Orchestration on Amazon EKS and Union.ai helps facilitate compliance.
• Useful resource consciousness – It affords first-class help for compute provisioning, spot cases, and automated scaling.

The advantages of Flyte and Union.ai 2.0 elevate trendy orchestration to a first-class requirement: dynamic execution, fault tolerance, and useful resource consciousness are actually built-in, offering a extra developer-friendly expertise in comparison with 1.0.

Amazon EKS offers your compute, storage, and networking spine. Flyte (the open supply venture) handles workflow orchestration. Union.ai extends Flyte with infrastructure-aware orchestration, enterprise-grade safety, and turnkey scalability, supplying you with production-ready Flyte with out the DIY setup. Each Flyte and Union.ai 2.0 run on Amazon EKS, however serve totally different wants, as detailed within the following desk.

Enterprises like Woven Toyota, Lockheed Martin, Spotify, and Artera orchestrate tens of millions of {dollars} of compute yearly with Flyte and Union, accelerating experimentation by 25 instances sooner and reducing iteration cycles by 96%.

Each choices (open supply Flyte and Union.ai 2.0) combine with the open supply neighborhood, facilitating fast function rollout and steady enchancment.

Resolution overview

Though open supply Flyte offers highly effective orchestration capabilities, Union.ai 2.0 delivers the identical core expertise with enterprise-grade administration, eradicating the operational overhead so your crew can concentrate on constructing AI functions as an alternative of managing infrastructure. That is achieved by means of a hybrid structure that mixes managed simplicity with full knowledge management. The Regional management aircraft handles workflow metadata and coordination, whereas the Union Operator deploys straight into your EKS clusters—holding your knowledge, code, and secrets and techniques solely inside your AWS perimeter.

The next determine illustrates the operational stream between Union’s management aircraft and your knowledge aircraft. The Union-managed management aircraft (left) orchestrates workflows by means of Elastic Load Balancing (ELB), storing activity knowledge in Amazon S3 and execution metadata in Aurora. Inside your Amazon EKS atmosphere (proper), the information aircraft executes workflows that pull buyer code out of your container registry, entry secrets and techniques from AWS Secrets and techniques Supervisor, and skim/write knowledge to your S3 buckets—with the execution logs flowing to each CloudWatch and the Union management aircraft for observability.

Union.ai 2.0’s AWS integration structure is constructed on six key service parts that present end-to-end workflow administration:

• Management aircraft and knowledge aircraft – The management aircraft operates inside the Union.ai AWS account and serves because the central administration interface, offering customers with authentication and authorization capabilities, commentary and monitoring features, and system administration instruments. It additionally orchestrates execution placement on knowledge aircraft clusters and handles cluster management and administration operations. Union.ai 2.0 maintains one management aircraft per AWS Area, managing the Regional knowledge planes. Out there Areas for knowledge aircraft deployment embrace us-west, us-east, eu-west, and eu-central, with ongoing enlargement to extra Areas.
• Knowledge aircraft object retailer – This part shops knowledge comprising information, directories, knowledge frames, fashions, and Python-pickled varieties, that are handed as references and skim by the management aircraft.
• Container registry – This part incorporates registry knowledge that embrace names of workflows, duties, launch plans, and artifacts; enter and output varieties for workflows and duties; execution standing, begin time, finish time, and length of workflows and duties; model data for workflows, duties, launch plans, and artifacts; and artifact definitions. With the Union.ai 2.0 structure, you possibly can retain full possession of your knowledge and compute sources whereas it manages the infrastructure operations. The Union.ai 2.0 operator resides within the knowledge aircraft and handles administration duties with least privilege permissions. It permits cluster lifecycle operations and offers help engineers with system-level log entry and alter implementation capabilities—with out exposing secrets and techniques or knowledge. Safety is additional strengthened by means of unidirectional communication: the information aircraft operator initiates the connections to the management aircraft, not the reverse.
• Logging and monitoring – CloudWatch offers centralized logging and monitoring by means of deep integration with Flyte. The system routinely builds logging hyperlinks for every execution and shows them within the console, with hyperlinks pointing on to the AWS Administration Console and the particular log stream for that execution—a function that considerably accelerates troubleshooting throughout failures.
• Safety – Safety is dealt with by means of IAM roles for service accounts (IRSA), which maps the id between Kubernetes sources and the AWS companies they rely on. These configurations allow safer, fine-grained entry management for backend companies, and Union.ai 2.0 provides enterprise role-based entry management (RBAC) for consumer entry management on prime of those AWS security measures.
• Storage layer – Amazon S3 serves because the sturdy storage layer for workflows and knowledge. Whenever you register a workflow with Flyte, your code is compiled right into a language-independent illustration that captures the workflow definition, enter, and output varieties. This illustration is packaged and saved in Amazon S3, the place FlytePropeller—Flyte’s execution engine—retrieves it to instruct the respective compute framework (akin to Kubernetes or Spark) to run workflows and report standing. Uncooked enter knowledge used to coach and validate fashions can be saved in Amazon S3. Union.ai 2.0 now features a new integration with Amazon S3 Vectors, enabling vector storage for Retrieval Augmented Technology (RAG), semantic search, and agentic AI workflows.

With this strong infrastructure in place, Union.ai 2.0 on Amazon EKS excels at orchestrating a variety of AI/ML workloads. It handles large-scale mannequin coaching by orchestrating distributed coaching pipelines throughout GPU clusters with automated useful resource provisioning and spot occasion help. For knowledge processing, it may course of petabyte-scale datasets with dynamic parallelism and environment friendly activity fanout, scaling to 100,000 activity fanouts with 50,000 concurrent actions in Union.ai 2.0. Through the use of Union.ai 2.0 and Flyte on Amazon EKS, you possibly can construct and deploy agentic AI methods—long-running, stateful AI brokers that make autonomous choices at runtime. For manufacturing deployments, it helps real-time inference with low-latency mannequin serving, utilizing reusable containers for sub-100 millisecond activity startup instances. All through your complete course of, Union.ai 2.0 offers complete MLOps and mannequin lifecycle administration, automating all the things from experimentation to manufacturing deployment with built-in versioning and rollback capabilities.

These capabilities are exemplified in specialised implementations like distributed coaching on AWS Trainium cases, the place Flyte orchestrates large-scale coaching workloads on Amazon EKS.

Deployment choices for Union.ai 2.0 on Amazon EKS

Union.ai 2.0 and Flyte provide three versatile deployment fashions for Amazon EKS, every balancing managed comfort with operational management. Choose the strategy that most closely fits your crew’s experience, compliance necessities, and growth velocity:

• Union BYOC (absolutely managed) – The quickest path to manufacturing. Union.ai 2.0 manages the infrastructure, upgrades, and scaling whereas your workloads run in your AWS account. This selection is right for groups that wish to focus solely on AI growth reasonably than infrastructure operations.
• Union Self Managed – You possibly can deploy Union.ai 2.0’s managed management aircraft whereas sustaining management of your knowledge and compute sources in your AWS account. This selection combines the advantages of managed companies with knowledge sovereignty and governance necessities.
• Flyte OSS on Amazon EKS – You possibly can deploy and function open supply Flyte straight in your EKS cluster utilizing the AWS Cloud Growth Package (AWS CDK). This selection offers most management and is right for groups with robust Kubernetes experience who wish to customise their deployment. (edited) 

The Amazon EKS Blueprints for AWS CDK Union add-on helps AWS clients deploy, scale, and optimize AI/ML workloads utilizing Union on Amazon EKS. It offers modular infrastructure as code (IaC) AWS CDK templates and curated deployment blueprints for working scalable AI workloads, together with:

• Mannequin coaching and fine-tuning pipelines
• Giant language mannequin (LLM) inference and serving
• Multi-model deployment and administration
• Agentic AI pipeline orchestration

Union.ai 2.0 and Flyte present IaC templates for deploying on Amazon EKS:

• Terraform modules – Preconfigured modules for deploying Flyte on Amazon EKS with finest practices for networking, safety, and observability
• AWS CDK help – AWS CDK constructs for integrating Union into current AWS infrastructure
• GitOps workflows – Assist for Flux and ArgoCD for declarative infrastructure administration

The Union add-on is obtainable by weblog publication, and the Flyte add-on is coming—maintain watching the GitHub repo.

These templates automate the provisioning of EKS clusters, node teams (together with GPU cases), IAM roles, S3 buckets, Aurora databases, and the required Flyte parts.

Conditions

To begin utilizing this answer, you have to have the next conditions:

• An AWS account with acceptable permissions.
• Amazon EKS model on normal help.
• Required IAM roles. Utilizing IAM roles for service accounts, Flyte can map id between the Kubernetes sources and AWS companies it relies on. These configurations are for the backend and don’t intervene with user-control aircraft communication

How Union.ai 2.0 helps Amazon S3 Vectors

As AI functions more and more depend on vector embeddings for semantic search and RAG, Union.ai 2.0 empowers groups with Amazon S3 Vectors integration, simplifying vector knowledge administration at scale. Constructed into Flyte 2.0, this function is obtainable as we speak. Amazon S3 Vectors delivers purpose-built, cost-optimized vector storage for semantic search and AI functions. With Amazon S3 degree elasticity and sturdiness for storing vector datasets with subsecond question efficiency, Amazon S3 Vectors is right for functions that have to construct and develop vector indexes at scale. Union.ai 2.0 offers help for Amazon S3 Vectors for RAG, semantic search, and multi-agent methods. In case you’re utilizing Union.ai 2.0 as we speak with Amazon S3 as your object retailer, you can begin utilizing Amazon S3 Vectors instantly with minimal configuration modifications.

To set it up, use Boto’s devoted APIs to retailer and question vectors. Your Amazon S3 IAM roles are already in place. Simply replace the permissions.

By combining Flyte 2.0’s orchestration with Amazon S3 Vector help, multi-agent buying and selling simulations can scale to a whole lot of brokers that study from historic knowledge, share business insights, and execute coordinated methods in actual time. These architectural benefits help refined AI functions like multi-agent methods that require each semantic reminiscence and real-time coordination.

To study extra, discuss with the instance use case of a multi-agent buying and selling simulation utilizing Flyte 2.0 with Amazon S3 Vectors. On this instance, you’ll study to construct a buying and selling simulation that includes a number of brokers that signify crew members in a agency, illustrating their interactions, strategic planning, and collaborative buying and selling actions

Contemplate a multi-agent buying and selling simulation the place AI brokers work together, check methods, and repeatedly study from their experiences. For reasonable agent habits, every agent should retain context from earlier interactions, basically constructing a reminiscence of semantic artifacts that inform future choices. The method contains the next steps:

1. After every simulation spherical, embed the agent’s learnings into vector representations utilizing embedding fashions.
2. Retailer embeddings in Amazon S3 utilizing Amazon S3 Vectors with acceptable metadata and tags.
3. Throughout subsequent executions, retrieve related reminiscences utilizing semantic search to floor agent choices in previous expertise.

With Flyte 2.0, your brokers already run in an orchestration-aware atmosphere. Amazon S3 turns into your vector retailer. It’s cheap, quick, and absolutely built-in, assuaging the necessity for separate vector databases. For the steps and related code to implement the multi-agent buying and selling simulation, discuss with the GitHub repo.

In abstract, this structure helps ship measurable benefits for manufacturing AI methods:

• Decreased operational complexity – Consolidate your AI/ML orchestration and vector storage on a single atmosphere, assuaging the necessity to provision, keep, and safe separate vector database infrastructure
• Vital price financial savings – Amazon S3 Vectors delivers considerably decrease storage prices in comparison with purpose-built vector databases, whereas offering subsecond similarity search efficiency at scale
• Zero-friction AWS integration – Use your current Amazon S3 infrastructure, IRSA configuration, and digital non-public cloud (VPC) networking—no extra authentication layers or community configurations are required
• Battle-tested scalability – Construct on the 99.999999999% sturdiness and elastic scalability of Amazon S3 to help vector datasets from gigabytes to petabytes with out re-architecture

Buyer success: Woven by Toyota

Toyota’s autonomous driving arm, Woven by Toyota, confronted challenges orchestrating complicated AI workloads for his or her autonomous driving expertise, requiring petabyte-scale knowledge processing and GPU-intensive coaching pipelines. After outgrowing their open supply Flyte implementation, they migrated to Union.ai’s managed service on AWS in 2023. The affect was transformative: over 20 instances sooner ML iteration cycles, tens of millions of {dollars} in annual price financial savings by means of spot occasion optimization, and hundreds of parallel staff enabling huge scale.

“Union.ai’s wealth of experience has enabled us to focus our efforts on key ADAS-related functionalities, transfer quick, and depend on Union.ai to ship knowledge at scale,”

– Alborz Alavian, Senior Engineering Supervisor at Woven by Toyota.

Learn the complete case research about Woven by Toyota’s migration to Union.ai.

Conclusion

Union.ai and Flyte present the muse for dependable, scalable AI on Amazon EKS to your AI/ML workflows, akin to constructing autonomous methods, coaching LLMs, or orchestrating complicated knowledge pipelines.To get began, select your path:

Concerning the authors

ND Ngoka is Senior Options Architect at AWS with specialised concentrate on AI/ML and storage applied sciences. Guides clients by means of complicated architectural choices, enabling them to construct resilient, scalable options that drive enterprise outcomes.

Samhita Alla is a Senior Options Engineer for Partnerships at Union.ai, the place she leads the technical execution of strategic integrations throughout the AI stack, from distributed coaching and experiment monitoring to knowledge platform integrations. She works carefully with companions and cross-functional groups to judge feasibility, construct production-ready options, and ship technical content material that drives real-world adoption.

Kristy Cook dinner is Head of Partnerships at Union.ai, the place she builds strategic alliances throughout the AI/ML ecosystem targeted on sustained development. Having cast impactful partnerships at Meta, Yahoo, and Neustar she brings deep experience in operationalizing AI options at scale.

Jim Fratantoni is a GenAI Account Supervisor at AWS, targeted on serving to AI startups scale and co-sell with AWS. He’s captivated with working with founders to collectively go to market and drive enterprise buyer success.

Theo Rashid is an Utilized Scientist at Amazon constructing probabilistic machine studying and forecasting fashions. He’s an energetic open supply contributor, and is captivated with open supply tooling throughout the machine studying stack, from probabilistic programming libraries to workflow orchestration. He holds a PhD in Epidemiology and Biostatistics from Imperial School London.

Alex Fabisiak is a Senior Utilized Scientist at Amazon engaged on utilized forecasting and provide chain issues. He focuses on probabilistic and causal modeling as they relate to optimum coverage choices. He holds a PhD in Finance from UCLA.