💥 Welcome to Data on Amazon EKS (DoEKS) 💥
Data on Amazon EKS(DoEKS) is a tool for users to build aws managed and self-managed scalable data platforms on Amazon EKS. This repo provides Infrastructure as Code(IaC) templates(e.g., Terraform, AWS CDK etc.), sample Apache Spark/ML jobs, references to AWS Data blogs, Performance Benchmark reports and Best Practices for deploying Data Solutions on Amazon EKS.
Note: Data on EKS is under active development for number of patterns. Please refer to the issues section to see the work in progress features.
Data on EKS(DoEKS) solution is categorized into the following areas.
🎯 Data Analytics on EKS
🎯 AI/ML on EKS
🎯 Distributed Databases on EKS
🎯 Streaming Platforms on EKS
🎯 Scheduler Workflow Platforms on EKS
In this repository you will find multiple deployment examples for bootstrapping Data platforms with Amazon EKS Cluster and the Kubernetes add-ons.
🚀 EMR on EKS with Apache YuniKorn - EMR on EKS with Apache YuniKorn custom batch scheduler.
🚀 EMR on EKS with Karpenter - Scaling Spark jobs with EMR on EKS and Karpenter.
🚀 EMR on EKS with FSx for Lustre - EMR on EKS Cluster with FSx for Lustre as a high performance Spark shuffle storage.
🚀 Spark Operator on EKS - Spark Operator with Apache YuniKorn on Amazon EKS.
🚀 Amazon Manged Workflows for Apache Airflow (MWAA) - Trigger EMR on EKS jobs with Amazon Managed Workflows for Apache Airflow (MWAA).
🚀 Self-managed Airflow on EKS - Deploy self-managed Apache Airflow with best practices on Amazon EKS.
🚀 Ray on EKS - Deploy Ray Operator on EKS with sample scripts.
Checkout the DoEKS Website for instructions to deploy the Data on EKS patterns and run sample tests.
Kubernetes is the most widely known system for large-scale orchestration of containerized software. It became more mature for running stateful workloads with the introduction of several storage options in version 1.19. In addition, with an introduction of Spark on Kubernetes and the flexibility that Kubernetes offers have motivated many users to migrate their existing Hadoop based clusters to Kubernetes.
Deploying and managing Kubernetes clusters and scaling data workloads is still challenging for many users because they are expected to be familiar with Kubernetes and data workloads. To address this, we chose to launch this new Data on EKS (DoEKS) tool to help simplify the journey for the users who want to run Spark on EKS, Kubeflow, MLFlow, Airflow, Presto, Kafka, Cassandra etc. or any other data workloads.
Data on EKS(DoEKS) is maintained by AWS Solution Architects. It is not part of an AWS service, and support is provided best effort by the Data on EKS Blueprints community.
Please use the Issues section of this GitHub to post feedback, submit feature ideas, or report bugs.
See CONTRIBUTING for more information.
This library is licensed under the Apache 2.0 License.
We invite everyone who is passionate about data on Kubernetes to join this initiative.
Built with ❤️ at AWS.