Automate AIOps with Amazon SageMaker Unified Studio Projects

An automation framework that creates GitHub repositories with AIOps templates when SageMaker Unified Studio projects are created, orchestrating the complete lifecycle from model training to automated endpoint deployment.

Repository Structure

├── README.md                          # This comprehensive guide
├── aiops-seed-code/                   # AI project templates and seed code
│   └── regression/                    # Regression model templates
│       ├── model_build/               # Model training and pipeline code
│       │   ├── .github/workflows/     # Build pipeline GitHub Actions
│       │   ├── ml_pipelines/          # SageMaker pipeline definitions
│       │   └── source_scripts/        # Training, preprocessing, evaluation scripts          
│       └── model_deploy/              # Model deployment and endpoint code
│           ├── .github/workflows/     # Deployment pipeline GitHub Actions
│           ├── config/dev/            # Endpoint configuration files
│           ├── deploy_endpoint/       # Deployment scripts and utilities
│           └── tests/                 # Integration and unit tests                             
└── images                             # Architecture diagrams
└── smus-cdk/                          # CDK infrastructure code
    ├── app.py                         # Main CDK application entry point
    ├── lambda/                        # Lambda function implementations
    │   ├── check-project-status/      # Project status monitoring
    │   ├── create-deploy-repository/  # Repository creation and setup
    │   └── sync-repositories/         # Repository synchronization
    ├── layers/                        # Lambda layer definitions
    │   ├── git-layer/                 # Docker file to build git executable
    │   └── python-layer/              # Python dependencies layer
    └── ml_ops_smus/                   # Core framework implementation
        └── constructs/                # CDK construct definitions

1. Architecture Overview

The SMUS framework implements an event-driven architecture that automates the complete AIOps lifecycle through a sequential workflow, seamlessly connecting SageMaker Unified Studio project creation with production-ready infrastructure.

The first step is configuring SageMaker Unified Studio environment, setting up domains, project profiles, and establishing the foundational infrastructure required for automated project creation and management.
GitHub connections are configured and necessary AWS infrastructure is deployed including EventBridge rules, Step Functions workflows, and Lambda functions that will orchestrate the automated repository setup and deployment processes.
Data scientists log into SageMaker Unified Studio and create a new project by selecting from available project templates defined in the project profile and configuring GitHub integration settings.
Project creation generates a CreateProject event that is captured by EventBridge, triggering a Step Functions workflow that automatically creates and configures both build and deploy repositories in your GitHub organization, complete with template-specific seed code and GitHub Actions workflows.
Code changes are pushed to the build repository or the workflow is manually triggered, causing the GitHub Actions build pipeline to automatically activate, executing environment setup, dependency installation, and pipeline validation.
The build workflow orchestrates the execution of the SageMaker pipeline, which processes data through preprocessing, feature engineering, model training, and evaluation with comprehensive monitoring and logging.
ML pipeline tracking occurs if tracking server is setup, enabling experiment tracking and model lineage management throughout the training process.
Model registration automatically occurs upon successful pipeline completion, registering the trained model in SageMaker Model Registry with detailed metadata, training metrics, and lineage information, initially set to "PendingManualApproval" status.
Data scientists or ML engineers review the model performance metrics and manually approve the model in SageMaker Model Registry, changing its status from "PendingManualApproval" to "Approved".
The model approval event is automatically detected by EventBridge, which invokes a deployment Lambda function that triggers the GitHub Actions deployment workflow in the deploy repository using the workflow_dispatch mechanism.
The deployment workflow retrieves the approved model, applies infrastructure as code definitions using AWS CDK, and provisions or updates a SageMaker endpoint with comprehensive validation, error handling, and rollback capabilities.
The deployed endpoint becomes active and ready to serve real-time predictions, completing the automated journey from project creation to production deployment with full traceability and governance.

2. Prerequisites and Setup

AWS Account Requirements

AWS Account with appropriate IAM permissions for CDK deployment
- Required AWS Permissions: PowerUserAccess managed policy (or equivalent permissions for IAM, Lambda, Step Functions, EventBridge, Secrets Manager, CloudFormation, and S3)
AWS CLI configured with credentials
Access to AWS services: CDK, Lambda, Step Functions, EventBridge, Secrets Manager, IAM, SageMaker, DataZone, Glue, S3

Development Environment

Python: 3.9 or later
AWS CDK CLI: v2.188.0 or later - Follow AWS CDK Getting Started Guide
Docker Desktop: Required for building Lambda layers - Follow Docker Desktop installation guide and ensure Docker is running
Git: For repository operations
Node.js: 18.x or later (for CDK)

GitHub Requirements

GitHub Organization: You need a GitHub organization where build and deploy repositories will be created
Personal Access Token: A GitHub PAT (Classic) with the following permissions needs to be created
- repo (Full control of private repositories)
- workflow (Update GitHub Action workflows)
- write:packages (Upload packages to GitHub Package Registry)

You can find instructions on how to create PAT (Classic) here.

3. Configuration

Update Configuration File

Before deployment, configure the config.py file located in smus-cdk/ml_ops_smus/config.py:

# smus-cdk/ml_ops_smus/config.py configuration parameters:
GitConfig(
    public_smus_aiops_org="smus-test",  # Organization with AIOPs templates 
    public_smus_aiops_org_repo="aiops-test",  # Source repository name
    public_smus_aiops_org_repo_folder="aiops-seed-code",  # Folder containing templates
    public_repo_branch="main",  # Source repository branch
    oidc_role_github_workflow="aiops-smus-github-action",  # GitHub workflow execution role
    private_github_organization="your-github-organization",  # Your GitHub organization
    private_deploy_repo_default_branch="main",  # Branch name for repositories
    github_token_secret_name="ml-ops-smus-github-token",  # Secret name in AWS Secrets Manager
)

Configuration Parameters Details

public_smus_aiops_org: GitHub organization containing the template repositories
public_smus_aiops_org_repo: Repository name containing ML project templates
public_smus_aiops_org_repo_folder: Folder within the template repository containing project templates

oidc_role_github_workflow: # IAM role name that GitHub Actions can assume. The role can be created in one of the following ways:

Automatic Creation: If this role doesn't exist, it will be automatically created by cdk along with the OIDC provider for GitHub.
Use Existing Role: If you have an existing OIDC setup and role, provide the name of your existing role here so that role can be used with github actions.

Manual Creation: If you want to manually create the OIDC setup without using CDK:

Follow AWS Documentation to create OIDC provider and required permissions
Create a role with the following trust relationship:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "arn:aws:iam::<account-id>:oidc-provider/token.actions.githubusercontent.com"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringLike": {
                    "token.actions.githubusercontent.com:sub": "repo:your-github-organization/*"
                }
            }
        }
    ]
}

Attach the following policy to the role:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "sagemaker:CreatePipeline",
                "sagemaker:UpdatePipeline",
                "sagemaker:StartPipelineExecution",
                "sagemaker:DescribePipeline",
                "sagemaker:DescribePipelineExecution",
                "sagemaker:ListPipelineExecutions",
                "sagemaker:AddTags",
                "sagemaker:ListTags",
                "sagemaker:ListProcessingJobs",
                "sagemaker:DescribeProcessingJob",
                "sagemaker:DescribeImageVersion",
                "sagemaker:CreateProcessingJob",
                "sagemaker:ListPipelineExecutionSteps",
                "sagemaker:ListModelPackages",
                "sagemaker:StopProcessingJob",
                "sagemaker:CreateModel",
                "sagemaker:CreateEndpointConfig",
                "sagemaker:CreateEndpoint",
                "sagemaker:DeleteModel",
                "sagemaker:DeleteEndpointConfig",
                "sagemaker:DeleteEndpoint",
                "sagemaker:DescribeModel",
                "sagemaker:DescribeEndpointConfig",
                "sagemaker:DescribeEndpoint",
                "sagemaker:InvokeEndpoint"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [

8000
                "cloudformation:ListStacks",
                "cloudformation:DescribeStacks",
                "cloudformation:CreateStack",
                "cloudformation:UpdateStack",
                "cloudformation:DeleteStack",
                "cloudformation:ListStackResources",
                "cloudformation:DescribeStackEvents",
                "cloudformation:GetTemplateSummary",
                "cloudformation:ValidateTemplate",
                "cloudformation:GetTemplate",
                "cloudformation:CreateChangeSet",
                "cloudformation:DeleteChangeSet",
                "cloudformation:ListChangeSets",
                "cloudformation:DescribeChangeSet",
                "cloudformation:ExecuteChangeSet",
                "cloudformation:SetStackPolicy",
                "cloudformation:DescribeStackResource"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "ecr:CreateRepository",
                "ecr:DeleteRepository",
                "ecr:DescribeRepositories",
                "ecr:PutLifecyclePolicy",
                "ecr:SetRepositoryPolicy"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "ssm:*",
            "Resource": "arn:aws:ssm:*:*:parameter/cdk-bootstrap/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "iam:CreateRole",
                "iam:Get*",
                "iam:Delete*",
                "iam:List*",
                "iam:Put*",
                "iam:Tag*",
                "iam:Attach*",
                "iam:Detach*",
                "iam:Update*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:Put*",
                "s3:Get*",
                "s3:List*",
                "s3:Create*",
                "s3:Delete*"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "sts:AssumeRole",
            "Resource": [
                "arn:aws:iam::*:role/cdk-*",
                "arn:aws:iam::*:role/cdk-gitactions-*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "glue:GetTable",
                "glue:GetDatabase",
                "glue:GetPartitions"
            ],
            "Resource": "*"
        },
        {
            "Effect": "Allow",
            "Action": "iam:PassRole",
            "Resource": "*",
            "Condition": {
                "StringEquals": {
                    "iam:PassedToService": "sagemaker.amazonaws.com"
                }
            }
        },
        {
            "Effect": "Allow",
            "Action": [
                "logs:CreateLogGroup",
                "logs:CreateLogStream",
                "logs:PutLogEvents"
            ],
            "Resource": "*"
        }
    ]
}

Provide the name of this manually created role in the configuration

private_github_organization: Your GitHub organization where build/deploy repos will be created
github_token_secret_name: Name of the secret in AWS Secrets Manager for GitHub token

4. CDK Deployment

Environment Setup

Clone the repository:

git clone https://github.com/aws-samples/sample-smus-aiops.git
cd sample-smus-aiops/smus-cdk

Create and activate virtual environment:

python -m venv .venv
source .venv/bin/activate  # Linux/MacOS

.venv\Scripts\activate  # Windows

Install dependencies:

pip install -r requirements.txt

CDK Bootstrap and Deployment

Verify Docker is running:

docker info

Bootstrap CDK (if not already done):

cdk bootstrap

Synthesize the stack (optional verification):

cdk synth

Deploy the stack:

cdk deploy --require-approval never

Verify Deployment

After successful deployment, verify the following resources are created:

Lambda Functions:
- ai-ops-check-project-status
- ai-ops-sync-repositories
- ai-ops-create-deploy-repo
- MlOpsSmusStack-model-approval-trigger
Step Functions: ml-ops-project-setup
EventBridge Rules:
- ml-ops-smus-datazone-project-rule-v2
- MlOpsSmusStack-model-approval-rule
IAM Role: aiops-smus-github-action
Secrets Manager: ml-ops-smus-github-token
Identity Provider: token.actions.githubusercontent.com

5. Post-Deployment Configuration

Update GitHub Token Secret

Navigate to AWS Secrets Manager Console
Find the secret: ml-ops-smus-github-token
Click "Retrieve secret value" → "Edit"
Update the token field with your GitHub Personal Access Token
Save the changes

Verify IAM Role Trust Relationship

Check that the GitHub workflow role has the correct trust relationship:

aws iam get-role --role-name aiops-smus-github-action --query 'Role.AssumeRolePolicyDocument'

The trust relationship should include:

{
  "StringLike": {
    "token.actions.githubusercontent.com:sub": "repo:your-github-organization/*"
  }
}

Verify EventBridge Rules

Confirm EventBridge rules are active:

aws events describe-rule --name "ml-ops-smus-datazone-project-rule-v2"
aws events describe-rule --name "MlOpsSmusStack-model-approval-rule"

Git Connection Setup

Create SageMaker Unified Studio Domain (if needed)

Navigate to SageMaker Unified Studio Console
Create a new domain or use existing domain following Create a Amazon SageMaker Unified Studio domain - quick setup
Note the domain ID (format: dzd_xxxxxxxxxx)

Configure Git Connection in SageMaker Unified Studio

Before creating projects, you need to set up a Git connection to enable integration between SMUS and your GitHub repository.

Navigate to your SageMaker Unified Studio domain
Go to domain details page → Connections tab
Click "Create Git connection" → Select "GitHub"
Configure connection:
- Connection name: aws-github-smus-connection
- Click "Connect to GitHub"
- You'll be redirected to Authorization page
- Click "Authorize AWS connector for GitHub"
- Select "Install a new app"
Install AWS connector for GitHub:
- Select your target organization (same as private_github_organization in config)
- Repository access: Select "All repositories"
- Click "Install & Authorize" → "Connect"
Enable connection:
- Select your newly created connection
- Click "Enable" to complete setup

Create Custom Project Profile

You need to create a custom project profile that will be used for your ML projects.

Navigate to domain details page
Find "Project profiles" tile → Click "Create"
Configure Project profile:
- Name: regression
  
  Note: The Project profile name must match a folder name inside aiops-seed-code. For this use case, we use 'regression' as our templates are in aiops-seed-code/regression/. You can extend this by adding other use cases (e.g., 'classification', 'forecasting') in aiops-seed-code/, each containing their own model_build and model_deploy templates. During project creation, when a project profile is selected, based on the project profile name, the corresponding template code will be used to set up your build and deploy repositories.
- Creation option: "Create from a template"
- Capabilities: "All capabilities"
Default tooling blueprint deployment settings:
- Keep default Account and region settings
- Git connection: Select aws-github-smus-connection
- Enable: "Users can edit Git connection"
- Authorization: "Users and groups"
Authorization settings:
- Choose: "Selected users and groups"
- Select SSO users and groups who can access this project profile
Project profile readiness: Check "Enable project profile on creation"

6. SageMaker Unified Studio Project Creation

Create SageMaker Project

Navigate to your SMUS domain by clicking "Open unified studio"
Sign in with SSO using the username and password for the user granted access to the project profile
In the SMUS home page, click "Create project"
Configure project:
- Project name: Enter your project name (e.g., smus-blog-test)

Note: Do not use underscore (_) in the project name as it will be used in SageMaker pipeline which doesn't support underscores

Project profile: Select regression
Click "Continue"

Customize blueprint parameters:
- Under Tooling: Select aws-github-smus-connection as the connection name
- Connection provider: Select "New repository and new branch"
- Repository name: Enter build repo name (e.g., smus-blog-build-repo)
- Click "Continue"
Review and create:
- Review your selections
- Click "Create project"
- Wait 5-10 minutes for project creation to complete

Repository Naming Convention

After project creation, you'll see two repositories created in your private GitHub organization:

Build repository: {repo-name-specified} (e.g., smus-blog-build-repo)
Deploy repository: {project-id}-{datazone-domain-id}-deploy-repo

Workflow Trigger Behavior

Build Repository: Workflow triggers on code changes but requires TRIGGER_PIPELINE_EXECUTIO 8BD0 N=true to execute
Deploy Repository: Workflow only triggers via model approval events or manual execution (no automatic triggers on code changes)

7. Glue Database and Table Configuration

Create Table in SageMaker Unified Studio

After project creation, you need to create a Glue table with your data:

Login to your project in SageMaker Unified Studio
Navigate to Data → Lakehouse → awsdatacatalog
Find your Glue database: glue_db_****
Click the table tile option on the side and click create table
Upload dataset:
- Download abalone dataset from your build repository: aiops-seed-code/regression/model_build/ml_pipelines/data/abalone-dataset.csv
- Table name: Rename to abalone
- Click "Next"
In the add data page:
- Review the schema
- Click "Create table"

Note: During table creation, make note of the Glue database and table names as you'll need them in the next step.

Update GitHub Secrets

Update the GitHub secrets in both build and deploy repositories with these Glue database and table names:

For Build Repository

Navigate to: your-org/{repo-name-specified}
Go to Settings → Secrets and variables → Actions
Update the following secrets:
- GLUE_DATABASE: Update from glue_db to your actual database name (e.g., glue_db_3hwskube5dybhj)
- GLUE_TABLE: Update from abalone to your actual table name (e.g., abalone)

For Deploy Repository

Navigate to: your-org/{project-id}-{datazone-domain-id}-deploy-repo
Go to Settings → Secrets and variables → Actions
Update the same secrets:
- GLUE_DATABASE: Your actual database name
- GLUE_TABLE: Your actual table name

Enable Pipeline Execution

After updating the Glue database and table secrets, you need to enable pipeline execution:

For Build Repository

Navigate to: your-org/project-build-repo
Go to Settings → Secrets and variables → Actions → Variables tab
Create a new repository variable:
- Name: TRIGGER_PIPELINE_EXECUTION
- Value: true
Click "Add variable"

Note: By default, pipeline execution is disabled to prevent failures when Glue configuration is not yet updated. This variable must be set to true to enable the SageMaker pipeline execution.

8. Build and Model Creation Process

After updating the GitHub secrets and enabling pipeline execution, the build workflow can be triggered. The workflow includes a safety check that prevents execution until the TRIGGER_PIPELINE_EXECUTION variable is set to true, ensuring that Glue database and table configurations are properly updated before running the pipeline.

For the first run, you'll need to trigger the workflow manually using GitHub Actions workflow dispatch. Alternatively, you can trigger the workflow by adding an empty line to any file in the repository.

For subsequent runs, any code changes made to modify the job or make changes will automatically trigger the build pipeline responsible for creating the model.

Manual Workflow Trigger

Follow these steps to manually trigger the workflow:

Ensure Prerequisites:
- Glue database and table secrets are updated (Section 6)
- TRIGGER_PIPELINE_EXECUTION variable is set to true
Go to build repository GitHub Actions
Select the workflow
Click "SageMaker Pipeline build SMUS project"
Click "Run workflow"

Note: If the TRIGGER_PIPELINE_EXECUTION variable is not set to true, the workflow will exit gracefully with instructions on how to enable it.

Verify Model Registration

After successful build pipeline execution:

Navigate to SageMaker Unified Studio
Go to Build → AI OPS → Model Registry
Find your model package group: aiops-{project-id}-models
Verify model version 1 with status "PendingManualApproval"

9. Model Approval and Deployment Process

Manual Model Approval

Navigate to SageMaker Unified Studio
Go to Build → AI OPS → Model Registry
Find your model package group: aiops-{project-id}-models
Select the model package you want to approve
Click "Update model approval status"
Change status from "PendingManualApproval" to "Approved"
Add approval comments (optional)
Click "Update status"

Automated Deployment Trigger

When a model is approved:

SageMaker emits approval event to EventBridge
EventBridge rule MlOpsSmusStack-model-approval-rule detects the event
Model approval Lambda MlOpsSmusStack-model-approval-trigger is invoked to:
- Extract project information from the event
- Construct deploy repository name: project-deploy-repo
- Trigger GitHub Actions workflow in deploy repository via API
- Use workflow_dispatch trigger with appropriate inputs

Once the automated deployment Lambda successfully triggers the workflow, the following deployment steps are executed in the deploy repository:

Deploy Workflow Execution

Find approved model in SageMaker Model Registry
Create SageMaker model from approved model package
Create endpoint configuration with instance settings
Deploy SageMaker endpoint for real-time inference
Test endpoint with sample data
Update endpoint if already exists

Monitor Deployment

Check GitHub Actions in deploy repository
Monitor workflow execution status and logs
Monitor SageMaker endpoints:

# List endpoints
aws sagemaker list-endpoints --sort-by "CreationTime" --sort-order "Descending"

# Check endpoint status
aws sagemaker describe-endpoint --endpoint-name "your-endpoint-name"

Verify Deployed Endpoint

Navigate to SageMaker Unified Studio
Go to Model Development → Inference → Endpoints
Verify endpoint status: "InService"
Test endpoint with sample data if needed

10. Troubleshooting

Common Issues and Solutions

1. GitHub Token Issues

Problem: Repository creation fails with authentication errors
Solution: Verify GitHub token in Secrets Manager, ensure correct permissions

2. Source Repository Files Not Visible in JupyterLab

Problem: When opening JupyterLab in your project, you don't see the expected files in the src folder
Solution:
1. In SageMaker Studio, open your project
2. Go to "Build, IDE & applications"
3. Launch JupyterLab
4. Once in JupyterLab, open a new Terminal
5. Navigate to the source directory: cd /home/sagemaker-user/src
6. Run git reset --hard to clear any local changes
7. Run git pull to sync with the remote repository
8. Refresh your JupyterLab file browser to see the updated files

3. IAM Role Trust Relationship

Problem: GitHub Actions cannot assume IAM role
Solution: Verify trust relationship includes correct GitHub organization

4. EventBridge Not Triggering

Problem: Step Functions not triggered on project creation
Solution: Check EventBridge rule is enabled and pattern matches

5. Model Approval Not Triggering Deploy

Problem: Deploy workflow not triggered after model approval
Solution: Verify model approval EventBridge rule and Lambda logs

6. Glue Database/Table Not Found

Problem: Build pipeline fails with Glue database errors
Solution: Update GitHub secrets with correct Glue database/table names

Log Locations

Lambda Functions: /aws/lambda/{function-name}
Step Functions: /aws/stepfunctions/{state-machine-name}
EventBridge: /aws/events/rule/{rule-name}
GitHub Actions: Repository → Actions tab

11. Clean-up

Endpoint Cleanup

When you no longer need the deployed endpoints, clean them up to avoid ongoing costs:

# Delete endpoint (stops inference)
aws sagemaker delete-endpoint --endpoint-name <endpoint-name>

# Delete endpoint configuration
aws sagemaker delete-endpoint-config --endpoint-config-name <config-name>

# Delete model
aws sagemaker delete-model --model-name <model-name>

Pipeline Cleanup

# Delete SageMaker pipeline
aws sagemaker delete-pipeline --pipeline-name "githubactions-<project-id>"

# Clean up S3 artifacts (be careful - this removes all data)
aws s3 rm s3://<artifact-bucket>/githubactions-<project-id>/ --recursive

Model Registry Cleanup

# Delete model package group (removes all model versions)
aws sagemaker delete-model-package-group --model-package-group-name <group-name>

Infrastructure Cleanup

# Destroy CDK stack (removes all SMUS infrastructure)
cd smus-cdk
cdk destroy

# Clean up CDK bootstrap (optional - only if not used by other projects)
# aws cloudformation delete-stack --stack-name CDKToolkit

SageMaker Unified Studio Cleanup

Delete project in SageMaker Unified Studio
Remove project profile if no longer needed
Delete Deploy and Build repository created in GitHub organization
Delete AWS resources manually
- IAM Role aiops-smus-github-action and IAM Identity Provi 5DE4 der token.actions.githubusercontent.com may need to be deleted manually
Disable Git connection if not used by other projects

Name		Name	Last commit message	Last commit date
aiops-seed-code/regression		aiops-seed-code/regression
images		images
smus-cdk		smus-cdk
CODE_OF_CONDUCT.md		CODE_OF_CONDUCT.md
CONTRIBUTING.md		CONTRIBUTING.md
LICENSE		LICENSE
README.md		README.md

Latest commit

History

Folders and files

Repository files navigation