initial commit

sannya-singal · May 28, 2024 · 2086392 · 2086392
1 parent 7f2ed57
commit 2086392
Show file tree

Hide file tree

Showing 14 changed files with 460 additions and 1 deletion.
diff --git a/.gitignore b/.gitignore
@@ -0,0 +1,3 @@
+venv
+lambda_function.zip
+tests/__pycache__/
diff --git a/README.md b/README.md
@@ -1 +1,170 @@
-# sqs-s3-lambda-multi-account-multi-region
+# Testing cross-account and cross-region SQS S3 Lambda setup
+
+This repository contains an example on how to deploy a multi-account, multi-region setup in which a Lambda function in one account invoked by a REST API sends a message to an SQS queue in another account and another region and uploads a file to an S3 bucket in the same account and region as the SQS queue.
+
+LocalStack supports cross-account setup that allows you to namespace resources based on the AWS Account ID. In this example, you can use three local AWS accounts to simulate the cross-account setup. They are:
+
+- **Account A** (`111111111111`): The account that contains the SQS queue and the S3 bucket in `us-east-1` region.
+- **Account B** (`222222222222`): The account that contains the SQS queue and the S3 bucket in `eu-west-1` region.
+- **Account C** (`333333333333`): The account that contains the Lambda function and the API Gateway in `us-east-1` region.
+
+## Prerequisites
+
+- [Docker](https://docs.docker.com/get-docker/)
+- [LocalStack](https://docs.localstack.cloud/getting-started/installation/)
+- [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-install.html) & [`awslocal`](https://github.com/localstack/awscli-local)
+
+Start LocalStack with the following command:
+
+```bash
+DEBUG=1 ENFORCE_IAM=1 localstack start
+```
+
+You can follow the instructions below or run the whole sample using the provided three scripts.
+
+```bash
+./accountA.sh
+./accountB.sh
+./accountC.sh
+```
+
+## Create the SQS queue and S3 bucket in Account A
+
+For Account A we will open a terminal and provide the following `AWS_ACCOUNT_ID` and `AWS_SECRET_ACCESS_KEY` environment variables:
+
+```bash
+export AWS_ACCESS_KEY_ID=111111111111
+export AWS_SECRET_ACCESS_KEY=test
+```
+
+Run the following commands to store the account ID, queue name and bucket name in variables:
+
+```bash
+ACCOUNT_ID=$(awslocal sts get-caller-identity --query Account --output text)
+QUEUE_NAME="team-queue-${ACCOUNT_ID}"
+BUCKET_NAME="team-bucket-${ACCOUNT_ID}"
+```
+
+Create the SQS queue in Account A and set the policy to allow Account C to send messages to the queue in Account A:
+
+```bash
+QUEUE_URL=$(awslocal sqs create-queue --queue-name $QUEUE_NAME --attributes VisibilityTimeout=300 --query 'QueueUrl' --output text)
+awslocal sqs set-queue-attributes --queue-url $QUEUE_URL --attributes file://sqs-policy1.json
+```
+
+Create the S3 bucket in Account A and set the policy to allow Account C to upload files to the bucket in Account A:
+
+```bash
+awslocal s3api create-bucket --bucket $BUCKET_NAME
+awslocal s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://s3-policy1.json
+```
+
+## Create the SQS queue and S3 bucket in Account B
+
+For Account B we will open another terminal and provide the following `AWS_ACCOUNT_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_REGION=eu-west-1` environment variables:
+
+```bash
+export AWS_ACCESS_KEY_ID=222222222222
+export AWS_REGION=eu-west-1
+export AWS_SECRET_ACCESS_KEY=test
+```
+
+Run the following commands to store the account ID, queue name and bucket name in variables:
+
+```bash
+ACCOUNT_ID=$(awslocal sts get-caller-identity --query Account --output text)
+QUEUE_NAME="team-queue-${ACCOUNT_ID}"
+BUCKET_NAME="team-bucket-${ACCOUNT_ID}"
+```
+
+Create the SQS queue in Account B and set the policy to allow Account C to send messages to the queue in Account B:
+
+```bash
+QUEUE_URL=$(awslocal sqs create-queue --queue-name $QUEUE_NAME --region eu-west-1 --attributes VisibilityTimeout=300 --query 'QueueUrl' --output text)
+awslocal sqs set-queue-attributes --queue-url $QUEUE_URL --attributes file://sqs-policy2.json
+```
+
+Create the S3 bucket in Account B and set the policy to allow Account C to upload files to the bucket in Account B:
+
+```bash
+awslocal s3api create-bucket --bucket $BUCKET_NAME
+awslocal s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://s3-policy2.json
+```
+
+## Create the Lambda function and API Gateway in Account C
+
+For Account C we will open another terminal and provide the following `AWS_ACCOUNT_ID` and `AWS_SECRET_ACCESS_KEY` environment variables:
+
+```bash
+export AWS_ACCESS_KEY_ID=333333333333
+export AWS_SECRET_ACCESS_KEY=test
+```
+
+Create lambda execution role and attach it to the policy:
+
+```bash
+awslocal iam create-role --role-name common-lambda-role --assume-role-policy-document file://trust-policy.json
+awslocal iam put-role-policy --role-name common-lambda-role --policy-name common-lambda-policy --policy-document file://lambda-policy.json
+```
+
+Create the deployment package for the Lambda function and create the function:
+
+```bash
+zip lambda_function.zip lambda_function.py
+awslocal lambda create-function --function-name CommonLambda \
+                --runtime python3.11 \
+                --role arn:aws:iam::333333333333:role/common-lambda-role \
+                --handler lambda_function.handler \
+                --zip-file fileb://lambda_function.zip \
+                --timeout 300
+```
+
+Create the API Gateway and store the API ID and the resource ID in variables:
+
+```bash
+awslocal apigateway create-rest-api --name "CommonLambdaApi"
+API_ID=$(awslocal apigateway get-rest-apis --query 'items[?name==`CommonLambdaApi`].id' --output text)
+RESOURCE_ID=$(awslocal apigateway get-resources --rest-api-id $API_ID --query 'items[?path==`/`].id' --output text)
+```
+
+Create a resource and a POST method for the API Gateway:
+
+```bash
+UPLOAD_RESOURCE_ID=$(awslocal apigateway create-resource --rest-api-id $API_ID --parent-id $RESOURCE_ID --path-part upload --query 'id' --output text)
+awslocal apigateway put-method --rest-api-id $API_ID --resource-id $UPLOAD_RESOURCE_ID --http-method POST --authorization-type "NONE"
+```
+
+Run the following command to integrate the POST method with the Lambda function and deploy the API Gateway:
+
+```bash
+awslocal apigateway put-integration --rest-api-id $API_ID --resource-id $UPLOAD_RESOURCE_ID --http-method POST --type AWS_PROXY --integration-http-method POST --uri arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:333333333333:function:CommonLambda/invocations
+awslocal apigateway create-deployment --rest-api-id $API_ID --stage-name prod
+```
+
+Add the permission to the Lambda function to be invoked by the API Gateway:
+
+```bash
+awslocal lambda add-permission \
+                --function-name CommonLambda \
+                --statement-id 1 \
+                --action lambda:InvokeFunction \
+                --principal apigateway.amazonaws.com \
+                --source-arn "arn:aws:execute-api:us-east-1:333333333333:$API_ID/*/*/*" \
+                --source-account 333333333333
+```
+
+## Test the setup by invoking the API Gateway endpoint and checking the SQS queue and S3 bucket in Account A and Account B:
+
+Create a virtual environment and install the required packages:
+
+```bash
+python3 -m venv venv
+source venv/bin/activate
+pip install -r requirements-dev.txt
+```
+
+Run the following command to test the setup:
+
+```bash
+pytest tests/test_api.py
+```
diff --git a/accountA.sh b/accountA.sh
@@ -0,0 +1,26 @@
+export AWS_ACCESS_KEY_ID=111111111111
+export AWS_SECRET_ACCESS_KEY=test
+
+
+ACCOUNT_ID=$(awslocal sts get-caller-identity --query Account --output text)
+echo "> Current Account ID: $ACCOUNT_ID"
+
+
+QUEUE_NAME="team-queue-${ACCOUNT_ID}"
+BUCKET_NAME="team-bucket-${ACCOUNT_ID}"
+
+echo "> Creating Queue: $QUEUE_NAME..."
+QUEUE_URL=$(awslocal sqs create-queue --queue-name $QUEUE_NAME --attributes VisibilityTimeout=300 --query 'QueueUrl' --output text)
+echo "Queue created URL: $QUEUE_URL"
+
+echo "> Setting Queue Policy for $QUEUE_URL..."
+awslocal sqs set-queue-attributes --queue-url $QUEUE_URL --attributes file://sqs-policy1.json
+
+echo "> Creating Bucket: $BUCKET_NAME..."
+awslocal s3api create-bucket --bucket $BUCKET_NAME
+echo "Bucket created: $BUCKET_NAME"
+
+echo "> Setting Bucket Policy for $BUCKET_NAME..."
+awslocal s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://s3-policy1.json
+
+echo "Setup completed for Account ID: $ACCOUNT_ID"
diff --git a/accountB.sh b/accountB.sh
@@ -0,0 +1,25 @@
+export AWS_ACCESS_KEY_ID=222222222222
+export AWS_REGION=eu-west-1
+export AWS_SECRET_ACCESS_KEY=test
+
+
+ACCOUNT_ID=$(awslocal sts get-caller-identity --query Account --output text)
+echo "> Current Account ID: $ACCOUNT_ID"
+
+
+QUEUE_NAME="team-queue-${ACCOUNT_ID}"
+BUCKET_NAME="team-bucket-${ACCOUNT_ID}"
+
+echo "> Creating Queue: $QUEUE_NAME..."
+QUEUE_URL=$(awslocal sqs create-queue --queue-name $QUEUE_NAME --region eu-west-1 --attributes VisibilityTimeout=300 --query 'QueueUrl' --output text)
+echo "Queue created URL: $QUEUE_URL"
+echo "> Setting Queue Policy for $QUEUE_URL..."
+awslocal sqs set-queue-attributes --queue-url $QUEUE_URL --attributes file://sqs-policy2.json
+
+echo "> Creating Bucket: $BUCKET_NAME..."
+awslocal s3api create-bucket --bucket $BUCKET_NAME
+echo "Bucket created: $BUCKET_NAME"
+echo "> Setting Bucket Policy for $BUCKET_NAME..."
+awslocal s3api put-bucket-policy --bucket $BUCKET_NAME --policy file://s3-policy2.json
+
+echo "Setup completed for Account ID: $ACCOUNT_ID"
diff --git a/accountC.sh b/accountC.sh
@@ -0,0 +1,36 @@
+export AWS_ACCESS_KEY_ID=333333333333
+export AWS_SECRET_ACCESS_KEY=test
+
+awslocal iam create-role --role-name common-lambda-role --assume-role-policy-document file://trust-policy.json
+awslocal iam put-role-policy --role-name common-lambda-role --policy-name common-lambda-policy --policy-document file://lambda-policy.json
+
+zip lambda_function.zip lambda_function.py
+
+awslocal lambda create-function --function-name CommonLambda \
+                --runtime python3.11 \
+                --role arn:aws:iam::333333333333:role/common-lambda-role \
+                --handler lambda_function.handler \
+                --zip-file fileb://lambda_function.zip \
+                --timeout 300
+
+awslocal apigateway create-rest-api --name "CommonLambdaApi"
+
+API_ID=$(awslocal apigateway get-rest-apis --query 'items[?name==`CommonLambdaApi`].id' --output text)
+
+RESOURCE_ID=$(awslocal apigateway get-resources --rest-api-id $API_ID --query 'items[?path==`/`].id' --output text)
+
+UPLOAD_RESOURCE_ID=$(awslocal apigateway create-resource --rest-api-id $API_ID --parent-id $RESOURCE_ID --path-part upload --query 'id' --output text)
+
+awslocal apigateway put-method --rest-api-id $API_ID --resource-id $UPLOAD_RESOURCE_ID --http-method POST --authorization-type "NONE"
+
+awslocal apigateway put-integration --rest-api-id $API_ID --resource-id $UPLOAD_RESOURCE_ID --http-method POST --type AWS_PROXY --integration-http-method POST --uri arn:aws:apigateway:us-east-1:lambda:path/2015-03-31/functions/arn:aws:lambda:us-east-1:333333333333:function:CommonLambda/invocations
+
+awslocal apigateway create-deployment --rest-api-id $API_ID --stage-name prod
+
+awslocal lambda add-permission \
+                --function-name CommonLambda \
+                --statement-id 1 \
+                --action lambda:InvokeFunction \
+                --principal apigateway.amazonaws.com \
+                --source-arn "arn:aws:execute-api:us-east-1:333333333333:$API_ID/*/*/*" \
+                --source-account 333333333333
diff --git a/lambda-policy.json b/lambda-policy.json
@@ -0,0 +1,12 @@
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Effect": "Allow",
+      "Action": "*",
+      "Resource": "*"
+    }
+  ]
+}
+
+
diff --git a/lambda_function.py b/lambda_function.py
@@ -0,0 +1,46 @@
+import json
+import boto3
+import os
+
+def handler(event, context):
+    body = json.loads(event.get("body"))
+    account = body.get("account")
+    region = body.get("region")
+    content = body.get("content")
+    filename = body.get("filename")
+
+    s3 = boto3.client("s3", region_name=region)
+    sqs = boto3.client("sqs", region_name=region)
+
+    bucket_name = f"team-bucket-{account}"
+    queue_name = f"team-queue-{account}"
+
+    s3.put_object(
+        Bucket=bucket_name,
+        Key=filename,
+        Body=content
+    )
+
+    queue_url = sqs.get_queue_url(
+        QueueName=queue_name,
+        QueueOwnerAWSAccountId=account
+    ).get("QueueUrl")
+
+    sqs.send_message(
+        QueueUrl=queue_url,
+        MessageBody=f"File {filename} uploaded to bucket {bucket_name}"
+    )
+
+    response_body = {
+        "account": account,
+        "region": region,
+        "bucket_name": bucket_name,
+        "bucket_key": filename,
+        "queue_url": queue_url,
+    }
+    return {
+        "statusCode": 200,
+        "body": json.dumps(response_body)
+    }
+
+
diff --git a/requirements-dev.txt b/requirements-dev.txt
@@ -0,0 +1,3 @@
+pytest==6.2.5
+requests
+boto3
diff --git a/s3-policy1.json b/s3-policy1.json
@@ -0,0 +1,23 @@
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Principal": {
+        "AWS": [
+          "arn:aws:iam::333333333333:root"
+        ]
+      },
+      "Effect": "Allow",
+      "Action": [
+        "s3:GetObject",
+        "s3:PutObject",
+        "s3:DeleteObject",
+        "s3:ListBucket"
+      ],
+      "Resource": [
+        "arn:aws:s3:::team-bucket-111111111111",
+        "arn:aws:s3:::team-bucket-111111111111/*"
+      ]
+    }
+  ]
+}
diff --git a/s3-policy2.json b/s3-policy2.json
@@ -0,0 +1,25 @@
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Principal": {
+        "AWS": [
+          "arn:aws:iam::333333333333:root"
+        ]
+      },
+      "Effect": "Allow",
+      "Action": [
+        "s3:GetObject",
+        "s3:PutObject",
+        "s3:DeleteObject",
+        "s3:ListBucket"
+      ],
+      "Resource": [
+        "arn:aws:s3:::team-bucket-222222222222",
+        "arn:aws:s3:::team-bucket-222222222222/*"
+      ]
+    }
+  ]
+}
+
+
diff --git a/sqs-policy1.json b/sqs-policy1.json
@@ -0,0 +1,3 @@
+{
+  "Policy": "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Effect\": \"Allow\", \"Principal\": {\"AWS\": [\"arn:aws:iam::333333333333:root\"]}, \"Action\": \"sqs:*\", \"Resource\": [\"arn:aws:sqs:us-east-1:111111111111:team-queue-111111111111\"]}]}"
+}
diff --git a/sqs-policy2.json b/sqs-policy2.json
@@ -0,0 +1,3 @@
+{
+  "Policy": "{\"Version\": \"2012-10-17\", \"Statement\": [{\"Effect\": \"Allow\", \"Principal\": {\"AWS\": [\"arn:aws:iam::333333333333:root\"]}, \"Action\": \"sqs:*\", \"Resource\": [\"arn:aws:sqs:eu-west-1:222222222222:team-queue-222222222222\"]}]}"
+}