AWS Lambda function to parse S3 server log files and export metrics to AWS CloudWatch.

• Description
• Extra S3 Metrics available
• Deployment
• FAQ
• Contributing
• License
• Author

NOTE: AWS introduced S3 request metrics at re:Invent 2016. This makes parts of below README outdated as of December 2016 and it might not be the most optimal way of achieving your goals. Before reading further please check out AWS documentation: S3 Request Metrics. There are still extra, more specific metrics provided by s3logs-cloudwatch that the "official way" does not provide.

AWS S3 is a managed storage service. The only metrics available in AWS CloudWatch for S3 are NumberOfObjects and BucketSizeBytes. In order to understand your S3 usage better you need to do some extra work. Reasons can be multiple: understanding your AWS bill, getting better idea of how your deployed applications use S3, access auditing, security, performance considerations or proactive monitoring because you love metrics.

AWS offers server access logging to track requests for access to your bucket. This is a feature that is disabled by default, but you can enable it free of charge (you will be charged for storage used for storing files and data transfer for access to delivered log files).

To find out more about Server Access Logging feature of S3, head over to Server Access Logging section in Amazon Simple Storage Service Developer Guide.

For more detailed information, head over to Server Access Log Format section in Amazon Simple Storage Developer Guide.

ObjectSize

The total size of the object in question.

1. Create a new S3 bucket for storing your S3 logs, for example com-companyname-s3logs

2. Enable logging on com-companyname-mybucket bucket

   For information on how to enable bucket logging, see Enabling Logging Using the Console.

   IMPORTANT: Set com-companyname-s3logs as Target Bucket for log objects. Specify com-companyname-mybucket/ as Target Prefix. Target Prefix value must end with a / for s3logs-cloudwatch Lambda function to properly set BucketName dimension in exported CloudWatch metrics

   

3. Clone this GitHub repository so you have it locally and prepare deployment package. Upload deployment package to S3.

   $ git clone git@github.com:maginetv/s3logs-cloudwatch.git
   $ cd s3logs-cloudwatch/

   Edit configuration.ini file if you would like to adjust the settings or change which metrics will be exported to CloudWatch. Then zip the files and put resulting zip somewhere on S3.

   $ zip s3logs-cloudwatch.zip lambda_function.py configuration.ini
   $ s3cmd cp s3logs-cloudwatch.zip s3://com-companyname-packages/lambda/s3logs-cloudwatch.zip

4. Use provided CloudFormation template to deploy s3logs-cloudwatch AWS Lambda function and IAM role that is required to run it.

   Go to CloudFormation in AWS Console and create the stack using provided CloudFormation template cloudformation/cloudformation-deploy.json.

   

5. Last step is creating and enabling triggers (event sources) for deployed Lambda function. Head over to AWS Lambda in AWS console and find s3logs-cloudwatch function in functions list. Deployed function is supposed to be run every time a new S3 log file is delivered to com-companyname-s3logs bucket. Picture below shows the configuration that has to be in place for the trigger to work:

   

   If you need to enable s3logs-cloudwatch to graph S3 metrics from multiple S3 buckets just add another trigger (with different prefix/suffix setting).

6. That's it. Your new Cloudwatch metrics will start appearing soon.

A: Enable S3 lifecycle rules for com-companyname-s3logs bucket. You can configure com-companyname-s3logs bucket to delete log files permanently after certain period (for example 1 or 7 days). Check Object Lifecycle Management in Amazon S3 Developer Guide.

Released under the Apache 2.0 license.

Copyright 2016 Magine AB

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.