Repository navigation
Moloch 0.11.4 has an experimental PCAP writer to S3. The expected use case is low volume capture process on each EC2 instance writing to S3 instead of a central moloch tap writing to local disk. One big benefit only a capture node on each EC2 instance is required. It does not require that the view is running. This mean the EC2 instances can come and go and the PCAP will still be in S3. For bucket security, the capture nodes can be configured with an S3 configuration that ONLY has PutObject permissions.
WARNING - Currently all the writing from moloch-capture to S3 is NOT sent over TLS, is NOT compressed, and is NOT AMZ server sided encrypted at rest. Waiting to get some feedback or see how we use this before adding a bunch more features.
These instructions are still a work in progress, please provide feedback and suggestions. These instructions assume only one AWS account will be writing into the bucket using two different users. If you will be writing to a central bucket from many AWS accounts/VPCs then you will need to use a bucket policy.
- Create a S3 bucket
example-moloch-pcap - Create a IAM Access policy
MolochCapturePolicy
{
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": [
"arn:aws:s3:::example-moloch-pcap",
"arn:aws:s3:::example-moloch-pcap/*"
]
}
]
}
- Create two IAM users and download their access keys
**
MolochCapture- used on capture nodes. Attach the newly created MolochCapturePolicy **MolochViewer- used on viewer nodes. AmazonS3FullAccess - On the capture nodes you'll want a config file with the following settings
plugins=writer-s3.so;#OTHER PLUGINS
pcapWriteMethod=s3
s3Region=us-east-1
s3Bucket=example-moloch-pcap
s3AccessKeyId=THEACESSKEYID_FOR_MOLOCHCAPTURE
s3SecretAccessKey=THESECRETACCESSKEY_FOR_MOLOCHCAPTURE
- On the viewer node you'll need to set the following config. Remember this can be a central viewer node.
pcapWriteMethod=s3
viewerPlugins=writer-s3/index.js
s3AccessKeyId=THEACESSKEYID_FOR_MOLOCHVIEWER
s3SecretAccessKey=THESECRETACCESSKEY_FOR_MOLOCHVIEWER
s3ExpireDays=10