Automation - Enable S3 Logging to all buckets using Nodejs

Logging helps to get the detailed access logs in a bucket. We can enable the S3 logging for each bucket through console.

But what if we need to enable logging for all the buckets in one go? Here’s a post to automate enabling the S3 logging using Nodejs.
Refer to the script on this link → https://github.com/powerupcloud/Enable-S3-Logging.git

Prerequisites:

  • Ensure the IAM Role / IAM User has the required S3 access to list buckets, get bucket location and enable logs.
  • A text file containing bucket names and their locations at each line as shown below:
bucket01,None  
bucket02,None  
bucket03,ap-south-1  
bucket04,ap-southeast-1  
bucket05,ap-southeast-1  
.......

Here, “None” represents us-east-1 region. The following aws cli commands can be used for getting all the bucket names and their regions in a text file:

aws s3api list-buckets  --output text | awk '{print $3}' > /tmp/bucket_list.txt  
for i in `cat /tmp/bucket_list.txt`; do aws s3api get-bucket-location --bucket $i --output text  >> /tmp/bucket_region.txt; done  
  • Ensure the following node modules are installed:
    • aws-sdk
    • line-reader
  • Ensure the buckets are created where the logs will get stored. Also, the logging target bucket must be in the same region as the bucket being logged (i.e. we cant store the us-east-1 region buckets logs in ap-southeast-1 region bucket). So, In our case, we have created the three logs buckets in three different regions:
    • logs-us-east-1
    • logs-ap-south-1
    • logs-ap-southeast-1 The logs for buckets located in us-east-1 will get stored in “logs-us-east-1” bucket. The logs for buckets located in ap-south-1 will get stored in “logs-ap-south-1” bucket. Similarly, for ap-southeast-1 buckets, the logs will get stored in “logs-ap-southeast-1” bucket.

Script to Enable Logging

Following is a snippet of the script:

    lineReader.eachLine('./bucket_list.txt', function(line, last) {
        if (line.indexOf('None') > -1) {
            bucket = line.substring(0, line.indexOf(","))
            var params = {
                Bucket: bucket,
                BucketLoggingStatus: {
                    LoggingEnabled: {
                        TargetBucket: 'logs-us-east-1',
                        TargetGrants: [{
                            Grantee: {
                                Type: 'Group',
                                URI: 'http://acs.amazonaws.com/groups/s3/LogDelivery'
                            },
                            Permission: 'FULL_CONTROL'
                        }],
                        TargetPrefix: bucket + '/'
                    }
                }
            }
            s3.putBucketLogging(params, function(err, data) {
                if (err) console.log(err, err.stack); // an error occurred
                else console.log(data); // successful response
            });
        }

Description:

  • line-reader module is used to read the file line by line.
  • If-else condition is used to check if the current line contains the word “None” (which represents the us-east-1 region). If it matches, the logging will be enabled for the bucket included in the same line that matched.
  • putBucketLogging() API is being used for enabling the S3 logging.
  • TargetBucket is set to the name of the bucket that we created for storing the logs of us-east-1 buckets.
  • TargetPrefix can be set as required.

We have used the similar If-else block to enable logging for buckets in different regions. The “None” will be replaced by the other region name (for example, ap-south-1) and the target bucket name will be replaced with the bucket that we created for storing the logs (for example, logs-ap-south-1).

& that’s it..!! Hope it was useful. Happy Automation..!! :)

Priyanka Sharma

Priyanka is Senior Cloud and DevOps Engineer. She can churn out CloudFormation templates at a moment's notice and play with Chef/Ansible. Dancing, music, badminton and word games are her hobbies

comments powered by Disqus