Capture EC2 launch/termination events using CloudTrail, CloudWatch & Lambda

Bakrudeen, Software Engineer at Powerupcloud has contributed to this post.

Its a good problem to have if you have an environment which scales up so dynamically that you end up with hundreds of autoscaling events per day and it becomes to keep track of how many instances of what type were launched and terminated. At the end of the day, you want to keep costs in control and find that balance where your environment is scaling in response to spikes in traffic or workload at the same time its not costing you an arm and a leg.

We have a client who had this problem and we had to come up with a way to track autoscaling events. The goal was to track how many servers of what instance type were launched, how many were terminated. We tapped into CloudTrail events, sent them to CloudWatch and triggered lambda to parse the logs to write to a DynamoDB table. Once this data is in DynamoDB, the rest was trivial. Even Though the terminate event doesn't tell us what instance type was terminated, it was just a matter of doing a left outer join (once you move data to a relational database, that is). Even though this is a use case which is very specific to our client, I thought explaining the way we went about it would help someone on the interweb. So what follows is a step by step.

Create CloudTrail and CloudWatch

Create a New CloudTrial and specify a bucket to store the logs
Once then CloudTrail is created you can enable the cloudwatch Logs

Enable CloudTrail Log in CloudWatch

  • Select the cloudTrail which you need to enable the cloudwatch.
  • You can see the cloudWatch log enabling option

Now select our CloudTrail Log
Give the new log or existing log name which need to be appear in cloudwatch.
Finally click the Allow button for enable the logs to cloudwatch

Create IAM Role for Lambda and DynamoDB

Choose a role name you want
Select the Service Role Type AWS Lambda
Select AWS Service AWS Lambda AmazonDynamoDBFullAccess Policy Type.
A note here. For demo purposes, I just chose full access. In production, you should only be allowing minimum necessary access.
Click the Create Role Button :

Create a Lambda Function to Store Data in DynamoDB

Create a new Lambda Function
Check below images and use the code below
NodeJs Code for storing launch and terminate historical data to DynamoDB tables. We are going to store data in two separate tables - one for instances created and the other for terminated.

var aws = require('aws-sdk');  
const zlib = require('zlib');  
exports.handler = (event, context, callback) => {  
    const payload = new Buffer(event.awslogs.data, 'base64');
    zlib.gunzip(payload, (err, res) => {
        if (err) {
            return callback(err);
        }
        const parsed = JSON.parse(res.toString('utf8'));  
        for (var i=0; i<parsed['logEvents'].length; i++){
        parsed1 =parsed['logEvents'][i]['message'];
        const parsedfinal = JSON.parse(parsed1.toString('utf8'));
        console.log('Decoded final in:',parsedfinal);
}
 console.log('Decoded final out:',parsedfinal);
 var creationDate = "";
 var userName = parsedfinal.userIdentity.userName; 
 var instancesSet = parsedfinal.responseElements.instancesSet;
 console.log("instancesSet", instancesSet);
 console.log("userName", userName);  
 var eventName = parsedfinal['eventName'];
 var eventTime = parsedfinal['eventTime'];
 var awsRegion = parsedfinal['awsRegion'];
 if(eventName === "RunInstances")
 {
      for (var j=0; j<instancesSet.items.length; j++){
           instanceId = instancesSet['items'][j]['instanceId'];
           instanceType  = instancesSet['items'][j]['instanceType'];
           console.log("instanceId", instanceId);         
           console.log("instanceType", instanceType);
       }
      ddb = new aws.DynamoDB({params: {TableName: 'Titans-instance-start-state'}});    
 var itemParams = {Item: {Id: {S: instanceId} ,Region: {S: awsRegion},userName: {S: userName},instanceType: {S: instanceType},eventName: {S: eventName},creationDate: {S: eventTime} }};
 ddb.putItem(itemParams, function(err, data)
{
  if(err) { context.fail(err)}

  else {
           console.log(data);
           context.succeed();
      }
  });
 }
 else if(eventName === "TerminateInstances")
 {
      for (var k=0; k<instancesSet.items.length; k++){
           instanceId = instancesSet['items'][k]['instanceId'];
           console.log("instanceId", instanceId);
       }
      ddb1 = new aws.DynamoDB({params: {TableName: 'Titans-instance-terminate-state'}});    
  var itemParams1 = {Item: {Id: {S: instanceId} ,Region: {S: awsRegion},userName: {S: userName},eventName: {S: eventName},creationDate: {S: eventTime} }};    
  ddb1.putItem(itemParams1, function(err, data)
{
  if(err) { context.fail(err)}

  else {
           console.log(data);
           context.succeed();
      }
  });
 }
  callback(null, `Successfully processed ${parsed.logEvents.length} log events.`);
    });
};

Select the CloudWatch Log
Now we will stream the logs with Lambda
Now select our lambda function
Select log format as JSON
This is the main configuration, filter the logs only for Instance launch and terminate events.
Use this for filter pattern

{ $.eventName = "RunInstances" ||  $.eventName = "TerminateInstances" }

Create DynamoDB Tables

We are now all set! Go ahead and create a few micro instances and test to see if your setup is working fine.

In my case, here is how the instance create table looks like.

And the terminate instances table

Dashboard and joining both tables

We use ReDash extensively to visualize all sorts of data. Redash can connect DynamoDB directly but we chose to export this data everyday to a couple of mysql tables so that we can run a left outer join.

For example, number of instances created per day

SELECT DATE(TIMESTAMP(REPLACE(REPLACE(createdate, 'T', ' '), 'Z', ''))) AS CrDate,  
       instancetype AS InstanceType,
       count(*) AS CreateCount
FROM myinventory.startinstance  
GROUP BY 1, 2  
ORDER BY CrDate DESC, CreateCount Desc  

Instance created and terminated list

SELECT T1.id AS InstanceID,  
       T1.createdate AS CreatedON,
       T1.instancetype AS InstanceType,
       T1.username AS CreatedBy,
       T1.region AS region,
       T2.createdate AS TerminatedON,
       T2.username AS TerminatedBy
FROM ailinventory.startinstance T1  
LEFT OUTER JOIN myinventory.stopinstance T2 ON T1.id = T2.id  
WHERE MONTH(CAST(REPLACE(REPLACE(T1.createdate, 'T', ' '), 'Z', '') AS DATETIME)) = 12  
ORDER BY T1.createdate DESC  

We end up with a report like this:

Hope you found this useful. Happy autoscaling! :)

comments powered by Disqus