Enable Alert for an Error Pattern on Elasticbeanstalk Environment

In this blog, we are showing on how to get notified if particular error pattern occurs in Elasticbeanstalk environment using Cloudformation Template.

Here we have chosen the Application which runs on Java 8 Tomcat 8.

WorkFlow

  • Provision Elasticbeanstalk using Cloudformation template
  • Push the required application logs to Cloudwatch Logs
  • Create Metric Filter using the required Error Pattern
  • Enable Cloudwatch Alarm on the respective metric
  • Install Cloudwatch logs agent in the server using the script in .ebextensions

Provision the environment using Cloudformation Template

Refer to the template (beanstalk-custom-logs.yaml) here → https://github.com/powerupcloud/CloudFormation_Templates.git
The following AWS Resources are being created by this template:

  • IAM EC2 Policy: which have permissions to create the log group, put logs and create a log stream into it.
    RolePolicies:
        Properties:
          PolicyDocument:
            Statement:
            - Action: [
                    "logs:PutLogEvents",
                    "logs:CreateLogGroup",
                    "logs:CreateLogStream"
                ]
              Effect: Allow
              Resource: '*'
            Version: '2012-10-17'
          PolicyName: root
          Roles:
          - Ref: RootRole
        Type: AWS::IAM::Policy
  • IAM EC2 Service Role: with the attached above inline policy
    RootRole:
        Properties:
          RoleName: "ec2role"
          AssumeRolePolicyDocument:
            Statement:
            - Action:
              - sts:AssumeRole
              Effect: Allow
              Principal:
                Service:
                - ec2.amazonaws.com
            Version: '2012-10-17'
          Path: /
  • IAM Instance Profile: to be assigned to the EC2 instance. The profile will have the above IAM Role attached with it.
    RootInstanceProfile:
        Properties:
          Path: /
          Roles:
          - Ref: RootRole
        Type: AWS::IAM::InstanceProfile

Once this Instance Profile is assigned to the Instance, it will have the permissions to the log group created through the template.

  • CloudWatch Log Group: A log group will be created with name “TomcatErrorLogGroup”.
    CloudwatchLogGroup:
        Properties:
          LogGroupName: TomcatErrorLogGroup
          RetentionInDays: '14'
        Type: AWS::Logs::LogGroup
  • CloudWatch Metric Filter: The filter to be applied on the loggroup for which the notification will be sent.
    CWTomcatMetricFilter:
        DependsOn: TomcatEnvironment
        Properties:
          FilterPattern:
            Ref: FilterPattern
          LogGroupName:
            Ref: CloudwatchLogGroup
          MetricTransformations:
          - MetricName: CWLTomcatErrorMetric
            MetricNamespace: EBTomcatErrors
            MetricValue: 1
        Type: AWS::Logs::MetricFilter
  • SNS Topic: Topic subscribed to an email provided in parameters as input.
    AlarmNotificationTopic:
        Properties:
          Subscription:
          - Endpoint:
              Ref: AlarmEmail
            Protocol: email
          TopicName: CWTopic
        Type: AWS::SNS::Topic
  • Cloudwatch Alarm: Alarm will be created for the filter pattern. So, whenever the pattern matches, a notification will be sent to the email subscribed in the SNS topic.( The NameSpace and MetricnName can be changed as required).
    AWSEBCWLTomcatErrorPercentAlarm:
        DependsOn:
        - CWTomcatMetricFilter
        - AlarmNotificationTopic
        Properties:
          AlarmActions:
          - Ref: AlarmNotificationTopic
          AlarmDescription: Application is returning too many tomcat error responses (percentage
            too high).
          ComparisonOperator: GreaterThanThreshold
          EvaluationPeriods: '1'
          MetricName: CWLTomcatErrorMetric
          Namespace: EBTomcatErrors
          Period: '60'
          Statistic: Average
          Threshold: '0.10'
          Unit: Count
        Type: AWS::CloudWatch::Alarm
  • Beanstalk Application: an application will get created with the name “tomcat-app”. (The name of the application can be changed in the template as required) . A sample tomcat application version will be created.
    TomcatApplication:
        Properties:
          ApplicationName: tomcat-app
          ApplicationVersions:
          - Description: Version 1.0
            SourceBundle:
              S3Bucket:
                Fn::Join:
                - '-'
                - - elasticbeanstalk-samples
                  - Ref: AWS::Region
              S3Key: elasticbeanstalk-sampleapp.war
            VersionLabel: Initial Version
          Description: AWS Elastic Beanstalk Tomcat Application
        Type: AWS::ElasticBeanstalk::Application
  • Beanstalk Environment: An environment for the above application will be created with the sample tomcat application. Also, with this template, the Stickiness for the ELB will be enabled for both the ports 80 and 443.
    TomcatEnvironment:
        DependsOn:
        - TomcatApplication
        Properties:
          ApplicationName:
            Ref: TomcatApplication
          Description: AWS Elastic Beanstalk Environment running Tomcat Application
          EnvironmentName:
            Ref: EnvironmentName
          OptionSettings:
          - Namespace: aws:autoscaling:launchconfiguration
            OptionName: EC2KeyName
            Value:
              Ref: InstanceKeyName
          - Namespace: aws:autoscaling:launchconfiguration
            OptionName: IamInstanceProfile
            Value:
              Ref: RootInstanceProfile
          - Namespace: aws:autoscaling:launchconfiguration
            OptionName: InstanceType
            Value:
              Ref: InstanceType
          - Namespace: aws:elb:listener:443
            OptionName: InstancePort
            Value: '80'
          - Namespace: aws:elb:listener:443
            OptionName: InstanceProtocol
            Value: HTTP
          - Namespace: aws:elb:listener:443
            OptionName: ListenerProtocol
            Value: HTTPS
          - Namespace: aws:elb:listener:443
            OptionName: SSLCertificateId
            Value: xxxxxxxxxxxxxxxx
          - Namespace: aws:elb:policies:LBSessionStickinessPolicy
            OptionName: LoadBalancerPorts
            Value: 443,80
          - Namespace: aws:elb:policies:LBSessionStickinessPolicy
            OptionName: Stickiness Cookie Expiration
            Value: 0
          SolutionStackName: 64bit Amazon Linux 2016.09 v2.3.1 running Tomcat 8 Java 8
          VersionLabel: Initial Version
        Type: AWS::ElasticBeanstalk::Environment

Once you have the template available, go to the cloudformation console. Create a Stack and upload the template.
alt

Provide the parameters values.

alt

We have provided the Filter Pattern as “ERROR”. So whenever “ERROR” will be printed in the log stream, it will get filtered and a notification will be sent to the AlarmEmail value in parameters.
Click Next. Review and Create it.

alt

Once, the cloudformation stack will get created, our tomcat environment will be available with a sample application running on it.

alt

alt

Also, the cloudwatch log group “TomcatErrorLogGroup” with the metric filters will be available to which logs will be printed.

alt

So, Once the whole environment is ready for you with just a template, upload your tomcat application with the .ebextensions provided on this link here → https://github.com/powerupcloud/ebextensions.git
The .ebextensions has the following config files:

  • cwl-setup.config: It will install the cloudwatch log agent on the beanstalk server through which is used to publish the log data to cloudwatch.
  • cwl-tomcat-metrics.config: In this config file, you need to mention the log file and log group name to which the logs will get published.
    Mappings:
      CWLogs:
        TomcatErrorLogGroup:
          LogFile: "/var/log/tomcat8/catalina.out"
          TimestampFormat: "%d/%b/%Y:%H:%M:%S %z"

    Resources :
      ## Register the files/log groups for monitoring
      AWSEBAutoScalingGroup:
        Metadata:
          "AWS::CloudFormation::Init":
            CWLogsAgentConfigSetup:
              files:
                "/tmp/cwlogs/conf.d/tomcat-errors.conf":
                  content : |
                    [tomcat-error_log]
                    file = `{"Fn::FindInMap":["CWLogs", "TomcatErrorLogGroup", "LogFile"]}`
                    log_group_name = TomcatErrorLogGroup
                    log_stream_name = {instance_id}
                  mode  : "000400"
                  owner : root
                  group : root

So, here we have mentioned the log file as /var/log/tomcat8/catalina.out and the log group name
“TomcatErrorLogGroup” which is created by the template. (Ensure the log group name is same as created with the template. If you are changing the log group name in the template, you need to change the name here too).

  • eb-logs.config: It is used in the log management for Cloudwatch log agent. once the application is deployed, you can check the publishing logs in /var/log/awslogs.log.

Getting Alarm for the Matching Pattern

Go to the Beanstalk Console. Upload and Deploy your tomcat application with .ebextensions.

alt

It will start deploying the application to the environment.

alt

Once the application is deployed, we can see the cataline.out logs in the log stream created in the log group “TomcatErrorLogGroup”.

alt

To verify the alarm notification, log into the server and echo “ERROR” (the filter pattern created through the template) in the catalina.out.

echo "ERROR" >> /var/log/tomcat8/catalina.out  

Check the log stream for logs.

alt

Also, you can see the metric graph in cloudwatch:

alt

And you can verify with the publishing log in the server with log file /var/log/awslogs.log.

alt

Once the pattern matched with “ERROR”, the notification will be sent to the email provided.

and That’s it..!! Hope it will help in monitoring the error logs easily. Happy Logging..!! :)

Priyanka Sharma

Priyanka is Senior Cloud and DevOps Engineer. She can churn out CloudFormation templates at a moment's notice and play with Chef/Ansible. Dancing, music, badminton and word games are her hobbies

comments powered by Disqus