Learning Cloud 014: How to Manage Logs in AWS

 

Hi there and welcome to the 14th episode of the Learning Cloud podcast. The place for all things Cloud and DevOps.  My name is Raf Calderon

I’m excited to be back Podcasting.  It’s been a while since I last recorded.  I needed time to clear my mind and organize my thoughts.  But hey I’m back and I’m glad and it feels good.  It feels like riding a bike.  You don’t forget it.

Today I want to talk about Logs, in particular how to manage Logs in the cloud.

Intro

Working with Logs is a challenging task.

Computing environments generate massive amount of log files that are filled with all kinds of information: System Logs, application logs, security logs.  These files are being generated everyday, all year round.  Resulting in  multiple GBs of unstructured plain text data.

In most cases, manually going through plain log files, grepping for particular strings is daunting.

So we turn to professional applications to help us in this task.

There are products that are the Leaders in this market. They are Splunk and Elk

 

Splunk

  • – An Enterprise product sold and supported by Splunk Inc.  Targets large enterprise Market.
  • – Used a lot by security groups for long term archival.  To do forensics.
  • – It’s costly

 

Elk

  •  – An Opensource product.  ELK is an acronym for ElasticSearch, Logstash and Kibana.  
  •  – You can buy support from Elastic.
  •  – Since it’s ELK is an opensource it’s used by different companies. Mostly small but also large
  •  – I’ve seen it used in production to track app metrics in a more dynamic way

 

Transition:

Now I want to switch my focus to the cloud.  The loggin solution in AWS is called Cloud Watch.

Amazon Cloud Watch Logsis a managed service that enables you to store your EC2 logs in the cloud.

How does it Work

  • Agent
  • AWS Console

Some features

  • -Archiving/retention
  • -Metrics: see failed login attempts in system log
  • -Alarms

 

Demo you should try

If your’re interested in getting some hands on practice with CloudWatch logs this is easy.  This is why I love working with AWS

Launch an EC2 instance and send logs to it

  1. Launch an instance, with appropriate role
  2. Install the agent
  3. Configure the agent:
    1. Region
    2. Log file
    3. Log group name
    4. Stream name
  4. Watch the logs

 

I’ll put the link to the cloud watch Logs user guide where you can get more information.

Closing

That’s it for today. Please leave a comment on the post if you have enjoyed it., I love to hear your thoughts.  That’s it for today and remember

“If you’re not willing to learn no one can help you, if you’re determined to learn no one can stop you.”

 

Bye bye for now, and till next time.

Links

http://blog.takipi.com/splunk-vs-elk-the-log-management-tools-decision-making-guide/

http://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cwl-ug.pdf