Mahira Technology Private Limited's Blog
Monitoring Apache Logs on EC2 Instances with Amazon CloudWatch: A Step-by-Step Guide

Monitoring Apache Logs on EC2 Instances with Amazon CloudWatch: A Step-by-Step Guide

Mahira Technology Private Limited's photo
Mahira Technology Private Limited
·

4 min read

OverView :-

This guide will walk you through the process of setting up #CloudWatch Logs to collect and analyze #Apachelogs from an #EC2 instance. We’ll cover the necessary steps, from configuring the EC2 instance to creating #Log Groups and defining metrics. By the end of this document, you’ll have a robust #Apache log #monitoring solution in place, #enhancing your ability to manage your #EC2 instances effectively.

Pre-requestisites:-

  • Aws account with required permissions

Procedure :-

Step-1 :- First Login to your #Aws Management console and navigate to ec2.

Step-2 :- create an ec2 instance with below user-data for installing apache on ec2 instance and #monitoring the logs.

#!/bin/bash
sudo su root
echo "export APP_ENV=dev" >> /etc/environment
echo "export TYPE_SERVER=AWS" >> /etc/environment
echo { '"agent"': { '"metrics_collection_interval"': '10', '"run_as_user"': '"root"', '"logfile"': '"/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log"' }, '"logs"': { '"logs_collected"': { '"files"': { '"collect_list"': [{ '"file_path"': '"/var/www/html/mahira/var/log/**.log"', '"log_group_name"': '"mahira"', '"log_stream_name"': '"/var/www/html/mahira/var/log/"', '"timezone"': '"Local"', '"retention_in_days"': '30'}, {'"file_path"': '"/var/log/httpd/*/error_log*"','"log_group_name"': '"httpd_error_log"','"log_stream_name"': '"httpd_error_log"','"timezone"': '"Local"','"retention_in_days"': '30'}]}},'"log_stream_name"': '"application_stream"'},'"metrics"': {'"append_dimensions"': {'"InstanceId"': '"${aws:InstanceId}"'},'"metrics_collected"': {'"disk"': {'"measurement"': ['"used_percent"'],'"metrics_collection_interval"': '320','"resources"': ['"/"']}}}} > /tmp/amazon-cloudwatch-agent.json
sudo yum update -y amazon-linux-extras{ '"file_path"': '"/var/www/html/mahira/var/log/crons/five-minutes.log"','"log_group_name"': '"mahira-cron-five-minutes"','"log_stream_name"': '"mahira-cron-five-minutes"', '"timezone"': '"Local"','"retention_in_days"': '30'}, {'"file_path"': '"/var/www/html/mahira/var/log/crons/six-minutes.log"','"log_group_name"': '"mahira-cron-six-minutes"','"log_stream_name"': '"mahira-cron-six-minutes"','"timezone"': '"Local"','"retention_in_days"': '30'}, {'"file_path"': '"/var/log/httpd/*/error_log*"','"log_group_name"': '"httpd_error_log"','"log_stream_name"': '"httpd_error_log"','"timezone"': '"Local"','"retention_in_days"': '30'}]}},'"log_stream_name"': '"application_stream"'},'"metrics"': {'"append_dimensions"': {'"InstanceId"': '"${aws:InstanceId}"'},'"metrics_collected"': {'"disk"': {'"measurement"': ['"used_percent"'],'"metrics_collection_interval"': '320','"resources"': ['"/"']}}}} > /tmp/amazon-cloudwatch-agent.json
sudo yum update -y amazon-linux-extras
sudo yum install -y httpd httpd-tools mod_ssl wget gcc docker ruby php-devel ImageMagick-devel
sudo systemctl start httpd
sudo systemctl enable httpd
sudo sed -i 's/Environment=LANG=C/&\nEnvironment=TYPE_SERVER=AWS/' /usr/lib/systemd/system/httpd.service
systemctl daemon-reload
systemctl restart httpd
echo "Web Page" > /var/www/html/health.html
sudo yum install amazon-cloudwatch-agent -y
sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -s -c file:/tmp/amazon-cloudwatch-agent.json{ '"file_path"': '"/var/www/html/mahira/var/log/crons/fifteen-minutes.log"', '"log_group_name"': '"mahira-cron-fifteen-minutes"','"log_stream_name"': '"mahira-cron-fifteen-minutes"','"timezone"': '"Local"', '"retention_in_days"': '30' }, {'"file_path"': '"/var/log/httpd/mahira-app-access.log"','"log_group_name"': '"mahira-app-access"','"log_stream_name"': '"mahira-app-access"','"timezone"': '"Local"','"retention_in_days"': '30'}, {'"file_path"': '"/var/log/httpd/*/access_log*"','"log_group_name"': '"httpd_access_log"','"log_stream_name"': '"httpd_access_log"','"timezone"': '"Local"','"retention_in_days"': '30'}, {'"file_path"': '"/var/log/httpd/*/error_log*"','"log_group_name"': '"httpd_error_log"','"log_stream_name"': '"httpd_error_log"','"timezone"': '"Local"','"retention_in_days"': '

Step-3 :- Create the ec2 webserver with above configuration.

Step 4 :- Set Up #CloudWatch Logs Agent

Amazon provides a CloudWatch Logs Agent that you can install on your EC2 instances to send log data to CloudWatch Logs. Here's how to set it up:

  1. Install the CloudWatch Logs Agent:

  2. Connect to your #EC2 instance and follow the AWS documentation to install the CloudWatch Logs Agent.

  3. Configure the Agent:

  4. During the agent setup, you will be prompted to configure which logs you want to send to CloudWatch Logs. Be sure to specify the paths to your Apache log files (e.g., /var/log/httpd/access.log and /var/log/httpd/error.log) in the configuration.

  5. Start the Agent:

  6. Once configured, start the #CloudWatch Logs Agent.

7. Also Check the apache has being installed or not on the ec2 #web server by running the below command

sudo systemctl status httpd

Step 5 :- Create a #CloudWatch Logs Group

Now that the #CloudWatch Logs Agent is set up, you need to create a CloudWatch Logs group to receive the log data.

  1. Navigate to CloudWatch in the AWS Management Console.

  2. Click on "Logs" in the navigation pane.

  3. Click the "Create Log Group" button.

  4. Name your log group (e.g., "ApacheLogs") and click "Create".

Step 6 :- Create #Log Streams

Each EC2 instance should have its own log stream within the log group. The Log Streams are where the log data from each instance will be stored.

  1. Within your log group ("ApacheLogs"), click "Create Log Stream".

  2. Name your log stream. You might want to use the EC2 instance name or identifier to keep them organized.

  3. Repeat this process for each #EC2 instance you want to monitor.

  • As u can see the log groups in cloud watch and u can also monitor the logs events by clicking on “Search all log streams”.

Step 7 :- Define Metric Filters and Alarms (Optional)

You can set up metric filters to extract specific information from your logs and create #CloudWatch Alarms to trigger actions based on log data. For example, you could create an alarm that notifies you when the number of 404 errors in your #Apache logs exceeds a certain threshold.

Conclusion :-

Integrating #Apache logs with #Amazon CloudWatch empowers you with real-time visibility into your application’s behavior and performance. By centralizing log data, you can streamline troubleshooting, detect anomalies, and respond proactively to incidents.

In this guide, we’ve demonstrated how to configure an #EC2 instance to send Apache logs to CloudWatch Logs, set up log groups, and create custom metrics and alarms based on log data. This powerful combination of #Apache and CloudWatch gives you the tools you need to maintain the #reliability and #security of your applications.

EC2 instanceApache LogsAmazon CloudwatchCloud Computingapachemonitoring#CloudWatchLogsAWS