ElasticSearch Daily Backup and Snapshot Deletion Using Lambda Function

Introduction:

In today’s cloud-driven landscape, data management and backup strategies are essential for ensuring the integrity and availability of critical information. ElasticSearch, a popular distributed search and analytics engine, plays a pivotal role in handling vast volumes of data. However, to safeguard this data and ensure its recoverability in the face of unexpected events, regular backups and snapshot management are imperative. This is where AWS Lambda, a serverless compute service, comes into play. This guide outlines the creation of an AWS Lambda function with the primary purpose of automating daily backups of an ElasticSearch cluster and the removal of snapshots older than seven days. The Lambda function is designed to simplify the backup and snapshot management process, enhancing data resilience in the AWS environment.

Purpose: Describe the primary purpose of the #Lambda function, which is to perform daily backups of an #ElasticSearch cluster and delete snapshots older than 7 days.

Step 1: Sign in to AWS Console

Step 2: Create a Lambda Function

2.1. In the AWS Management Console, navigate to the Lambda service.

2.2. Click the “Create function” button.

2.3. Choose “Author from scratch.”

2.4. Configure the following settings:

Function name: Provide a unique name for your Lambda function (e.g., “ElasticSearchBackup”).
Runtime: Select the runtime for your function (e.g., Python 3.11).
Execution role: Use existing role with the necessary permissions for Cloudwatch , S3 and ES. And and the below inline permission also.

{
 "Version": "2012–10–17",
 "Statement": [
 {
 "Effect": "Allow",
 "Action": "iam:PassRole",
 "Resource": "arn:aws:iam::account-id:role/role-name"
 },
 {
 "Effect": "Allow",
 "Action": "es:ESHttpPut",
 "Resource": "arn:aws:es:ca-central-1:account-id:domain/domain-name/*"
 }
 ]
}

2.5. Click the “Create function” button.

2.6. Upload the zip file which contain the lambda funtion code and necessary permission.

import boto3
import requests
from requests_aws4auth import AWS4Auth
import datetime
import time
def lambda_handler(event, context):
 day = datetime.date.today()
 today = time.strftime("%Y-%m-%d")
 seven_days_ago = (day - datetime.timedelta(days=7)).strftime("%Y-%m-%d")

 host = 'https://domain.ca-central-1.es.amazonaws.com/'
 region = 'ca-central-1'
 service = 'es'
 credentials = boto3.Session().get_credentials()
 awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

 path = '_snapshot/s3-backup'
 url = host + path

 payload = {
 "type": "s3",
 "settings": {
 "bucket": "s3-backup",
 "region": "ca-central-1",
 "role_arn": "arn:aws:iam::123456789:role/efs_lambda_backup_role"
 }
 }

 headers = {"Content-Type": "application/json"}

 r = requests.put(url, auth=awsauth, json=payload, headers=headers)

 print(r.status_code)
 print(r.text)

 path = '_snapshot/s3-backup/' + today + '/'
 url = host + path

 r = requests.put(url, auth=awsauth)

 print(r.text)

 list_snapshots_url = host + '_snapshot/s3-backup/_all'

 r = requests.get(list_snapshots_url, auth=awsauth)
 snapshots = r.json().get('snapshots', [])

 for snapshot in snapshots:
 snapshot_name = snapshot['snapshot']
 snapshot_date = datetime.datetime.strptime(snapshot_name, '%Y-%m-%d')

 if snapshot_date < datetime.datetime.strptime(seven_days_ago, '%Y-%m-%d'):
 delete_snapshot_url = host + '_snapshot/s3-backup/' + snapshot_name
 r = requests.delete(delete_snapshot_url, auth=awsauth)
 print(f"Deleted snapshot {snapshot_name}")

Step 3: Configure the Function

3.1. In the “Basic settings” section, you can configure your function’s memory, timeout, and other settings.

3.2. Configure the trigger for your Lambda function (e.g., CloudWatch Events for scheduled executions).

Step 4: Save and Test

4.1. Click the “Save” button to save your Lambda function configuration.

4.2. You can test your Lambda function by clicking the “Test” button, creating a new test event, and invoking it manually.

That’s it! You’ve created a Lambda function, written code and uploaded the function to #AWS Lambda. Adjust your code and dependencies as needed for your specific #ElasticSearch backup and snapshot deletion requirements.

Conclusion :-

In the modern era of cloud computing, automation is key to efficient data management and disaster recovery. The AWS Lambda function detailed in this guide provides a robust solution for safeguarding ElasticSearch data. By automating daily backups and pruning old snapshots, it not only ensures data availability but also optimizes storage utilization. This approach empowers organizations to focus on their core operations with the confidence that their ElasticSearch data is protected and recoverable. AWS Lambda, with its serverless architecture, offers a cost-effective and scalable way to manage ElasticSearch data, enhancing data resilience and streamlining operations. As organizations continue to rely on AWS for their infrastructure needs, tools like AWS Lambda prove invaluable in simplifying and securing their data management processes.