Mahira Technology Private Limited's Blog
Deep Dive: Harnessing AWS Elasticsearch Snapshot and Restore with Amazon S3

Deep Dive: Harnessing AWS Elasticsearch Snapshot and Restore with Amazon S3

Mahira Technology Private Limited's photo
Mahira Technology Private Limited
·

2 min read

Overview:-

Prerequisites

  1. AWS account with required permissions

Amazon S3 bucket

Stores manual snapshots for your #Amazon ES domain. Make a note of the bucket’s name. You need it in two places:

  • Resource statement of the #IAM policy that is attached to your #IAM role

  • Python client that is used to register a #snapshot repository.

Create a bucket in Amazon S3.

my bucket name : elasticsearch-backup-indices

Once the bucket is created get the bucket arn

arn:aws:s3:::elasticsearch-backup-indices

AWS Identity and Access Management (IAM)

Delegates permissions to #Amazon Elasticsearch Service. The rest of this chapter refers to this role as TheSnapshotRole.

The trust relationship for the role must specify #Amazon Elasticsearch Service in thePrincipal statement.

To create this #IAM policy, open the #IAM console, switch to the Policies tab, and choose Create Policy. Select Create Your Own Policy, and give your policy a name.Ex : elasticsearchbackup-policy

  • Attach the following permissions, Make sure to change the bucket ARNs
{ 
 "Version": "2012–10–17", 
 "Statement": [ 
 { 
 "Sid": "VisualEditor0", 
 "Effect": "Allow", 
 "Action": [ 
 "iam:PassRole", 
 "s3:ListBucket" 
 ], 
 "Resource": [ 
 "arn:aws:iam::YOUR-ACCOUNT-ID:role/es-s3-backup", 
 "arn:aws:s3:::elasticsearch-backup-indices" 
 ] 
 }, 
 { 
 "Sid": "VisualEditor1", 
 "Effect": "Allow", 
 "Action": [ 
 "s3:PutObject", 
 "s3:GetObject", 
 "s3:DeleteObject" 
 ], 
 "Resource": "arn:aws:s3:::elasticsearch-backup-indices/*" 
 }, 
 { 
 "Sid": "VisualEditor2", 
 "Effect": "Allow", 
 "Action": "es:ESHttpPost", 
 "Resource": "arn:aws:es:region:YOUR-ACCOUNT-ID:domain/YOU-ELASTIC-SEARCH-DOMAIN-NAME" 
 } 
 ] 
}

This policy document grants list, get, put, and delete object permissions to whomever assumes the role to which it is attached. When you’re done, choose Create Policy.

  • Attach the following trust relationship to the role
{ 
 "Version": "2012–10–17", 
 "Statement": [{ 
 "Sid": "", 
 "Effect": "Allow", 
 "Principal": { 
 "Service": "es.amazonaws.com" 
 }, 
 "Action": "sts:AssumeRole" 
 }] 
 }

Registering a Manual Snapshot Repository

You must register a snapshot repository with #Amazon Elasticsearch Service before you can take manual index snapshots. This one-time operation requires that you sign your AWS request with credentials that are allowed to access

You use Elasticsearch’s _snapshot API action to register a repository with #Amazon ES.

import boto3
import requests
from requests_aws4auth import AWS4Auth
host = '' # domain endpoint with trailing /
region = ''
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
# Register repository
path = '_snapshot/dev_carecore' # the OpenSearch API endpoint
url = host + path
payload = {
"type": "s3",
"settings": {
"bucket": "dev-carecore-backup01",
"region": "ca-central-1",
"role_arn": "arn:aws:iam::123456789:role/dev_lambda"
        }
  }
headers = {"Content-Type": "application/json"}
r = requests.put(url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)
path = '_snapshot/dev_carecore/dev_carecore-2023–08–07'
url = host + path
r = requests.put(url, auth=awsauth)
print(r.text)
Run the script
chmod 700 /tmp/register-repo.py 
python /tmp/register-repo.py
AWSAmazon S3elasticsearchCloud ComputingawscloudPythoncoding