Cross-Account Migration: Retrieving and Copying Glacier Deep Archive Objects Between S3 Buckets using Ec2 instance

OverView :-

This document provides a step-by-step guide on how to retrieve objects from #AmazonS3. It assumes you have an AWS account and basic knowledge of #AWSServices.

Prerequisites :-

• Access to 2 AWS account

• S3 bucket in source and destination account.

• Sufficient permissions to interact with #Amazon S3

• Familiarity with the AWS Command Line Interface (CLI) or an SDK of your choice

• Knowledge of the #AmazonS3 and aws cli installed on your system.

Retrieve Objects from #AmazonS3:- Follow these steps to retrieve objects from Amazon S3.

Step-1:- First login to your AWS source account and open your s3 source bucket & then add the below bucket policy to the s3 bucket.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "",
            "Effect": "Allow",
            "Principal": {
                "AWS": "arn:aws:iam::123456789:root"
            },
            "Action": [
                "s3:RestoreObject",
                "s3:ListBucket",
                "s3:GetObject"
            ],
            "Resource": [
                "arn:aws:s3:::source-bucket/*",
                "arn:aws:s3:::source-bucket"
            ]
        }
    ]
}

Step-2 :- Next Login to your destination account and navigate to ec2 service & Create an #EC2 Key Pair and download the Keypair.

Step-3:- Open your Iam service and Create a new #IAM role. Ensure that the IAM role has the necessary permissions to access the S3 bucket and retrieve objects from #S3GlacierDeepArchive. Make sure that the role has below trust relation-ship policy and s3 full access.

#trust relation-ship policy
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Service": "ec2.amazonaws.com"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

Step-4:- Launch an EC2 Instance with the Ec2 keypair and Iam role which are being created above and attach to that ec2 instance. Make sure that the instance is running in the same AWS region as your S3 Glacier Deep Archive bucket.

Step-5 :-Check If the AWS Command Line Interface (CLI) tool is not already installed, you can follow the AWS CLI installation instructions relevant to your operating system. Configure the AWS credentials of destination s3 account. While connected to the EC2 instance, execute the command aws configure to properly configure the AWS CLI. Provide the necessary AWS access key, secret access key, default region, and preferred output format. Access keys can be obtained from the IAM section of the #AWSManagementConsole.

Step-6:- Establish a connection to the EC2 instance using SSH or another remote access method.

Step-7:- Retrieve Objects from #S3 Glacier Deep Archive: you can utilize the following command to list the objects stored in the #S3Glacier Deep Archive bucket:

aws s3api list-objects --bucket YOUR-BUCKET-NAME --query 'Contents[?StorageClass==`DEEP_ARCHIVE`].{Key: Key}' --output text > output.txt

7. Replace YOUR_BUCKET_NAME with the name of your S3 bucket. This command will list the objects stored in the specified bucket, including those archived in S3 Glacier Deep Archive.

8. Next Use the below command to restore the objects in s3 —

cat object-list.txt | while read key; do
    aws s3api restore-object --bucket YOUR-BUCKET-NAME --key "$key" --restore-request '{"Days":2,"GlacierJobParameters":{"Tier":"Bulk"}}'
done

Replace your-bucket-name with the name of your S3 bucket and $key with the key of the object you want to retrieve.

• Generally the objects will be retrived after 48 hours with Bulk retrivel.

7. Once the objects are being retrived, u can copy or sync the Objects from the source bucket to destination bucket using below command.

aws s3 sync s3://SOURCE_BUCKET s3://DESTINATION_BUCKET --force-glacier-transfer

Go to the destination account’s s3 bucket and check whether the objects are being copied or not.

Conclusion :-

By following these corrected steps, you’ll be able to successfully interact with and retrieve objects from your S3 Glacier Deep Archive bucket using the EC2 instance and the AWS CLI. #AWS #AmazonS3 #DataStorage #AWSCLI #DataManagement