A Guide to Amazon OpenSearch Data Cross-Region Migration

Introduction

Developers may easily incorporate search and analytics functionality into their apps due with the help of the robust and fully managed Amazon OpenSearch service. It is a fork of the well-known open-source Elasticsearch search and analytics engine and is made to offer users a safe and scalable search experience.

Developers can index and search huge amounts of data in real-time with OpenSearch and utilize cutting-edge analytics to learn more about their data. It is especially helpful for businesses that deal with a lot of data and must rapidly and effectively search and analyze it.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Features of Amazon OpenSearch

Scalability: AWS OpenSearch can scale up or down according to the size of the data and the number of queries. It can handle petabytes of data and thousands of queries per second.
Open-source technology: AWS OpenSearch is built on open-source Elasticsearch and Kibana technology, which means it has a large community of contributors and users constantly improving and adding features to the platform.
Easy setup and management: AWS OpenSearch is fully managed, which means AWS takes care of the infrastructure and management tasks. This lets you focus on your data and applications instead of worrying about server maintenance and upgrades.
Real-time data analytics: AWS OpenSearch enables real-time data analysis, making it possible to identify trends and anomalies as they occur.
Data visualization: AWS OpenSearch includes Kibana, a powerful data visualization tool that allows you to create custom dashboards and reports.
Security: AWS OpenSearch includes various security features, including data encryption in transit and at rest, fine-grained access controls, and integration with AWS Identity and Access Management (IAM).

Steps to migrate data from one cluster to another in a different region

Step 1: Create two OpenSearch clusters in a different region

step1

step1b

Step 2: Create two Amazon S3 buckets in the regions as the OpenSearch cluster

step2

step2b

Indices in the source OpenSearch cluster

step2c

step2d

step2e

Step 3: Create a policy for allowing permission for Amazon S3 buckets

#{
"Version": "2012-10-17",
"Statement": [
{
"Action": [
"s3:ListBucket"
],
"Effect": "Allow",
"Resource": [
"<source bucket arn>",
"<destination bucket arn>"
]
},
{
"Action": [
"s3:GetObject",
"s3:PutObject",
"s3:DeleteObject"
],
"Effect": "Allow",
"Resource": [
"<source bucket arn/*>",
"<destination bucket arn>/*"
]
}
]
}

"Version": "2012-10-17",

"Statement": [

{

"Action": [

"s3:ListBucket"

"Effect": "Allow",

"Resource": [

"<source bucket arn>",

"<destination bucket arn>"

]

{

"Action": [

"s3:GetObject",

"s3:PutObject",

"s3:DeleteObject"

"Effect": "Allow",

"Resource": [

"<source bucket arn/*>",

"<destination bucket arn>/*"

]

}

]

}

step3

Attach the policy to a role

step3b

Step 4: Create an inline policy for allowing https for source and destination clusters and attach it to the IAM user

# {

    "Version": "2012-10-17",

    "Statement": [

        {

            "Effect": "Allow",

            "Action": "iam:PassRole",

            "Resource": "<role arn>"

        },

        {

            "Effect": "Allow",

            "Action": "es:ESHttpPut",

            "Resource": [

                "<opensearch-source arn./*",

                "<opensearch-destination arn>/*"

            ]

        }

    ]

}

# {

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": "iam:PassRole",

"Resource": "<role arn>"

{

"Effect": "Allow",

"Action": "es:ESHttpPut",

"Resource": [

"<opensearch-source arn./*",

"<opensearch-destination arn>/*"

]

}

]

}

step4

step4b

Step 5: In the source cluster map, the user under security

Add: IAM arn under users and IAM role under backend role

step5

step5b

Step 6: Create an instance and SSH into it

step6

Step 7: Create a file and save it with the following code changing endpoint URL, bucket name, and role ARN

#vi /tmp/source.py
import boto3
import requests
from requests_aws4auth import AWS4Auth
host = 'opensearch cluster end point/' # domain endpoint with trailing /
region = 'us-east-1' # e.g. us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)
# Register repository
path = '_snapshot/manual-snapshot-repo' # the OpenSearch API endpoint
url = host + path
payload = {
  "type": "s3",
  "settings": {
    "bucket": "<source bucket name>",
    "region": "us-east-1",
    "role_arn": "<arn role"
  }
}
headers = {"Content-Type": "application/json"}
r = requests.put(url, auth=awsauth, json=payload, headers=headers)
print(r.status_code)
print(r.text)
#sudo yum -y install python-pip
#sudo pip install boto3
#sudo pip install requests-aws4auth
#chmod 700 /tmp/source.py

#vi /tmp/source.py

import boto3

import requests

from requests_aws4auth import AWS4Auth

host = 'opensearch cluster end point/' # domain endpoint with trailing /

region = 'us-east-1' # e.g. us-west-1

service = 'es'

credentials = boto3.Session().get_credentials()

awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token)

# Register repository

path = '_snapshot/manual-snapshot-repo' # the OpenSearch API endpoint

url = host + path

payload = {

"type": "s3",

"settings": {

"bucket": "<source bucket name>",

"region": "us-east-1",

"role_arn": "<arn role"

}

headers = {"Content-Type": "application/json"}

r = requests.put(url, auth=awsauth, json=payload, headers=headers)

print(r.status_code)

print(r.text)

#sudo yum -y install python-pip

#sudo pip install boto3

#sudo pip install requests-aws4auth

#chmod 700 /tmp/source.py

step7

step7b

step7c

step7d

Step 8: Register the snapshot

#sudo python3 /tmp/source.py

1	#sudo python3 /tmp/source.py

step8

Step 9: Login into the source cluster and go to dev tools. Enter the following

#GET _snapshot
#PUT _snapshot/manual-snapshot-repo/migration-snapshot
#GET _snapshot/_status

#GET _snapshot

#PUT _snapshot/manual-snapshot-repo/migration-snapshot

#GET _snapshot/_status

step9

step9b

step9c

Step 10: Go to IAM policy, create a replication policy, and attach it to a role

#{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                "<source bucket arn>"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": [
                " <source bucket arn> /*"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:*"
            ],
            "Resource": " <destination bucket arn> /*"
        }
    ]
}

"Version": "2012-10-17",

"Statement": [

{

"Effect": "Allow",

"Action": [

"s3:*"

"Resource": [

"<source bucket arn>"

]

{

"Effect": "Allow",

"Action": [

"s3:*"

"Resource": [

" <source bucket arn> /*"

]

{

"Effect": "Allow",

"Action": [

"s3:*"

"Resource": " <destination bucket arn> /*"

}

]

}

step10

step10b

step10c

Step 11: Create a replication role for the source bucket –> select the destination bucket –> attach the replication role create –> select replicate existing objects

step11

step11b

step11c

Wait till the replication is completed

step11d

Step 12: Repeat the same steps from step 5 to step 8 in the destination region

step12

step12b

Step 13: Login into the destination cluster and go to dev tools

# GET _snapshot/manual-snapshot-repo/_all?pretty
# POST _snapshot/manual-snapshot-repo/migration-snapshot/_restore
{"indices": "movies-index"}

# GET _snapshot/manual-snapshot-repo/_all?pretty

# POST _snapshot/manual-snapshot-repo/migration-snapshot/_restore

{"indices": "movies-index"}

step13

You can see your restored index under–> Index management –> indices

step13b

Conclusion

AWS OpenSearch is a sophisticated and fully managed search and analytics service that allows developers to include its functionality in their applications. OpenSearch is a great option for businesses that deal with big amounts of data and need to search and analyze it fast and effectively due to its comprehensive search and analytics tools, scalability, and security capabilities.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

FAQs

1. How does AWS OpenSearch work?

ANS: – AWS OpenSearch works by ingesting data from various sources into a cluster, which then indexes the data and makes it searchable using a powerful search engine. Users can then query the data and visualize the results using the OpenSearch API or a web interface.

2. What are some use cases for AWS OpenSearch?

ANS: – AWS OpenSearch can be used for various use cases, including e-commerce search, log analytics, security analytics, and website search.

3. How does AWS OpenSearch differ from Elasticsearch?

ANS: – AWS OpenSearch is based on Elasticsearch, but it is a fully managed service provided by AWS. This means that AWS handles the service’s infrastructure, scaling, and maintenance while users can focus on using the service to analyze their data.

WRITTEN BY Swapnil Kumbar

Swapnil Kumbar is a Research Associate - DevOps. He knows various cloud platforms and has working experience on AWS, GCP, and azure. Enthusiast about leading technology in cloud and automation. He is also passionate about tailoring existing architecture.