Login
May 25, 2023|
Voiced by Amazon Polly |
Introduction
Developers may easily incorporate search and analytics functionality into their apps due with the help of the robust and fully managed Amazon OpenSearch service. It is a fork of the well-known open-source Elasticsearch search and analytics engine and is made to offer users a safe and scalable search experience.
Features of Amazon OpenSearch
- Scalability: AWS OpenSearch can scale up or down according to the size of the data and the number of queries. It can handle petabytes of data and thousands of queries per second.
- Open-source technology: AWS OpenSearch is built on open-source Elasticsearch and Kibana technology, which means it has a large community of contributors and users constantly improving and adding features to the platform.
- Easy setup and management: AWS OpenSearch is fully managed, which means AWS takes care of the infrastructure and management tasks. This lets you focus on your data and applications instead of worrying about server maintenance and upgrades.
- Real-time data analytics: AWS OpenSearch enables real-time data analysis, making it possible to identify trends and anomalies as they occur.
- Data visualization: AWS OpenSearch includes Kibana, a powerful data visualization tool that allows you to create custom dashboards and reports.
- Security: AWS OpenSearch includes various security features, including data encryption in transit and at rest, fine-grained access controls, and integration with AWS Identity and Access Management (IAM).
Pioneers in Cloud Consulting & Migration Services
- Reduced infrastructural costs
- Accelerated application deployment
Steps to migrate data from one cluster to another in a different region
Step 1: Create two OpenSearch clusters in a different region
Step 2: Create two Amazon S3 buckets in the regions as the OpenSearch cluster
Indices in the source OpenSearch cluster
Step 3: Create a policy for allowing permission for Amazon S3 buckets
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 |
#{ "Version": "2012-10-17", "Statement": [ { "Action": [ "s3:ListBucket" ], "Effect": "Allow", "Resource": [ "<source bucket arn>", "<destination bucket arn>" ] }, { "Action": [ "s3:GetObject", "s3:PutObject", "s3:DeleteObject" ], "Effect": "Allow", "Resource": [ "<source bucket arn/*>", "<destination bucket arn>/*" ] } ] } |
Attach the policy to a role
Step 4: Create an inline policy for allowing https for source and destination clusters and attach it to the IAM user
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
# { Â Â Â "Version": "2012-10-17", Â Â Â "Statement": [ Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â "Effect": "Allow", Â Â Â Â Â Â Â Â Â Â Â "Action": "iam:PassRole", Â Â Â Â Â Â Â Â Â Â Â "Resource": "<role arn>" Â Â Â Â Â Â Â }, Â Â Â Â Â Â Â { Â Â Â Â Â Â Â Â Â Â Â "Effect": "Allow", Â Â Â Â Â Â Â Â Â Â Â "Action": "es:ESHttpPut", Â Â Â Â Â Â Â Â Â Â Â "Resource": [ Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "<opensearch-source arn./*", Â Â Â Â Â Â Â Â Â Â Â Â Â Â Â "<opensearch-destination arn>/*" Â Â Â Â Â Â Â Â Â Â Â ] Â Â Â Â Â Â Â } Â Â Â ] } |
Step 5: In the source cluster map, the user under security
Add: IAM arn under users and IAM role under backend role
Step 6: Create an instance and SSH into it
Step 7: Create a file and save it with the following code changing endpoint URL, bucket name, and role ARN
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 |
#vi /tmp/source.py import boto3 import requests from requests_aws4auth import AWS4Auth host = 'opensearch cluster end point/' # domain endpoint with trailing / region = 'us-east-1' # e.g. us-west-1 service = 'es' credentials = boto3.Session().get_credentials() awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, region, service, session_token=credentials.token) # Register repository path = '_snapshot/manual-snapshot-repo' # the OpenSearch API endpoint url = host + path payload = { "type": "s3", "settings": { "bucket": "<source bucket name>", "region": "us-east-1", "role_arn": "<arn role" } } headers = {"Content-Type": "application/json"} r = requests.put(url, auth=awsauth, json=payload, headers=headers) print(r.status_code) print(r.text) #sudo yum -y install python-pip #sudo pip install boto3 #sudo pip install requests-aws4auth #chmod 700 /tmp/source.py |
Step 8: Register the snapshot
|
1 |
#sudo python3 /tmp/source.py |
Step 9: Login into the source cluster and go to dev tools. Enter the following
|
1 2 3 |
#GET _snapshot #PUT _snapshot/manual-snapshot-repo/migration-snapshot #GET _snapshot/_status |
Step 10: Go to IAM policy, create a replication policy, and attach it to a role
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 |
#{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ "<source bucket arn>" ] }, { "Effect": "Allow", "Action": [ "s3:*" ], "Resource": [ " <source bucket arn> /*" ] }, { "Effect": "Allow", "Action": [ "s3:*" ], "Resource": " <destination bucket arn> /*" } ] } |
Step 11: Create a replication role for the source bucket –> select the destination bucket –> attach the replication role create –> select replicate existing objects
Wait till the replication is completed
Step 12: Repeat the same steps from step 5 to step 8 in the destination region
Step 13: Login into the destination cluster and go to dev tools
|
1 2 3 |
# GET _snapshot/manual-snapshot-repo/_all?pretty # POST _snapshot/manual-snapshot-repo/migration-snapshot/_restore {"indices": "movies-index"} |
You can see your restored index under–> Index management –> indices
Conclusion
AWS OpenSearch is a sophisticated and fully managed search and analytics service that allows developers to include its functionality in their applications. OpenSearch is a great option for businesses that deal with big amounts of data and need to search and analyze it fast and effectively due to its comprehensive search and analytics tools, scalability, and security capabilities.
Making IT Networks Enterprise-ready – Cloud Management Services
- Accelerated cloud migration
- End-to-end view of the cloud environment
About CloudThat
CloudThat is an official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft Gold Partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best in industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.
Drop a query if you have any questions regarding AWS OpenSearch and I will get back to you quickly.
To get started, go through our Consultancy page and Managed Services Package that is CloudThat’s offerings.
FAQs
1. How does AWS OpenSearch work?
ANS: – AWS OpenSearch works by ingesting data from various sources into a cluster, which then indexes the data and makes it searchable using a powerful search engine. Users can then query the data and visualize the results using the OpenSearch API or a web interface.
2. What are some use cases for AWS OpenSearch?
ANS: – AWS OpenSearch can be used for various use cases, including e-commerce search, log analytics, security analytics, and website search.
3. How does AWS OpenSearch differ from Elasticsearch?
ANS: – AWS OpenSearch is based on Elasticsearch, but it is a fully managed service provided by AWS. This means that AWS handles the service’s infrastructure, scaling, and maintenance while users can focus on using the service to analyze their data.
WRITTEN BY Swapnil Kumbar
Swapnil Kumbar is a Research Associate - DevOps. He knows various cloud platforms and has working experience on AWS, GCP, and azure. Enthusiast about leading technology in cloud and automation. He is also passionate about tailoring existing architecture.
PREV
Comments