Query Data Quickly using Amazon OpenSearch

Introduction

Amazon OpenSearch is an open-source, community-driven OpenSearch project. It is available as both a managed and an unmanaged service on AWS. In September 2021, Amazon changed the name of its Amazon Elasticsearch Service to Amazon OpenSearch Service. Since then, the service has supported several new capabilities that OpenSearch has received in various updates. Some examples support cross-cluster replication, trace analytics, data streams, transforms, a new observability user interface, and notebooks in OpenSearch Dashboards. In addition to this, there have been significant improvements to k-NN, anomaly detection, PPL, SQL, and alerting.

How does it work?

To understand how Amazon OpenSearch works, let’s take an example of an application server generating lots of application data. These data can be sent directly to Amazon OpenSearch via an HTTP endpoint or other services such as Amazon Kinesis Data Firehose, Logstash, AWS IoT, etc. The Lucene index is generated at the backend when the data gets into OpenSearch.  Lucene index falls into the family of indexes known as an inverted index. That is because it can list the documents that contain that particular term. This is the inverse of the natural relationship, in which documents list terms. When a client opens the Amazon OpenSearch Dashboard generates a set of queries. These queries then go against the indices and search for the relevant information based on the words in the query and return the result that is ranked based on relevance.

Architecture

Amazon OpenSearch consists of a cluster which can be a single-node or multi-node. There are multiple cluster design options. The above illustration shows a basic architecture that includes a four-node cluster with one dedicated cluster manager node, one dedicated coordinating node, and two data nodes that are cluster manager eligible and also used for ingesting data. Now, let’s discuss the types of nodes mentioned above.

• Cluster Manager Node: It manages the overall operation of a cluster and keeps track of the cluster state. This includes creating and deleting indexes, tracking which nodes join and leave the cluster, monitoring the health of each node in the cluster (via ping requests), and allocating shards to nodes.
• Data Node: It stores and searches data and carries out all data-related operations like indexing, searching, and aggregating on local shards. These are the cluster’s worker nodes and those that use the most disc space.
• Coordinating Node: It delivers client requests to the shards on the data nodes and collects and aggregates the results into a final result before returning it to the client.

Features

AWS manages the software installation, upgrades, patching, scaling, and cross-region replication with no downtime. The Amazon OpenSearch service also includes a dashboard visualization tool called OpenSearch Dashboards, which helps visualize not only log and trace data but also machine-learning powered results for anomaly detection and search relevance ranking. Amazon OpenSearch Service delivers powerful transformational benefits for the entire organization, including:

• Managed: Fully increased operational excellence while using a popular open-source solution.
• Secure: Audit and secure the data with a data center and network architecture with built-in certifications.

• Observability: Systematically detect potential threats and react to a system’s state through an open-source machine learning, alerting, and visualization solution.
• Cost-conscious: Optimize time and resources for strategic work.
• Built-in Search Capabilities: It offers a number of features to help customize the search experience, such as full-text querying, autocomplete, scroll search, customizable scoring and ranking, and more.

Use Cases

Amazon OpenSearch can be used in real-time search and log analytics at scale. Some examples of this service are:

• Monitor and debug applications and infrastructure:

Easily store and analyze data for comprehensive visibility into the system performance with observability logs, metrics, and traces. Set up automated alerts when the system underperforms and identify the underlying cause of availability problems.

• Manage security and event information (SIEM):

For real-time threat detection and incident management, centralized logs of various applications and systems across the network are analyzed.

• Enable seamless, personalized search:

Help users find relevant data with a fast, personalized search experience within the applications, websites, and data lake catalogs.

• Observability:

Efficiently detect and fix problems, improve application health, and deliver better customer experiences.

Conclusion

Amazon OpenSearch serverless is a major new initiative from Amazon that allows businesses to search their data without worrying about managing the servers.

About CloudThat