A Guide to Stream Amazon CloudWatch Logs Data to Amazon OpenSearch Service

Introduction

The powerful and fully managed Amazon OpenSearch service enables developers to integrate search and analytics features into their applications seamlessly. This service is derived from the open-source Elasticsearch search and analytics engine and ensures a secure and scalable user search environment.

Features of Amazon OpenSearch

• Scalability – Amazon OpenSearch offers the flexibility to expand or shrink its resources based on data volume and query load. It can handle vast amounts of data and numerous queries per second.
• Open-source foundation – Built upon the open source Elasticsearch and Kibana technologies, Amazon OpenSearch benefits from a thriving community of contributors and users. This collaborative environment ensures continuous enhancements and feature additions.
• Simplified Setup and Management – With complete management provided by AWS, the complexities of infrastructure and administrative tasks are abstracted. This empowers users to prioritize their data and applications without the burden of server upkeep or updates.
• Real-time Data Analysis – Amazon OpenSearch facilitates the real-time interpretation of data, enabling the prompt identification of emerging trends and irregularities as they unfold.
• Data Visualization – Including Kibana within Amazon OpenSearch furnishes users with a potent instrument for crafting personalized dashboards and generating insightful reports.
• Robust Security – Comprehensive security features are an integral part of Amazon These include data encryption during transit and storage, meticulous access controls, and seamless integration with AWS Identity and Access Management (IAM).

Steps to Stream logs to Amazon OpenSearch

Step 1 – Create an Amazon OpenSearch cluster with the configuration required

Step 2 – Go to Amazon CloudWatch in the AWS console and select log groups

Create a log group if required, or if having an existing log group move to step 4.

Create a log stream and in the log stream, select log events and create a log event.

Step 3 – Go to AWS IAM roles and create a role for AWS Lambda execution

For the created role, attach Cloudwatchlogsreadonlyaccess, AWS Lambda execute and create inline policy for opensearchservice write.

Step 4 – Inside the log groups, create a subscription filter for Amazon OpenSearch

Select the Amazon OpenSearch cluster you want to stream and select the AWS Lambda execution role created in Step 3

Create the filter after entering the name of the filter.

Step 5 – Copy the OpenSearch domain URL from the AWS console and enter the master username and password given when creating the OpenSearch cluster

Go to security, select all_access from the given roles, enter the role arn created in step 3 under backend roles, and map the role arn.

Step 6 – Go to AWS Lambda in the console and select the function created by the subscription filter.

Create a test event to check if the logs are getting streamed to the OpenSearch cluster.

Once tested and AWS Lambda logs shows success. Go to OpenSearch Dashboard

Step 7 – In OpenSearch, go to index management and check for the logs getting streamed under the specific date.

If Amazon CloudWatch already has applications or services logs, it will directly get streamed to OpenSearch.
Note: Try creating a log event in log streams for the data ingested in OpenSearch.

Example: # 2023-07-25T12:34:56.789Z This is a sample log message.

Step 8 – In the OpenSearch dashboard, go to stack management and create the index pattern you want to see as a graph in the Discover dashboard. Once created, go to Discover and see the required index created.

Conclusion

Amazon Web Services (AWS) offers a robust and scalable search and analytics solution called Amazon OpenSearch. OpenSearch, a development of Amazon’s well-liked Elasticsearch service, offers users looking to create and maintain search applications in the cloud advanced capabilities and enhanced security. Amazon OpenSearch provides developers and business leaders with the necessary resources to swiftly and effortlessly develop powerful search and analytics applications, leveraging the robust infrastructure and extensive ecosystem offered by AWS.

Drop a query if you have any questions regarding Amazon OpenSearch and we will get back to you quickly.

About CloudThat