Integrate Amazon SQS with AWS Lambda using Amazon EventBridge Pipes

Introduction

Now easily stitch AWS services together with Amazon EventBridge Pipes. We used to use the AWS Glue to communicate between the different services, which can be a time-consuming manual exercise but an easier way to connect AWS services called Amazon EventBridge Pipes.

Amazon EventBridge Pipes is an AWS service that can receive events from various AWS services like AWS SQS, AWS Kinesis, and Amazon DynamoDB. To reduce costs, filter patterns can be configured during the filtering step to exclude unnecessary events from further processing. In the enrichment step, data from the source can be enhanced using built-in transformations or advanced transformations performed by AWS Lambda, API Gateway, or Step Functions.

Enrichments are invoked synchronously, ensuring the timely availability of the enhanced data. Pipes serve as conduits to send events or data to targets such as AWS Lambda, API Gateway, ECS Cluster, CloudWatch Log, Kinesis Stream, or API destinations. Transformers can be written in the Target step to specify how data should be sent to the targets.

AWS CloudTrail tracks Pipe invocations and their details, while Amazon CloudWatch provides metrics to monitor the health status of Pipes.

Amazon SQS

Amazon SQS is a managed message queuing service that enables asynchronous communication between cloud application components. It supports two types of message queues(Standard and FIFO) and provides reliable, scalable, and decoupled messaging infrastructure for building cloud-based applications and microservices.

AWS Lambda

AWS Lambda is a serverless computing service provided by Amazon Web Services (AWS) that allows developers to run code without managing servers or infrastructure.

Amazon Lambda lets you execute your code in response to events such as changes in data, user actions, or file uploads. You can use Lambda to build custom APIs, process data streams, run ETL jobs, and automate tasks.

Steps on how Amazon SQS and AWS Lambda are integrated by using Amazon EventBridge Pipes

Step 1: Create source Amazon SQS

Step 2: Create AWS Lambda for Enrichment during the Amazon EventBridge Pipe

Step 3: Create Destination AWS Lambda

Step 4:  Select the source as Amazon SQS

You can choose the Batch size and window for Amazon SQS messages.

Add filtering of your choice, here, as you can see, there is an event pattern called anything but matching hi text what it does is that it tries to find the hi text from Amazon SQS. If it’s in the message, it will filter out this and skips sending it to the next step, and apart from hi, whichever messages will come in will go to the next step.

Select enrichment-sqs-msg AWS Lambda

Choose destination AWS Lambda

Now send the message except hi.

As we can see below, the event has arrived in the AWS CloudWatch logs of AWS Lambda.

Conclusion

Amazon EventBridge Pipes offer a simple, efficient, and cost-effective way to configure advanced integrations with enhanced security, reliability, scalability, and the ability to monitor their performance using CloudTrail and CloudWatch. Amazon EventBridge Pipes provides a convenient and streamlined approach for connecting AWS services. It eliminates the need for manual and time-consuming communication setups using AWS Glue.

About CloudThat