Login
June 16, 2023Voiced by Amazon Polly |
Introduction
Now easily stitch AWS services together with Amazon EventBridge Pipes. We used to use the AWS Glue to communicate between the different services, which can be a time-consuming manual exercise but an easier way to connect AWS services called Amazon EventBridge Pipes.
Amazon EventBridge Pipes is an AWS service that can receive events from various AWS services like AWS SQS, AWS Kinesis, and Amazon DynamoDB. To reduce costs, filter patterns can be configured during the filtering step to exclude unnecessary events from further processing. In the enrichment step, data from the source can be enhanced using built-in transformations or advanced transformations performed by AWS Lambda, API Gateway, or Step Functions.
Enrichments are invoked synchronously, ensuring the timely availability of the enhanced data. Pipes serve as conduits to send events or data to targets such as AWS Lambda, API Gateway, ECS Cluster, CloudWatch Log, Kinesis Stream, or API destinations. Transformers can be written in the Target step to specify how data should be sent to the targets.
AWS CloudTrail tracks Pipe invocations and their details, while Amazon CloudWatch provides metrics to monitor the health status of Pipes.
Amazon SQS
Amazon SQS is a managed message queuing service that enables asynchronous communication between cloud application components. It supports two types of message queues(Standard and FIFO) and provides reliable, scalable, and decoupled messaging infrastructure for building cloud-based applications and microservices.
Pioneers in Cloud Consulting & Migration Services
- Reduced infrastructural costs
- Accelerated application deployment
AWS Lambda
AWS Lambda is a serverless computing service provided by Amazon Web Services (AWS) that allows developers to run code without managing servers or infrastructure.
Amazon Lambda lets you execute your code in response to events such as changes in data, user actions, or file uploads. You can use Lambda to build custom APIs, process data streams, run ETL jobs, and automate tasks.
Steps on how Amazon SQS and AWS Lambda are integrated by using Amazon EventBridge Pipes
Step 1: Create source Amazon SQS
Step 2: Create AWS Lambda for Enrichment during the Amazon EventBridge Pipe
1 2 3 4 | import json def lambda_handler(event, context): #here you can modify or do the necessary operations return event |
Step 3: Create Destination AWS Lambda
1 2 3 | import json def lambda_handler(event, context): return event |
Step 4: Select the source as Amazon SQS
You can choose the Batch size and window for Amazon SQS messages.
Add filtering of your choice, here, as you can see, there is an event pattern called anything but matching hi text what it does is that it tries to find the hi text from Amazon SQS. If it’s in the message, it will filter out this and skips sending it to the next step, and apart from hi, whichever messages will come in will go to the next step.
Select enrichment-sqs-msg AWS Lambda
Choose destination AWS Lambda
Now send the message except hi.
As we can see below, the event has arrived in the AWS CloudWatch logs of AWS Lambda.
Conclusion
Amazon EventBridge Pipes offer a simple, efficient, and cost-effective way to configure advanced integrations with enhanced security, reliability, scalability, and the ability to monitor their performance using CloudTrail and CloudWatch. Amazon EventBridge Pipes provides a convenient and streamlined approach for connecting AWS services. It eliminates the need for manual and time-consuming communication setups using AWS Glue.
Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.
- Reduced infrastructure costs
- Timely data-driven decisions
About CloudThat
CloudThat is an official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner and Microsoft Gold Partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best-in-industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.
Drop a query if you have any questions regarding Amazon EventBridge, I will get back to you quickly.
To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.
FAQs
1. Is Amazon EventBridge Pipes free?
ANS: – No, based on the number of requests entering a Pipe and payload size. For more info, refer to AWS Amazon EventBridge Pipes pricing.
2. What are the Source services available in Amazon EventBridge Pipes?
ANS: – The available Sources services in Amazon EventBridge Pipes are Amazon DynamoDB stream, Amazon Kinesis stream, Amazon MQ broker, Amazon MSK stream, Self-managed Apache Kafka stream, and Amazon SQS queue.
WRITTEN BY Suresh Kumar Reddy
Yerraballi Suresh Kumar Reddy is working as a Research Associate - Data and AI/ML at CloudThat. He is a self-motivated and hard-working Cloud Data Science aspirant who is adept at using analytical tools for analyzing and extracting meaningful insights from data.
PREV
Comments