AWS, Cloud Computing

5 Mins Read

A Guide to Ingest IoT Data to Kinesis Data Firehose and Store in S3

Voiced by Amazon Polly

Overview

As industrial automation is advancing at a rapid pace, IoT (Internet of Things) technology is proving to be quite useful in monitoring machines or devices, keeping track of them, and taking actions based on particular situations, which will help save time and accelerate corporate growth. These IoT devices generate a large amount of unstructured data, which must be kept in a specific location for real-time monitoring or analytics in the form of a file in a certain format, such as CSV, Apache Parquet, or Apache ORC, which is typically more efficient to query.

This blog will look at how to use the kinesis data firehose to store different IoT data as a file format in S3.

intro

Customized Cloud Solutions to Drive your Business Success

  • Cloud Migration
  • Devops
  • AIML & IoT
Know More

Steps to Create S3 Bucket

Step 1: To create an S3 bucket, log into your AWS account and type S3 into the console’s search bar. The S3 service will appear below; click on it.

Step1buc

Step 2: Go to the left blade and Select the Bucket option, then click the Create bucket option. In the bucket, name option put s3-bucket-for-iot-data and then Select AWS Region.

Step2buc

Step 3: Choose to create bucket option to create a bucket. We can see that the bucket with the name s3-bucket-for-iot-data has been successfully created.

Step3buc

Steps to Create IoT data delivery stream in Kinesis Data Firehose

Step 1: To create a Kinesis Data Firehose delivery stream, in the AWS console search bar enter Kinesis and then Select Kinesis service.

Step1kin

Step 2: At the left blade of Amazon Kinesis, select Delivery streams and then Create Delivery Stream by clicking the option which is created delivery stream. On the option Choose the source and destination select the Direct PUT option in the Source and Select Amazon S3 in the Destination part finally enter the Delivery stream name as IoT_data_delivery_stream.

Step2kin

In the Destination settings choose the Browse option which is below the S3 bucket.

Step2kinb

Step 3: Choose the previously created bucket which is s3-bucket-for-iot-data and then click choose an option. Enter the IoT-data at S3 bucket prefix – optional – to indicate that the kinesis data firehose will create a folder in S3 with the “YYYY/MM/dd/HH” format.

Step3kin

Step 4: Choose Buffer hints, compression, and encryption option and enter 1 MiB at the place of Buffer size and enter 60 seconds as Buffer interval.

Step4kin

Step 5: Finally, choose the Enable option in the Amazon CloudWatch error logging, and at the permissions choose the create or update IAM role option and down.

Step5kin

Click on Create deliver stream option you can find IoT_data_delivery_stream will be created in a matter of time.

Steps to Create IoT Rule to Send Data to the Kinesis Data Firehose

Step 1: Type IoT Core into the search box, then select IoT Core from the services menu.

Step1fire

Step 2: Click Act on the left blade of the IoT Core console, then Rule on the right blade. To create the Rule that will transfer IoT data to an S3 bucket, go to the Rules console and click Create.

Enter IoT_data_rule as the name and define the rule in the description section.

Step2fire

Step 3: In the Rule query statement, select 2016-03-23 as the SQL version, and type SELECT * FROM ‘iotdevice/+/data’, where * denotes reading the entire data sequence from the IoT topic iotdevice/+/data, and + denotes a wildcard character.

Step3fire

To add an action to a rule, select Add action.

Step3fireb

Step 4: To add an action, click Send a message to an Amazon Kinesis Firehose stream in the select an action blade and then click Configure action at the bottom page.

Step4fire

Then select the IoT_data_delivery_stream from the stream name

Step4fireb

Step 5: To create a role for the IoT core to send data to the kinesis data firehose click on create role option

Step5fire

Enter the role name as iot-kinesis-delivery-role and click Create Role.

Step5fireb

At the final to create a rule click create rule option.

Step5firec

Just a matter of time IoT rule will be created for sending data to the kinesis firehose.

Step5fired

Testing

  • Go to the AWS IoT Core dashboard, then to the left blade, select Test, and then MQTT test client to test an IoT rule.
  • On the MQTT Test Client page, select the publish to a topic option and enter “iotdevice/22/data” as the Topic name and the Message payload {“Temperature”: 20.5,”Humidity”: 50,”Co2″: 450} as the Message payload.
  • Click the Publish option to make the data publish again with the different data {“Temperature”: 21.5,”Humidity”: 60,”Co2″: 350}. The information has now been made published.

test1

  • To verify the published data, wait 60 seconds then open the S3 console, and choose the bucket which is created in the first step which is s3-bucket-for-iot-data.
  • We can be able to find the created folder named IoT-data2022 and open that folder to see the published data.

test2

  • You can find the file name IoT_data_delivery_stream with a date.

test3

  • If you download and open the file, we can find all ingested data within 60 seconds.

test4

Conclusion

Thus, the above integration flow shows how to inject the IoT data into the S3 with help of Amazon Kinesis data firehose. The advantage of using kinesis data firehoses is that we can create files with lots of IoT data and do transformations on the fly if necessary. This approach helps to collect data from a vast number of IoT devices and store it in the S3.

Get your new hires billable within 1-60 days. Experience our Capability Development Framework today.

  • Cloud Training
  • Customized Training
  • Experiential Learning
Read More

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is the first Indian Company to win the prestigious Microsoft Partner 2024 Award and is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 850k+ professionals in 600+ cloud certifications and completed 500+ consulting projects globally, CloudThat is an official AWS Premier Consulting Partner, Microsoft Gold Partner, AWS Training PartnerAWS Migration PartnerAWS Data and Analytics PartnerAWS DevOps Competency PartnerAWS GenAI Competency PartnerAmazon QuickSight Service Delivery PartnerAmazon EKS Service Delivery Partner AWS Microsoft Workload PartnersAmazon EC2 Service Delivery PartnerAmazon ECS Service Delivery PartnerAWS Glue Service Delivery PartnerAmazon Redshift Service Delivery PartnerAWS Control Tower Service Delivery PartnerAWS WAF Service Delivery PartnerAmazon CloudFront Service Delivery PartnerAmazon OpenSearch Service Delivery PartnerAWS DMS Service Delivery PartnerAWS Systems Manager Service Delivery PartnerAmazon RDS Service Delivery PartnerAWS CloudFormation Service Delivery PartnerAWS ConfigAmazon EMR and many more.

FAQs

1. What is use of Kinesis Data Firehose?

ANS: – Amazon Kinesis Data Firehose is an extract, transform, and load (ETL) service that collects, processes, and distributes streaming data to data lakes, data storage, and analytics services with high reliability. 

2. What type of data compression format is supported by AWS Kinesis Data Firehose?

ANS: – GZIP, ZIP, and SNAPPY compression formats. 

WRITTEN BY Vasanth Kumar R

Vasanth Kumar R works as a Sr. Research Associate at CloudThat. He is highly focused and passionate about learning new cutting-edge technologies including Cloud Computing, AI/ML & IoT/IIOT. He has experience with AWS and Azure Cloud Services, Embedded Software, and IoT/IIOT Development, and also worked with various sensors and actuators as well as electrical panels for Greenhouse Automation.

Share

Comments

    Click to Comment

Get The Most Out Of Us

Our support doesn't end here. We have monthly newsletters, study guides, practice questions, and more to assist you in upgrading your cloud career. Subscribe to get them all!