AI/ML, AWS

5 Mins Read

Step-by-Step Guide To Store AWS IoT data into S3 Bucket

1. Introduction

Nowadays, real-time IoT (Internet of Things) applications such as food industry, asset tracking, industrial device monitoring, agricultural production climate monitoring, irrigation application, vehicle tracking, home automation, CCTV Camera, and so on have increased dramatically. These industrial and commercial IoT applications generate massive amounts of data. However, keeping the data in the database for an extended period costs a lot of money and causes a severe business impact.

We can significantly reduce costs by storing these massive amounts of data in AWS S3(Simple Storage Service). AWS S3 is an object storage service that provides industrial data storage with high data availability, scalability, security, and low cost.

Here are some references on AWS Cloud, and AWS IoT Services.

Here, I will show how we can use AWS IoT rules to store IoT device data in AWS S3.

2. High-Level Architectural Diagram

AWS IoT data into S3 Bucket

  • Cloud Migration
  • Devops
  • AIML & IoT
Know More

3. Step-by-Step Guide:

1. Create S3 Bucket:

Log in to your AWS account, type S3 into the search box in the main menu, and then select S3 from the services menu.

AWS IoT data into S3 Bucket

In the left blade of the AWS S3 console, select the Buckets option and then click Create bucket option

AWS IoT data into S3 Bucket

Enter the Bucket name as s3-bucket-for-iot-data and select an AWS region as you prefeed to ensure that AWS IoT services and S3 are both in the same region. I have selected as US East (N. Virginia) us-east-1 as my region

AWS IoT data into S3 Bucket

Leave the rest of the options alone and click Create bucket at the bottom

AWS IoT data into S3 Bucket

We can see that the bucket has been successfully created with the name s3-bucket-for-iot-data

AWS IoT data into S3 Bucket

2. Create an IoT rule to send data to the S3:

In the search box, type IoT Core, and then pick IoT Core from the services selection.

AWS IoT data into S3 Bucket

In the IoT Core console, on the left blade, click Act, and then on the right blade, click rule. Finally, click Create in the Rules console to create the rule to transfer IoT data to the S3 bucket.

AWS IoT data into S3 Bucket

Enter IoT_data_rule_for_S3 as the name and define the rule in the description section.

AWS IoT data into S3 Bucket

Select 2016-03-23 as the SQL version in the Rule query statement and input the Rule Query statement as SELECT * FROM ‘iotdevice/+/datas3’, where * denotes reading the whole data sequence from the IoT topic iotdevice/+/datas3 and + denotes a wildcard character.

AWS IoT data into S3 Bucket

To add an action to a rule, select Add action

AWS IoT data into S3 Bucket

To add an action, click Store a message in an Amazon S3 bucket, select an action blade, and then click Configure action at the bottom page

AWS IoT data into S3 Bucket

In the section “Configure action” Choose the name of the S3 generated in the previous step, which is s3-bucket-for-iot-data, for S3 bucket name. In the key, type ${cast(topic(2) AS DECIMAL)}/${timestamp()} to help differentiate the data over devices id and time. Here ${cast(topic(2) AS DECIMAL)} indicates the devices id which is present in the IoT topic and ${timestamp()} is a current time of decimal value.

AWS IoT data into S3 Bucket

We need to provide role for AWS IoT rule to send data to S3 to create a new role click Create Role

In the Create a new role console, enter Iotdata_S3_role as the role name and select the Create role option

AWS IoT data into S3 Bucket

Brief period Iotdata_S3_role is created with attached policy and click Add action to add the role

AWS IoT data into S3 Bucket

Select Create rule at the bottom of the Create a rule page to create the rule. For example, we can see that the IoT_data_rule_for_S3 has been created.

AWS IoT data into S3 Bucket

3. Test IoT rule and S3:

To test an IoT rule, go to the AWS IoT Core dashboard, then to the left blade, select Test, and then MQTT test client. Click the publish to a topic option on the MQTT Test Client page and enter iotdevice/55/datas3 as the Topic name and Message payload {“temperature”: 22, “humidity”: 88, “co2”: 455}. To publish the data, click the Publish option. Now the data is published.

AWS IoT data into S3 Bucket

To verify the published data, open the S3 console and choose the bucket, creating the first step, s3-bucket-for-iot-data. Next, we can find the created folder named 55 and open that folder to see the published data.

AWS IoT data into S3 Bucket

The published data may be found in the objects console; click the time streamed data and Download to see the data.

AWS IoT data into S3 Bucket

4. Use Cases:

We require an IoT core with AWS S3 for long-term storage whenever we have many IoT devices that generate a large volume of data or videos from IoT cameras that must be taken as a backup.

5. Conclusion:

As a result, AWS IoT Core with the IoT Rule engine will assist in filtering IoT topics and the storage of data in AWS S3. AWS IoT core can receive and send millions of IoT data at a time, and the AWS IoT Rule engine can filter MQTT topics from IoT devices and send them to other AWS Services and a time stamp. For data backup and archive, AWS S3 will be used.

Get your new hires billable within 1-60 days. Experience our Capability Development Framework today.

  • Cloud Training
  • Customized Training
  • Experiential Learning
Read More

About CloudThat

CloudThat is the official AWS Advanced Consulting Partner and collaborates with top IT giants to provide cloud strategy, build solutions, and manage infrastructure. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

Feel free to drop a comment or any queries you have regarding AWS S3 bucket, IoT data storage, and any other AWS services we will get back to you quickly. To get started, go through our Expert Advisory page and Managed Services Package that is CloudThat‘s offerings.

FAQs

1. What is the application of AWS IoT core?

ANS: – We can receive and deliver millions of data to AWS Services and IoT Edge devices using AWS IoT Core. In IoT applications, the AWS IoT core is employed.

2. What is the use case of the IoT Rule?

ANS: – AWS IoT Rules allow you to filter MQTT topics from IoT devices and transfer them to other AWS services such as S3, Lambda, DynamoDB, IoT Analytics, etc.

WRITTEN BY Vasanth Kumar R

Vasanth Kumar R works as a Sr. Research Associate at CloudThat. He is highly focused and passionate about learning new cutting-edge technologies including Cloud Computing, AI/ML & IoT/IIOT. He has experience with AWS and Azure Cloud Services, Embedded Software, and IoT/IIOT Development, and also worked with various sensors and actuators as well as electrical panels for Greenhouse Automation.

Share

Comments

  1. Kush

    Aug 19, 2022

    Reply

    Thank You

  2. Nya Murray

    Aug 10, 2022

    Reply

    Well written, easy-to-use tutorial.

  3. Click to Comment

Get The Most Out Of Us

Our support doesn't end here. We have monthly newsletters, study guides, practice questions, and more to assist you in upgrading your cloud career. Subscribe to get them all!