A Guide to Upload Data to Amazon S3 using Node.js and AWS Lambda with Amazon API Gateway Fetch

Overview

In the rapidly changing world of software development, integrating various services and functions is crucial. One of the main integrations involves using Node.js and an AWS Lambda function (performed via API request) to send data to an Amazon S3 bucket. This blog will walk you through each step of setting up this functionality, allowing you to use the power of serverless computing to store and manage data in Amazon S3 buckets efficiently. Whether you’re building user-generated applications or need to use data storage, this integration will be a significant addition to your tool.

In this blog, we will look at how to load data into an Amazon S3 bucket using Node.js and AWS Lambda (performed via API requests).

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Introduction

Amazon Web Services (AWS) provides a powerful ecosystem of services that allows developers to build capacity and reliability.

Amazon S3 (Simple Storage Service) is one of AWS’s most popular services for data storage and storage. On the other hand, AWS Lambda is a serverless service that allows you to run code without maintaining or managing servers.

Pre-requisites

Before we begin, make sure you have the following in place:

AWS Account: You should have an AWS account with the necessary permissions to create AWS Lambda functions, Amazon API Gateway, and access Amazon S3 buckets.
js: Make sure you have Node.js installed on your development machine.
Serverless Framework (optional): You can use the Serverless Framework to simplify the deployment process, but it’s not mandatory.

Step-by-Step Guide

Step 1: Setting Up an Amazon S3 Bucket

Log in to your AWS Management Console and navigate the Amazon S3 service.
Create a new Amazon S3 bucket to store your uploaded data. Note down the bucket name, and you’ll need it later.

Step 2: Write the AWS Lambda Function

Create a new Node.js project or use an existing project.
Install the AWS SDK by running the npm install aws-sdk command.
Write a .js script for the AWS Lambda function that Node uses to upload the data to the Amazon S3 bucket. A simple example using aws-sdk:

const AWS = require('aws-sdk'); 

const s3 = new AWS.S3(); 
  

exports.handler = async (event) => { 

  try { 

    const { data, fileName } = JSON.parse(event.body); 

  

    const params = { 

      Bucket: 'your-s3-bucket-name', 

      Key: fileName, 

      Body: data, 

    }; 

  

    await s3.putObject(params).promise(); 

  

    return { 

      statusCode: 200, 

      body: JSON.stringify({ message: 'File uploaded successfully' }), 

    }; 

  } catch (error) { 

    return { 

      statusCode: 500, 

      body: JSON.stringify({ message: 'Error uploading file' }), 

    }; 

  } 

};

const AWS = require('aws-sdk');

const s3 = new AWS.S3();

exports.handler = async (event) => {

try {

const { data, fileName } = JSON.parse(event.body);

const params = {

Bucket: 'your-s3-bucket-name',

Key: fileName,

Body: data,

};

await s3.putObject(params).promise();

return {

statusCode: 200,

body: JSON.stringify({ message: 'File uploaded successfully' }),

};

} catch (error) {

return {

statusCode: 500,

body: JSON.stringify({ message: 'Error uploading file' }),

};

}

};

Replace ‘your-s3-bucket-name’, ‘your-file-key’, and ‘your-file-content’ with your S3 bucket file and the file you want to upload.

Step 3: Create Amazon API Gateway

In the AWS Management Console, navigate to the Amazon API Gateway service.
Create a new API or use an existing API.
Create new resources and methods (for example, POST ) to be used with your API. Set the integration type to Lambda Function and select the AWS Lambda function created in step 2.
Deploy the API to a stage (e.g. “Production”).

Step 4: Make an API Fetch Request

Now, you can request the API end to trigger the AWS Lambda function and send the data to your Amazon S3 bucket. Here’s a JavaScript example:

const fetch = require('node-fetch'); 

  

const apiUrl = 'YOUR_API_GATEWAY_URL'; 

const data = 'Your data to upload'; 

const fileName = 'example.txt'; 

  

const requestBody = { 

  data, 

  fileName, 

}; 

  

fetch(apiUrl, { 

  method: 'POST', 

  body: JSON.stringify(requestBody), 

  headers: { 

    'Content-Type': 'application/json', 

  }, 

}) 

  .then((response) => response.json()) 

  .then((data) => console.log(data)) 

  .catch((error) => console.error(error));

const fetch = require('node-fetch');

const apiUrl = 'YOUR_API_GATEWAY_URL';

const data = 'Your data to upload';

const fileName = 'example.txt';

const requestBody = {

data,

fileName,

};

fetch(apiUrl, {

method: 'POST',

body: JSON.stringify(requestBody),

headers: {

'Content-Type': 'application/json',

})

.then((response) => response.json())

.then((data) => console.log(data))

.catch((error) => console.error(error));

Replace “https://api-endpoint-url” with the URL of your Amazon API Gateway endpoint.

Step 5: Test and Deploy

Test your AWS Lambda function and API integration to ensure everything works as expected.
If you are using a serverless framework, use serverless deployment to deploy your services. Otherwise, use the AWS console to configure the AWS Lambda function and Amazon API Gateway.

Conclusion

In this blog post, we walked you through uploading data to an Amazon S3 bucket using Node.js. AWS Lambda and Node.js are triggered by API requests. This functionality can be used for many applications, such as collecting user-generated content and processing and storing data. With the power of AWS services, you can build capable, reliable applications that connect to the jobs you need.

Drop a query if you have any questions regarding uploading Data to Amazon S3 and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is an award-winning company and the first in India to offer cloud training and consulting services worldwide. As a Microsoft Solutions Partner, AWS Advanced Tier Training Partner, and Google Cloud Platform Partner, CloudThat has empowered over 850,000 professionals through 600+ cloud certifications winning global recognition for its training excellence including 20 MCT Trainers in Microsoft’s Global Top 100 and an impressive 12 awards in the last 8 years. CloudThat specializes in Cloud Migration, Data Platforms, DevOps, IoT, and cutting-edge technologies like Gen AI & AI/ML. It has delivered over 500 consulting projects for 250+ organizations in 30+ countries as it continues to empower professionals and enterprises to thrive in the digital-first world.

FAQs

1. How can Amazon API Gateway and AWS Lambda functions be secured to protect sensitive data?

ANS: – You can take many measures to increase security, such as using the Amazon API Gateway authentication and authorization mechanism. You can set API keys for AWS Lambda, AWS IAM (Identity and Access Management) roles, and granular permissions to control who can call your API and access your AWS Lambda roles. Also, consider using AWS Key Management Service (KMS) to encrypt sensitive data before uploading it to Amazon S3 for an extra layer of protection.

2. Can I use this method for large files or high traffic?

ANS: – Yes, you can adjust this model for larger files or high traffic situations. For large files, consider multiple uploads to Amazon S3 for efficiency. AWS Lambda and Amazon API Gateway can scale to handle more traffic, but you must fine-tune and monitor usage for efficiency and cost control. AWS also offers services like Amazon CloudFront and Amazon S3 Transfer Acceleration for optimized content delivery in high-traffic situations.