Migration of Data from Amazon RDS to Amazon S3 using AWS Lambda

Overview

In the ever-evolving world of information management, businesses always seek solutions to achieve value and efficiency. Amazon Web Services (AWS) is a powerful service for this purpose, and its most common use cases include using AWS Lambda to transfer data from Amazon RDS (Relational Database Service) to Amazon S3 (Simple Storage Service). This blog post aims to overview the data migration process, highlighting the benefits and steps of using AWS Lambda to complete data migration.

Need for Migration

As the organization grows, so does data collection. Amazon RDS provides a reliable solution, but as datasets grow, businesses often see the cost of moving historical data or less data to Amazon S3. Amazon S3 is a cost-effective data archiving solution designed for scalable and stable storage, providing high durability and easy and seamless integration with other AWS services.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Steps for migration of data using Node.js

Setting Up an AWS Lambda Function:

Start this process by creating a new Lambda function in the AWS Management Console.
Choose a runtime that is compatible with Node.js.
Define the permissions to ensure the AWS Lambda function has read access to the Amazon RDS store and write access to the target Amazon S3 bucket.
Below is the AWS Lambda code for data migration from Amazon RDS to Amazon S3.

import AWS from "aws-sdk";(create a lambda layer for importing Aws-sdk to integrate with aws services)
import mysql from "mysql2/promise";
const s3 = new AWS.S3();
const host = "sv2-xxxx-xxxx-xxxx-eu-west-2a.xxxxxxxxx.eu-west-2.rds.amazonaws.com";(provide host name of the rds here)
const user = "xxxxxx";(username)
const PORT = 3306;
const password = "xxxxxxx";(password of the database)
const database = "xxxxxxxxx";(name of the database)
const dbConfig = {
  host,
  user,
  password,
  database,
  port: PORT
};
#this below code gets the data from rds
async function rdsToS3() {
  try {
    const connection = await mysql.createConnection(dbConfig);
    const [tables, meta] = await connection.execute("SELECT * FROM table LIMIT 2;");
    console.log(tables, meta, "tables")

    let data = jsonToCsv(tables)
    console.log(data)
    uploadToS3("ct-datalake-bucket", "test.csv", data)
    // Close the connection
    connection.end();
    return {
      statusCode: 200,
      body: tables,
    };
  }
  catch (err) {
    console.error("Error executing query:", err);
    return {
      statusCode: 500,
      body: { error: "Internal Server Error", err },
    };
  }
}
# The below code will upload the data that is taken from Amazon RDS to Amazon S3
export async function uploadToS3(bucketName, fileName, content) {
  const params = {
    Bucket: bucketName,
    Key: fileName,
    Body: content,
    ContentType: "text/csv",
  };
  return await s3.upload(params).promise();
}
# the below code will export the data to s3 by converting into csv
export function jsonToCsv(jsonData) {
  console.log(jsonData, "jsonData");
  const header = Object.keys(jsonData[0]);
  let data = [header];
  jsonData.map((obj) => {
    let row = Object.values(obj);
    row = row.map((r) => {
      if (typeof r === "string") return r.replace(/[\n\r,]/g, "");
      else return String(r);
    });
    data.push(row);
  });
  data = data.map((d) => d.join(",")).join("\n");
  return data;
}
export async function handler(event) {
  let res = await rdsToS3();
  return {
    status: 200,
    res
  }
}

import AWS from "aws-sdk";(create a lambda layer for importing Aws-sdk to integrate with aws services)

import mysql from "mysql2/promise";

const s3 = new AWS.S3();

const host = "sv2-xxxx-xxxx-xxxx-eu-west-2a.xxxxxxxxx.eu-west-2.rds.amazonaws.com";(provide host name of the rds here)

const user = "xxxxxx";(username)

const PORT = 3306;

const password = "xxxxxxx";(password of the database)

const database = "xxxxxxxxx";(name of the database)

const dbConfig = {

host,

user,

password,

database,

port: PORT

};

#this below code gets the data from rds

async function rdsToS3() {

try {

const connection = await mysql.createConnection(dbConfig);

const [tables, meta] = await connection.execute("SELECT * FROM table LIMIT 2;");

console.log(tables, meta, "tables")

let data = jsonToCsv(tables)

console.log(data)

uploadToS3("ct-datalake-bucket", "test.csv", data)

// Close the connection

connection.end();

return {

statusCode: 200,

body: tables,

};

}

catch (err) {

console.error("Error executing query:", err);

return {

statusCode: 500,

body: { error: "Internal Server Error", err },

};

}

# The below code will upload the data that is taken from Amazon RDS to Amazon S3

export async function uploadToS3(bucketName, fileName, content) {

const params = {

Bucket: bucketName,

Key: fileName,

Body: content,

ContentType: "text/csv",

};

return await s3.upload(params).promise();

}

# the below code will export the data to s3 by converting into csv

export function jsonToCsv(jsonData) {

console.log(jsonData, "jsonData");

const header = Object.keys(jsonData[0]);

let data = [header];

jsonData.map((obj) => {

let row = Object.values(obj);

row = row.map((r) => {

if (typeof r === "string") return r.replace(/[\n\r,]/g, "");

else return String(r);

});

data.push(row);

});

data = data.map((d) => d.join(",")).join("\n");

return data;

}

export async function handler(event) {

let res = await rdsToS3();

return {

status: 200,

res

}

Advantages of using AWS Lambda for Migration

Serverless Architecture

AWS Lambda is a serverless computing service that relieves you of the burden of server configuration and management. This serverless architecture and auto-scaling features make Lambda ideal for automating data migration, allowing you to focus on your logic rather than worrying about infrastructure.

Cost-effective

Lambda adopts the pay-as-you-go model; You only pay for the computation time during code execution. This model has proven more efficient than managing dedicated servers or instances for regular data transfer operations.

Event-Driven Approach

AWS services, including Amazon RDS and Amazon S3, can trigger AWS Lambda functions. With an event-driven approach, you can configure Lambda to migrate data in response to specific events, such as changes to the RDS store or scheduled permissions.

Scalability and parallel processing

AWS Lambda power can easily be scaled horizontally to enable parallel processing. This is especially effective when working with large files and allows for a faster and more efficient migration process.

Set Amazon RDS and Amazon S3 triggers

Create a configuration for your AWS Lambda function. For Amazon RDS, leverage Amazon CloudWatch Events to trigger Lambda functions at scheduled times or in response to specific events. For Amazon S3, consider triggering a Lambda function or event notifications when new products are released.

Conclusion

Moving data from Amazon RDS to Amazon S3 using AWS Lambda provides a scalable, cost-effective, serverless solution for organizations looking to streamline their data management processes. By leveraging the power of AWS Lambda’s event-driven architecture, you can automate data migration to ensure your data remains accessible, secure, and integrated in the AWS cloud.

When you begin your data migration, carefully consider your business’s specific needs and customize the solution to meet your specific needs. AWS Lambda’s performance and seamless integration with other AWS services make it a valuable tool for optimizing data processing in the cloud.

Drop a query if you have any questions regarding Data Migration and we will get back to you quickly.

Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.

Reduced infrastructure costs
Timely data-driven decisions

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.

FAQs

1. How is AWS Lambda priced?

ANS: – AWS Lambda pricing is based on the number of requests and the time your code executes in 100ms increments. Users pay for the actual compute time consumed, and there is no charge when the code is not running.

2. Is it possible to run long-duration tasks with AWS Lambda?

ANS: – While AWS Lambda is designed for short-duration tasks, it supports functions with a maximum execution time of 15 minutes.

3. Can AWS Lambda function access other AWS services?

ANS: – Yes, Lambda functions can integrate with various AWS services, such as Amazon S3, Amazon DynamoDB, Amazon API Gateway, and more.