Serverless Migration of Data Using DynamoDB Export to S3

Overview

Amazon DynamoDB is a serverless NoSQL database that gives single-digit millisecond performance at any scale. It provides two integrated backup techniques i.e., on-demand and point-in-time recovery that are appropriate for backing up your tables for disaster recovery. But you cannot use this data for data analysis and ETL processes.

For this purpose, customers had to rely on Amazon EMR, AWS Data Pipeline, or AWS Glue services which require high read throughput, resulting in high operating and maintenance costs.

Introducing DynamoDB Export to S3 feature

Using this feature, you can export table data to the Amazon S3 bucket anytime within the point-in-time recovery window, up to 35 days. It does not consume any read capacity on the table while exporting data and the performance and availability of the table are unaffected. We can use AWS Management Console, AWS CLI, or the DynamoDB API to export DynamoDB table data to S3.

We can export data to another AWS account S3 bucket if you have the correct IAM permissions to write into that bucket and across the different regions. It supports two formats i.e., DynamoDB JSON and Amazon ION text format to export data to S3.  You can concurrently export up to 300 export tasks or up to 100 TB of table size. DynamoDB table export is intended to be faster than table scan export. The actual time it takes for the export to complete, however, is determined by the size of the table and how evenly the table data is dispersed.

Step-by-Step Guide to Export Amazon DynamoDB Table Data to S3

Step 1: Sign into the Amazon management console. Create an Amazon S3 bucket to store the Amazon DynamoDB tables data.

Step 2: Exporting data to Amazon S3

1. Enter DynamoDB in the amazon search bar and select Amazon DynamoDB
2. In the navigation pane, go to Exports to S3
3. Click on Export to S3. To send Amazon DynamoDB tables data to S3.

4. Select the Amazon DynamoDB table name that you want to export as the Source table.

5. Enable PITR, if you have not already enabled continuous backups

6. Under Destination S3 bucket, browse the bucket name where you want to export data.

7. Select the S3 bucket owner as an AWS account, where your S3 bucket is present.

8. Check on Additional settings for more options such as selecting specific point-in-time, file formats, and encryption keys. Here I am using the default

9. Click on Export

Note:  If you choose to encrypt your export using a key protected by AWS Key Management Service (AWS KMS), the key must be in the same Region as the destination S3 bucket.

10. Verify the status of the export, in DynamoDB export to the S3 console.

Step 3: Verify the exported table data in the destination Amazon S3 bucket.

1. Open the Amazon S3 console. Select the destination Bucket to see whether data is exported successfully or not.

2. Here you can see the folders created by Amazon DynamoDB export. It includes two manifest files in addition to the files containing your table data.

Conclusion

It is the easiest way to export data to S3 without building custom applications or writing code. You can perform ETL against data exported on S3 and you can import it back to DynamoDB anytime. By storing data in S3 other AWS services i.e., Amazon Athena or Amazon Redshift can easily access and perform analytics on the data.

About CloudThat