Transferring data between Amazon S3 buckets across AWS accounts using AWS Data Sync

Introduction

Transferring data between Amazon S3 buckets across AWS accounts using DataSync offers a streamlined and efficient way to securely move large volumes of data. AWS DataSync automates the process, ensuring data is transferred quickly and reliably, with features like incremental transfers and data validation. This method is ideal for migrating, synchronizing, or backing up data between S3 buckets in different accounts, all while maintaining security and integrity through AWS IAM roles and policies.

What is Data sync? How it works?

AWS DataSync is an online data transfer and discovery service that simplifies data migration and helps you quickly, easily, and securely transfer your file or object data to, from, and between AWS storage services.

On-premises storage transfers

DataSync works with the following on-premises storage systems:

• Network File System (NFS)
• Server Message Block (SMB)
• Hadoop Distributed File Systems (HDFS)
• Object storage

AWS storage transfers

AWS DataSync works with the following AWS storage services:

• Amazon S3
• Amazon EFS
• Amazon FSx for Windows File Server
• Amazon FSx for Lustre
• Amazon FSx for OpenZFS
• Amazon FSx for NetApp ONTAP

Use cases

Here are some main use cases for AWS DataSync:

• Migrate Data: Rapidly transfer active datasets to AWS storage with automatic encryption and data validation.
• Archive Cold Data: Move infrequently accessed data to long-term storage like Amazon S3 Glacier to free up on-premises capacity.
• Replicate Data: Copy data to various Amazon S3 storage classes or Amazon EFS and FSx for different storage needs.
• In-Cloud Processing: Transfer data to or from AWS for faster processing in industries such as machine learning, media, finance, and oil and gas.

Benefits

Using DataSync offers the following benefits:

• Simplify Migration Planning: Automated data collection and recommendations with AWS DataSync Discovery reduce time, effort, and costs, aiding budget planning and validating assumptions as you approach your migration.
• Automate Data Movement: AWS DataSync streamlines data transfers between storage systems and services, automating data-transfer processes and required infrastructure for high-performance and secure transfers.
• Transfer Data Securely: Provides end-to-end security, including encryption and integrity validation, while accessing AWS storage via AWS IAM roles and Amazon VPC endpoints to enhance data security.
• Move Data Faster: Accelerates transfers using a purpose-built protocol and multi-threaded architecture, speeding up migrations, analytics workflows, and data protection.
• Reduce Operational Costs: Cost-effective data movement with flat per-gigabyte pricing, eliminating the need for custom scripts or expensive transfer tools.

Demo

Steps to migrate Amazon S3 bucket data across AWS accounts:

Pre-requisites

1. For your source AWS account, there are two sets of permissions to consider with this kind of cross-account transfer:

• User permissions that allow a user to create AWS DataSync locations and tasks.
• AWS DataSync service permissions allow AWS DataSync to transfer data to the destination account bucket.

2. Disable the destination bucket’s access control lists

User permissions:

Step 1: In your source account, create an AWS DataSync AWS IAM role for destination bucket access

Step 2: In your destination account, update your Amazon S3 bucket policy

In your destination account, modify the destination Amazon S3 bucket policy to include the “DataSync IAM role” that you created in your source account.

Step 3: Create your AWS DataSync locations:

Create the AWS DataSync locations for your source and destination S3 buckets.

• Create your AWS DataSync Source location

• Create your AWS DataSync destination location

While still in your source account, launch CloudShell from the console:

Run the following command

• Replace destination-bucket with the name of the Amazon S3 bucket in your destination account.
• If your destination bucket is in a different Region than your source bucket, replace destination-bucket-region with the Region where the destination bucket resides
• Replace source-account-id with the source AWS account ID.
• Replace source-datasync-role with the AWS DataSync IAM role you created in your source account.

If the command returns a DataSync location ARN similar to this, you successfully created the location:

• Create Task

• Run the Task:

• Task History

Conclusion

In this blog article, we explored how to set up an AWS DataSync task that transfers objects between Amazon S3 buckets step-by-step without requiring the installation of an agent on Amazon EC2. Guidance on configuring tasks for cross-region and cross-account use cases was given in additional stages.

Drop a query if you have any questions regarding AWS DataSync and we will get back to you quickly.

About CloudThat