13 Easy Steps for Syncing Data from On-Premises To AWS S3 Using DataSync

Introduction

Data synchronization is syncing data across two or more devices and automatically updating changes between them to ensure system consistency.

While the massive amount of data stored in the cloud poses issues, it also makes it the ideal option for big data. Today’s data solutions provide quick and straightforward tools for avoiding repetitive activities, resulting in data in sync throughout the system.

Agent creation

Use the CLI command below to acquire the most recent DataSync Amazon Machine Image (AMI) ID for the selected AWS Region.

1. aws ssm get-parameter –name /aws/service/datasync/ami –region $region

Launch the agent using your AMI from the Amazon EC2 launch wizard from the AWS account where the source file system is stored. To start the AMI, go to the following URL: https://console.aws.amazon.com/ec2/v2/home?region=source-file-system-region#LaunchInstanceWizard:ami=ami-id

Note:- Please make sure that you are launching a machine having RAM of 16 GiB or more

Here I am launching t2.xlarge for my agent with public IP and opening an HTTP port.

Once your agent creating is done now, let us go to DataSync.

Step by Step Guide for Configuring DataSync

1. Open the DataSync service from the AWS Console and select the below option for the data transfer option. Click on Get Started.
   
2. Select the Hypervisor as AWS EC2, select the Endpoint type as Public service Endpoints, and provide the public IP of the agent in the agent address as below
   
3. Once the activation key retrieval is successful you can provide the agent with a name and click on create button then your agent will be created and shows Agent status as Online.
4. Before creating a task for the DataSync, create an EC2 (t2.micro) which will act as an On-premise system.
5. Please open the RDP and NFS ports in the On-premise security group
6. Now install the NFS server On-Premise such that it will sync the local files to the cloud through NFS.
7. Create a folder and text file inside it. In my case, I have created a folder with the name test and a text file with the name sampletext.txt, which we want to sync. Go to the properties of the folder. Select the NFS sharing option and click on Manage NFS sharing and check the share this folder as shown below.
8. Click Apply and then OK to save the configuration over the folder
   
9. Now go to the DataSync console and create a task and choose the configuration details as below:
   a. Location as the new location
   b. The location type should be Network File System (NFS)
   c. Select the agent which we created in the steps
   d. Provide the Ip address in the NFS server domain
   e. Provide the mount path in my case as /test
   f. The destination location is S3, and select the S3 bucket to where these files need to be sync
10. Leave the remaining fields as default and click on create
11. Select the autogenerate for creating the IAM in the provided fields
12. Once the previous step is completed, wait until your task’s status becomes available.
13. Now start the task and wait for a while; this starts syncing your on-premise data to AWS S3

Conclusion

We investigated and highlighted the relevance of data synchronization in this blog. When just the changed data is transferred, data synchronization will work smoothly. As a result, each synchronization procedure uses a marker to determine the most up-to-date information.

About CloudThat