{"id":12598,"date":"2022-06-06T06:10:34","date_gmt":"2022-06-06T06:10:34","guid":{"rendered":"https:\/\/blog.cloudthat.com\/?p=12598"},"modified":"2024-06-25T10:59:17","modified_gmt":"2024-06-25T10:59:17","slug":"simplify-cloud-data-migration-aws-datasync-amazons-online-data-transfer-service","status":"publish","type":"blog","link":"https:\/\/www.cloudthat.com\/resources\/blog\/simplify-cloud-data-migration-with-aws-datasync-amazons-online-data-transfer-service","title":{"rendered":"Simplify Cloud Data Migration with AWS DataSync \u2013 Amazon\u2019s Online Data Transfer Service"},"content":{"rendered":"\n\n\n\n\n\n\n\n\n\n\n\n
\n

TABLE OF CONTENT<\/strong><\/span><\/h2>\n<\/td>\n<\/tr>\n

1. Introduction to Amazon DataSync<\/a><\/td>\n<\/tr>\n
2. Step to Covert your Linux machine into a Samba Server<\/a><\/td>\n<\/tr>\n
3. Create a DataSync agent in EC2 in a VPC<\/a><\/td>\n<\/tr>\n
4. Create the agent in the DataSync in the AWS Portal
\n<\/a><\/td>\n<\/tr>\n
5. Create a Task<\/a><\/td>\n<\/tr>\n
6. Start the Transfer<\/a><\/td>\n<\/tr>\n
7. Summary<\/a><\/td>\n<\/tr>\n
8. About CloudThat <\/a><\/td>\n<\/tr>\n
9. FAQs<\/a><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Introduction to AWS DataSync<\/h2>\n

There has been increased demand to back up our files to a cloud environment for long-term storage to cover any disaster-related incidents. Customers want to securely migrate their data with the help of reliable utility tools which help them to do so. Moreover, they want to automate these tasks through a reliable mechanism to transfer their data to the cloud-like AWS in S3, EFS, Windows file server, etc.<\/p>\n

AWS DataSync helps connect your on-prem storage to S3 and much more with a reliable automation architecture. File share systems like NFS (network file system) and SMB (Server Message Block) can now be integrated into AWS DataSync to transfer your required files.<\/p>\n

DataSync allows you to transfer all the files or only the changed data the next time you start transferring; it does so by using metadata related to previous data captured, which helps decrease the transfer size and the related time to transfer. AWS DataSync uses a DataSync agent, either installed on on-premises hardware such as VMware, Hyper-V, or an EC2 machine using AWS-provided AMI. This server helps in connecting the source endpoint of the local server to the target endpoint on S3 EFS, etc.<\/p>\n

I will be showing a small but powerful setup where you can transfer the files into the S3 from SMB (Samba) server using AWS data DataSync.<\/p>\n

Step to Covert your Linux machine into a Samba Server<\/h2>\n

Here I am using an Amazon Linux AMI in another VPC to act as a remote Samba server<\/p>\n

    \n
  1. Launch Linux EC2 of your choice here; I am using Amazon Linux to install the SMB server.<\/li>\n
  2. In the security group, open port 22and port 445 to anywhere, i.e., 0.0.0.0\/0 range (later, we can change this to a dedicated IP)<\/li>\n
  3. It is an Amazon Linux 2 AMI; we need to follow the below steps to change the hostname to a user-friendly name\n
    sudo vi \/etc\/cloud\/cloud.cfg<\/pre>\n<\/li>\n
  4. At the end of the file, add the below line and save\n
    preserve_hostname: true<\/pre>\n<\/li>\n
  5. Change Hostname\n
    $ sudo hostnamectl set-hostname samba-server\r\n$ sudo yum update -y\r\n$ sudo reboot<\/pre>\n<\/li>\n
  6. Install Samba, Samba-client, and cifs-utils\n
    $ sudo su -\r\n# yum install -y samba samba-client cifs-utils<\/pre>\n<\/li>\n
  7. Configuration file changes\n
    # vim \/etc\/samba\/smb.conf<\/pre>\n
    security = user\r\n        hosts allow = ip address of Data DataSync agent VM  \r\n*** Add the loopback IP and the VPC starting ip as shown above\r\n        interfaces = lo eth0        passdb backend = smbpasswd:\/etc\/samba\/sambapasswd.txt        printing = cups\r\n        printcap name = cups\r\n        load printers = yes\r\n        cups options = raw\r\n[homes]\r\n        comment = Home Directories\r\n        valid users = %S, %D%w%S\r\n        browseable = No\r\n        read only = No\r\n        inherit acls = Yes\r\nprinters]\r\n        comment = All Printers\r\n        path = \/var\/tmp\r\n        printable = Yes\r\n        create mask = 0600\r\n        browseable = No\r\n[print$]\r\n        comment = Printer Drivers\r\n        path = \/var\/lib\/samba\/drivers\r\n        write list = @printadmin root\r\n        force group = @printadmin\r\n        create mask = 0664\r\n        directory mask = 0775\r\n[samba]\r\n        comment = Development documentation\r\n        read only = no\r\n        available = yes\r\n        path = \/smbfolder\r\n        public = yes\r\n        valid users = sambauser\r\n        write list = sambauser\r\n        writable = yes\r\n        browseable = yes<\/pre>\n
    Make sure to edit the config as above and change the host allow to your data DataSync agent to connect<\/li>\n
  8. We will now create a samba user to access the samba folder directory\n
    # useradd sambauser\r\n# passwd sambauser\r\n# smbpasswd -a sambauser\r\n# service smb restart<\/pre>\n<\/li>\n
  9. Create a directory for file share and give permissions\n
    # mkdir \/smbfolder\r\n# chmod 777 \/smbfolder<\/pre>\n<\/li>\n
  10. Restart and test with the data DataSync agent\n
    # service smb restart\r\n# testparm<\/pre>\n<\/li>\n<\/ol>\n
    Create a DataSync agent in EC2 in a VPC<\/h2>\n
    You can choose your DataSync AMI using the below command
    \n\"AWS<\/a><\/p>\n
    \"AWS<\/a><\/p>\n
    Create the agent in the DataSync in the AWS Portal by navigating to DataSync in the AWS Console<\/h2>\n
    Place the Data DataSync agent ID in the agent address<\/p>\n
    \"AWS<\/a><\/p>\n
    \"AWS<\/a><\/p>\n
    Create a Task<\/h2>\n