A Guide to Access Amazon S3 buckets and Objects from On-Premises over VPN

Introduction

AWS S3 (Simple Storage Service) is a popular cloud storage solution that offers scalable and durable storage for various types of data, including videos, images, documents, and backups. Amazon S3 is renowned for its high availability and low-cost qualities and for integrating well with other AWS services.

One option is Site-to-Site VPN, which creates a secure virtual tunnel between the client’s on-premises network and the AWS cloud. This enables clients to access Amazon S3 from their on-premises resources without the need to traverse the public internet, thereby reducing the risk of data interception or eavesdropping.

Another option is AWS Direct Connect, a dedicated network connection between the client’s data center and AWS. This provides a private, high-bandwidth connection for transmitting data between on-premises resources and Amazon S3, with lower latency and higher reliability than a VPN.

Today we will make a VPN using AWS and Azure and fetch Amazon S3 object over VPN privately. In this blog, I’m assuming the Azure VM is my on-premises server.

Creating Azure VPN resources

1. Create a Vnet in Azure CIDR 10.0.0.0/16
2. Create Subnets CIDR 10.0.1.0/24
3. Create a (VPG) Virtual Network Gateway. That will be used to create a Site-to-Site VPN connection. Hence, allow you to securely access resources in your virtual network from your on-premises network.

4. Create a virtual machine in the above subnet with a public IP address. This will help in accessing the machine from your system. Ensure that the SSH and ICMP ports are open for the required connectivity.

Creating AWS VPN resources

1. Create an AWS VPC with CIDR 20.0.0.0/16
2. Create a subnet with CIDR 20.0.4.0/16 in the same VPC
3. Create a customer gateway using the public IP of VNG created in Azure before
4. Create a virtual private gateway and attach it to your AWS VPC

5. Create a site-to-site VPN connection using the below details

• Enter the name of the connection
  Choose the virtual private gateway we created earlier.
• Choose the virtual private gateway we created earlier

• Choose the customer gateway we created earlier
• Choose Routing as Static
• Define static IP as of on-premises subnet where your resources are
• Tunnel inside IP version: IPv4
• Local IPv4: Same as your on-premises subnet
• Remote IPv4: Your AWS subnet and create the VPN connection

For more details on creating a VPN connection, you can follow my other blog: Configure AWS VPN Connectivity to a Third-Party Network

Peering networks using the VPN

Create a local network gateway in Azure and make a connection

In this typical Virtual Private Network (VPN) setup between Azure and AWS, after configuring the necessary components like Virtual Network Gateways (VNG) on Azure and Virtual Private Gateways (VPG) on AWS, you will see a status change in both the platforms indicating the VPN connection has been established successfully.

In a short while, you will see the status as “Connected” At the same time, in AWS, you will see the status of the VPN connection as “UP” in the VPN Connections section. This indicates that the VPN tunnel has been established, and the VPN connection between the two platforms is now active.

To route the data to/from Azure to AWS, enter the subnet’s route table where your aws resources are present, such as EC2 machines.

The route should have Destination as Azure Vnet CIDR and Target as AWS VPG ID

Creating a private link to Amazon S3

For private data transfers between Amazon VPCs and Amazon S3, AWS offers Amazon S3 Private Link (VPC endpoints). Private Link allows clients to access Amazon S3 over a private connection within their AWS VPC without using public IP addresses or traversing the internet for secure data transfers.

To create private connectivity from Azure VMs to Amazon S3 buckets in AWS, you can use an AWS PrivateLink or interface endpoint. This enables you to access Amazon S3 buckets in AWS over a private connection without exposing your data to the internet.

Here are the steps to create an interface endpoint for Amazon S3 in AWS:

• Navigate to the Amazon VPC console.
• In the left navigation pane, select “Endpoints” and click the “Create Endpoint” button.
• In the “Create Endpoint” wizard, select the “AWS services” option for “Service category” and select “com.amazonaws.<region>.s3” as the service name for “Service Name”.
• Choose the VPC in which you want to create the endpoint. In this case, select “VPC-22”.
• Select the subnets in which you want to create the endpoint. Ensure the subnets are associated with the same VPC as the endpoint.
• For the security group, select “Create new security group” and give it a name. In the rules section, add a rule to allow HTTPS traffic from the Azure VNet CIDR range “10.0.0.0/16”.
• Click “Create Endpoint” to create the Amazon S3 interface endpoint.

Now that you have created the Amazon S3 interface endpoint in AWS, you can access the S3 buckets over the private connection from your Azure VMs.

Some commands to access the Amazon S3 bucket contents from an Azure VM or on-premises machine:

• Install the AWS CLI on your Azure VM or on-premises machine.
• Configure the AWS CLI with the credentials of an IAM user with permission to access the S3 bucket.
• To list the Amazon S3 bucket’s contents using the following command:

• Replace “your-bucket-name” and “your-s3-endpoint-dns-name” with the names of your Amazon S3 bucket and Amazon S3 VPC endpoint, respectively.
• To copy a file from the Amazon S3 bucket to your Azure virtual machine or on-premises computer, use the following command:

• Place your bucket name in “your-bucket-name”, <your-file-name> with the name of the file you want to copy, and <your-s3-endpoint-dns-name> with the DNS name mentioned in the S3 VPC endpoint.

You will then see objects added to your local path, as mentioned.

Conclusion

The article explains how to access AWS S3 buckets and objects from on-premises over VPN. AWS provides private connectivity options for Amazon S3, including Site-to-Site VPN and AWS Direct Connect, which allow clients to establish a private and secure connection between their on-premises resources and S3. Additionally, AWS offers Amazon S3 Private Link, which allows clients to access Amazon S3 over a private connection within their AWS VPC. The article then provides a step-by-step guide on creating a VPN using Azure and AWS and fetching Amazon S3 objects over VPN privately. The guide includes creating Azure VPN resources, VPN resources, peering networks using the VPN, and creating a private link to Amazon S3.

About CloudThat