Upload Large Files Efficiently using Amazon S3 Multi-Part Upload with Console and Script

Introduction

In our previous blog, “Storage Made Easy with Amazon S3“, we discussed uploading objects to Amazon S3 buckets. However, it becomes important to use the multi-part upload feature when it comes to uploading large files. This feature allows uploads to be broken into parts, allowing for resumable and parallel uploads.
Amazon S3 is a popular cloud storage service that Amazon Web Services (AWS) provides scalable and secure storage options for your data. One of the key features of S3 is the ability to upload large files, making it an excellent option for storing and sharing big data. This blog will explore the steps to create a life cycle policy for multi-part uploads in the AWS console and a Python script to perform multi-part uploads.

Overview of Amazon S3 Life-Cyle Rule

The Amazon S3 Life Cycle Rule is a feature in Amazon S3 that helps automate moving or deleting objects in a bucket over time. Essentially, you can set up a rule that specifies how long an object should stay in a certain storage class or if it should be deleted entirely. This way, you can reduce storage costs and ensure you only keep the data you need.

Multi-part upload is a feature that allows you to upload large files in parts, making it easier to manage and reducing the risk of upload failures. Using the S3 Life Cycle Rule, you can ensure that the parts of a multi-part upload are stored efficiently, reducing costs, and are deleted when they are no longer needed. This helps maintain the organization of your data, reduces the risk of running out of storage space, and simplifies your data management over time.

Step by Step guide to Create Life Cycle Rule

Step 1: Log in to AWS Management Console and search S3 in the search bar. Then, Select S3.

Step 2: Select the Bucket you want to create a life cycle rule for.

Step 3:  Choose the Management tab, and choose Create lifecycle rule.

Step 4: Enter a name for your rule and choose the scope of the life cycle policy. You can apply the rule to all objects in the bucket or limit the scope to specific prefixes or tags.

Step 5: Check On Delete incomplete multipart uploads & Enter the number of days and click on Create rule.

Step 6: After creating, you can upload a single file which more than 5GB.

Step 7: Here is a 9.8 GB Single file Multi-Part Upload

Multi-Part Upload using Python Script

When uploading large files, it is recommended to use the multi-part upload feature in Amazon S3. This allows you to upload parts of a large file in parallel, improving the upload speed and reliability.

Here is a brief overview of the Python script to perform multi-part uploads in S3:

1. Import the boto3 and os libraries.
2. Create a boto3 client for Amazon S3.
3. Define the name of the bucket and the file path for the large file to be uploaded.
4. Determine the file size and use an if-else statement to perform a regular or multi-part upload.
5. For multi-part uploads, define the chunk size, create a multipart upload, and upload each file chunk in parallel.

It’s important to note that this is just a brief overview of the Python script.

Conclusion

The AWS S3 multi-part upload feature is a great way to upload large files efficiently and securely. Creating life cycle policies in the AWS console helps you manage your data effectively, and the Python script provides a convenient way to perform multi-part uploads in Amazon S3.

In case you need the script for Multi-Part Upload in Amazon S3, please contact our Technical Team. 

About CloudThat