Automating Data Management with Amazon S3 Bucket Lifecycle Policies using Scripts

Introduction

Amazon Simple Storage Service (S3) is a highly scalable and secure object storage service offered by Amazon Web Services (AWS). Amazon S3 provides businesses with a cost-effective solution for storing and retrieving vast data. AWS offers a powerful feature called Lifecycle Policies to optimize storage costs and improve performance. These policies automate the management of objects within Amazon S3 buckets, enabling users to define rules based on factors like object age, access patterns, or specific object tags. By automating these rules, you can efficiently manage your data and ensure compliance with data retention policies.

In this blog, we will explore the benefits of Lifecycle Policies and guide you through implementing them on your Amazon S3 buckets using scripts. By the end of this tutorial, you will be equipped with the knowledge to harness the full potential of Amazon S3 and automate data management for your applications.

Lifecycle Policies in Amazon S3

Lifecycle Policies in Amazon S3 define how objects should be managed over time. These rules are based on various factors, such as the age of objects, their usage, or specific object tags. Automating these rules allows you to optimize data storage and access costs while maintaining data availability and compliance.

The primary benefits of implementing Lifecycle Policies include the following:

Cost Optimization – Automatically transitioning less frequently accessed objects to lower-cost storage tiers, such as Amazon Glacier or Intelligent-Tiering, can significantly reduce storage costs.
Improved Performance – Lifecycle Policies can automatically move older data to cheaper storage classes, ensuring frequently accessed data remains in faster retrieval tiers, thereby optimizing performance.
Data Retention and Compliance – Set up automatic object expiration or transition to Glacier Deep Archive for long-term archival to comply with data retention policies and regulatory requirements.
Simplified Data Management – By eliminating the need for manual intervention, Lifecycle Policies streamline data management processes and reduce the potential for human error.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Prerequisites for Implementation

Before we proceed with implementing Lifecycle Policies using scripts, ensure that you have the following prerequisites in place:

AWS Account – Sign up for an AWS account if you do not have one already, and ensure you have access to the Amazon S3 service and permissions to configure Lifecycle Policies.
AWS Command Line Interface (CLI) – Install the AWS CLI on your local machine, as we will use it to interact with AWS services and apply the Lifecycle Policy.
Basic AWS Knowledge – Familiarize yourself with basic AWS services and working with the command line.

Step-by-Step Guide

Step 1 – Configuring AWS CLI or Visual Basic.

The first step is to set up the AWS CLI with your account credentials. Open a terminal or command prompt and enter the following command:

aws configure

1	aws configure

Using the “aws configure” command, set up the CLI using your AWS account credentials. Enter your AWS Access Key ID, Secret Access Key, default region, and output format.

Step 2 – Creating the Lifecycle Policy Script

Now that the AWS CLI is configured let’s create a script to define the Lifecycle Policy for your Amazon S3 bucket.

import boto3

def create_lifecycle_policy(bucket_name, source_storage_class, destination_storage_class, prefix):
    # Create AWS S3 client
    s3_client = boto3.client(
        "s3",
    )

    rule = {
        'Status': 'Enabled',
        'Prefix': prefix,  
        'Transitions': [
            {
                'StorageClass': destination_storage_class,
                'Days': 30  
            },
            {
                'StorageClass': 'DEEP_ARCHIVE',
                'Days': 90
            }
        ],
        'Expiration': {
            # 'Date': datetime(2015, 1, 1),
            'Days': 91,
            # 'ExpiredObjectDeleteMarker': True|False
        }
    }

    response = s3_client.put_bucket_lifecycle_configuration(
        Bucket=bucket_name,
        LifecycleConfiguration={
            'Rules': [rule]
        }
    )

    if response['ResponseMetadata']['HTTPStatusCode'] == 200:
        print(f"Lifecycle policy created for bucket '{bucket_name}' successfully.")
    else:
        print(f"Failed to create lifecycle policy for bucket '{bucket_name}'.")

bucket_name = 'ninjaa'
source_storage_class = 'STANDARD'  
destination_storage_class = 'STANDARD_IA' 
prefix=''

create_lifecycle_policy(bucket_name, source_storage_class, destination_storage_class, prefix)

import boto3

def create_lifecycle_policy(bucket_name, source_storage_class, destination_storage_class, prefix):

# Create AWS S3 client

s3_client = boto3.client(

"s3",

)

rule = {

'Status': 'Enabled',

'Prefix': prefix,

'Transitions': [

{

'StorageClass': destination_storage_class,

'Days': 30

{

'StorageClass': 'DEEP_ARCHIVE',

'Days': 90

}

'Expiration': {

# 'Date': datetime(2015, 1, 1),

'Days': 91,

# 'ExpiredObjectDeleteMarker': True|False

}

response = s3_client.put_bucket_lifecycle_configuration(

Bucket=bucket_name,

LifecycleConfiguration={

'Rules': [rule]

}

)

if response['ResponseMetadata']['HTTPStatusCode'] == 200:

print(f"Lifecycle policy created for bucket '{bucket_name}' successfully.")

else:

print(f"Failed to create lifecycle policy for bucket '{bucket_name}'.")

bucket_name = 'ninjaa'

source_storage_class = 'STANDARD'

destination_storage_class = 'STANDARD_IA'

prefix=''

create_lifecycle_policy(bucket_name, source_storage_class, destination_storage_class, prefix)

Step 3 – Applying the Lifecycle Policy

Here we have defined the rule in which the storage class will change to Standard IA after 30 days from its creation date and then will move to the Amazon Glacier Deep archive after 90 days.
We have also defined the date of deletion of the object, which is 91 days from the date of its creation.
We must define the bucket name, source storage class Standard by default in Amazon S3, and destination storage class.
Run the script with the correct credentials in the terminal or cmd.

step3

step3b

Conclusion

Using scripts to implement Lifecycle Policies on Amazon S3 buckets may vastly improve your data management capabilities. You may optimize storage costs, increase speed, and maintain compliance with data retention requirements by automating transitions and expirations based on established conditions.

We discussed the advantages of utilizing Lifecycle Policies in this blog and a step-by-step approach to setting and deploying a policy to an Amazon S3 bucket using AWS CLI commands. Consider investigating more complex options, including versioning, cross-region replication, and event-driven actions, as you progress along your cloud journey to improve the efficiency and robustness of your data management on Amazon S3.

Drop a query if you have any questions regarding Amazon S3 and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is an official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, AWS EKS Service Delivery Partner, and Microsoft Gold Partner, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best-in-industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.

FAQs

1. Can I have multiple rules in a Lifecycle Policy?

ANS: – Yes, you can have multiple rules in a Lifecycle Policy. A Lifecycle Policy can have many rules. Each rule can have its circumstances, such as prefixes, transition days, and expiration dates. Having many rules lets you handle objects with distinct properties differently, optimizing your data management approach.

2. What happens if I apply a new Lifecycle Policy to an existing bucket with objects already present?

ANS: – When you apply a new Lifecycle Policy to an existing bucket, the policy applies to all items in the bucket, including those already there. Any items that fulfill the policy’s requirements will have the actions, such as storage class transitions or expirations, applied immediately. Remember that once a policy is implemented, it cannot be reversed, so make sure it is well-defined and meets your data management needs.

3. Can I modify or remove a Lifecycle Policy from an Amazon S3 bucket using scripts?

ANS: – Yes, you may use scripts to alter or delete a Lifecycle Policy from an Amazon S3 bucket. To amend an existing policy, edit the JSON configuration file and apply it to the bucket again using the put-bucket-lifecycle-configuration command. If you want to entirely delete the policy, create a new JSON file with an empty rule array and apply it to the bucket. However, be cautious when eliminating rules because it may result in unforeseen effects such as missing data transitions or expirations. Always examine and test policy changes before implementing them in production environments.

WRITTEN BY Ayush Agarwal

Ayush Agarwal works as a Research Associate at CloudThat. He has excellent analytical thinking and carries an optimistic approach toward his life. He is having sound Knowledge of AWS Cloud Services, Infra setup, Security, WAR, and Migration. He is always keen to learn and adopt new technologies.