Automating Amazon CloudWatch Log Retention with AWS Lambda and Amazon EventBridge

Introduction

Managing Amazon CloudWatch Logs efficiently is crucial for both observability and cost optimization. By default, when new log groups are created in Amazon CloudWatch, their retention period is set to “Never Expire.” While this ensures logs are always available, it also results in unnecessary storage costs over time, especially in environments where logs are rarely accessed after a certain period.

To address this, I built a lightweight, serverless solution using AWS Lambda and Amazon EventBridge that automatically updates the retention period of newly created Amazon CloudWatch log groups to 90 days. This proactive approach ensures that logs are retained meaningfully while optimizing long-term storage costs.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Overview of the Solution

The core idea is simple: listen for the creation of new log groups using an Amazon EventBridge rule, and immediately trigger a Lambda function to check and enforce a 90-day retention policy if one isn’t already set.

This setup eliminates the need for manual intervention or periodic audits of log groups and allows teams to maintain compliance with data retention policies while controlling costs.

AWS Services Used

Here’s a quick overview of the AWS services used in this solution:

Amazon CloudWatch Logs: For storing and managing log data from various AWS services.
AWS Lambda: A serverless computing service that runs the logic that updates log group retention policies.
Amazon EventBridge: To monitor AWS API calls and trigger the Lambda function whenever a new log group is created.
AWS Identity and Access Management (IAM): To grant necessary permissions for the Lambda function to read and update Amazon CloudWatch log group configurations.

The Solution Architecture

The flow of the solution is as follows:

A new log group is created manually by an AWS service (like AWS Lambda, Amazon ECS, or Amazon API Gateway).
An Amazon EventBridge rule listens for the CreateLogGroup API event via AWS CloudTrail.
The AWS Lambda function is triggered, and the log group name is received as part of the event payload.
AWS Lambda describes the log group to check its current retention policy.
If retention is not set or differs from the default (90 days), it updates the retention policy accordingly.

This automation ensures consistent log group configuration across your AWS environment.

AWS Lambda Function Code

Here’s the complete AWS Lambda function that powers this solution:

import boto3
import os
import json

logs_client = boto3.client('logs')
DEFAULT_RETENTION_DAYS = int(os.environ.get('DEFAULT_RETENTION_DAYS', 90))

def lambda_handler(event, context):
    try:
        log_group_name = event['detail']['requestParameters']['logGroupName']
        print(f"Detected new log group: {log_group_name}")
    except KeyError as e:
        print(f"Error: Missing key: {e}")
        return {'statusCode': 400, 'body': 'Invalid event structure'}

    try:
        response = logs_client.describe_log_groups(
            logGroupNamePrefix=log_group_name,
            limit=1
        )
        current_retention_in_days = None
        for lg in response['logGroups']:
            if lg['logGroupName'] == log_group_name:
                current_retention_in_days = lg.get('retentionInDays')
                break

        if current_retention_in_days is None or current_retention_in_days != DEFAULT_RETENTION_DAYS:
            logs_client.put_retention_policy(
                logGroupName=log_group_name,
                retentionInDays=DEFAULT_RETENTION_DAYS
            )
            print(f"Retention updated to {DEFAULT_RETENTION_DAYS} days.")
        else:
            print(f"No update needed for {log_group_name}.")

        return {'statusCode': 200, 'body': 'Processed successfully'}

    except logs_client.exceptions.ResourceNotFoundException:
        print(f"Log group {log_group_name} not found.")
        return {'statusCode': 404, 'body': 'Log group not found'}
    except Exception as e:
        print(f"Unexpected error: {e}")
        raise

import boto3

import os

import json

logs_client = boto3.client('logs')

DEFAULT_RETENTION_DAYS = int(os.environ.get('DEFAULT_RETENTION_DAYS', 90))

def lambda_handler(event, context):

try:

log_group_name = event['detail']['requestParameters']['logGroupName']

print(f"Detected new log group: {log_group_name}")

except KeyError as e:

print(f"Error: Missing key: {e}")

return {'statusCode': 400, 'body': 'Invalid event structure'}

try:

response = logs_client.describe_log_groups(

logGroupNamePrefix=log_group_name,

limit=1

)

current_retention_in_days = None

for lg in response['logGroups']:

if lg['logGroupName'] == log_group_name:

current_retention_in_days = lg.get('retentionInDays')

break

if current_retention_in_days is None or current_retention_in_days != DEFAULT_RETENTION_DAYS:

logs_client.put_retention_policy(

logGroupName=log_group_name,

retentionInDays=DEFAULT_RETENTION_DAYS

)

print(f"Retention updated to {DEFAULT_RETENTION_DAYS} days.")

else:

print(f"No update needed for {log_group_name}.")

return {'statusCode': 200, 'body': 'Processed successfully'}

except logs_client.exceptions.ResourceNotFoundException:

print(f"Log group {log_group_name} not found.")

return {'statusCode': 404, 'body': 'Log group not found'}

except Exception as e:

print(f"Unexpected error: {e}")

raise

Setting Up the Amazon EventBridge Rule

To trigger the AWS Lambda function, configure an Amazon EventBridge rule with the following event pattern:

{
  "source": ["aws.logs"],
  "detail-type": ["AWS API Call via CloudTrail"],
  "detail": {
    "eventSource": ["logs.amazonaws.com"],
    "eventName": ["CreateLogGroup"]
  }
}

{

"source": ["aws.logs"],

"detail-type": ["AWS API Call via CloudTrail"],

"detail": {

"eventSource": ["logs.amazonaws.com"],

"eventName": ["CreateLogGroup"]

}

Target the AWS Lambda function you’ve deployed. Ensure that your Amazon EventBridge can read from Amazon CloudTrail events.

AWS IAM Permissions

Your AWS Lambda function will need permissions to:

Describe log groups
Put retention policies

Example AWS IAM policy snippet:

{
  "Effect": "Allow",
  "Action": [
    "logs:DescribeLogGroups",
    "logs:PutRetentionPolicy"
  ],
  "Resource": "*"
}

{

"Effect": "Allow",

"Action": [

"logs:DescribeLogGroups",

"logs:PutRetentionPolicy"

"Resource": "*"

}

Also, ensure that the AWS Lambda execution role has permission to write logs (standard Amazon CloudWatch permissions).

Cost Consideration

Storing logs indefinitely can significantly increase costs, especially in environments generating large volumes of log data. By enforcing a 90-day retention period, you’re:

Avoiding unnecessary long-term storage charges
Ensuring only relevant logs are kept
Keeping your log management process compliant and clean

Depending on your usage, this simple automation can save hundreds (or even thousands) of dollars over time.

Conclusion

Amazon CloudWatch Logs are invaluable for monitoring and debugging, but without thoughtful retention policies, they can become a silent budget drainer. With the help of AWS Lambda and Amazon EventBridge, you can enforce log retention automatically, ensuring compliance and cost efficiency with minimal effort.

By integrating this solution into your infrastructure, you’re simplifying operations and promoting best practices in AWS cost governance.

Drop a query if you have any questions regarding Amazon CloudWatch Logs and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is an award-winning company and the first in India to offer cloud training and consulting services worldwide. As a Microsoft Solutions Partner, AWS Advanced Tier Training Partner, and Google Cloud Platform Partner, CloudThat has empowered over 850,000 professionals through 600+ cloud certifications winning global recognition for its training excellence including 20 MCT Trainers in Microsoft’s Global Top 100 and an impressive 12 awards in the last 8 years. CloudThat specializes in Cloud Migration, Data Platforms, DevOps, IoT, and cutting-edge technologies like Gen AI & AI/ML. It has delivered over 500 consulting projects for 250+ organizations in 30+ countries as it continues to empower professionals and enterprises to thrive in the digital-first world.

FAQs

1. Can I set a different retention period instead of 90 days?

ANS: – Yes, you can change the value of DEFAULT_RETENTION_DAYS in the Lambda environment variable or directly in the code.

2. Will this AWS Lambda function affect existing log groups?

ANS: – No, it only triggers when a new log group is created.

3. What if I want different retention periods for different log groups?

ANS: – You can extend the logic to inspect the log group name or associated tags and set retention accordingly.

WRITTEN BY Deepak S

Deepak S is a Senior Research Associate at CloudThat, specializing in AWS services. He is passionate about exploring new technologies in cloud and is also an automobile enthusiast.