Centralized Automated Tagging Strategy with AWS Services

Overview

In the rapidly evolving cloud ecosystem, efficient resource management and cost allocation are paramount for organizations leveraging AWS services. This guide introduces a strategic approach to implementing an automated tagging system across AWS Organizations to enhance cost management and operational oversight.

Introduction

This concise guide explores leveraging core AWS services—AWS Config, AWS Organizations, Amazon DynamoDB, Amazon EventBridge, and AWS Systems Manager—to create a unified, automated tagging framework.

Aimed at organizations seeking to streamline cost allocation and ensure compliance, this introduction outlines the integration of these services to facilitate consistent tagging practices across AWS resources.

By adopting this approach, businesses can overcome common tagging challenges, improve operational transparency, and establish a solid foundation for accurate cost management and governance.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Establishing Requirements and Practical Applications for a Tagging Framework

A tagging strategy involves roles like resource management and cost tracking, initiated by a team from Finance to IT and Security, aiming to establish the needs of a tagging system. A standard tagging schema includes case-sensitive tag keys (e.g., CostCenter) and values (e.g., Production).

Use mechanisms for validation: Typical issues and obstacles in unenforced tagging include:

1) a lack of universal awareness of tagging requirements across teams, leading to non-compliance with an established tagging taxonomy and

2) inconsistencies in tagging due to varied infrastructure provisioning processes.

Explore further in our dedicated blog post on the topic to gain a deeper understanding of these common challenges and how to address them.

Architecture overview

In the following sections of this blog post, we will outline the steps to implement a cost allocation tagging strategy effectively across multiple accounts within an organization using AWS Organizations. Additionally, in the second part of this post, we will provide practical code examples corresponding to each of these steps.

A user generates an AWS resource without tags.
AWS Config tracks and validates changes in an organization’s cloud setup against predefined rules, offering ready-made and customizable rule options for comprehensive monitoring and compliance.
We use AWS Config and Custom Lambda Rules for validation and reporting, leveraging AWS Lambda. The ‘required-tags’ managed rule checks up to 6 tags but may not cover all resource types, necessitating custom rules for broader coverage. Consult AWS documentation for details on supported resources.
Administrators set tagging policies in an Amazon DynamoDB table, using AWS EventBridge to automatically collect AWS Organizations metadata like account IDs and contact details into another Amazon DynamoDB table, ensuring new accounts are seamlessly integrated.

Your tagging entries could look like this:

{
"ResourceType": "AWS::EC2::*",
"Tag": "MyTag1",
"Enabled": true,
"Required": true,
"ValuePattern": "[A-Z][a-z]",
"Values": [],
"AccountIds": []
}

{

"ResourceType": "AWS::EC2::*",

"Tag": "MyTag1",

"Enabled": true,

"Required": true,

"ValuePattern": "[A-Z][a-z]",

"Values": [],

"AccountIds": []

}

The schema defines tagging rules for resource types, using wildcards for broader applications. It specifies tag names’ enforcement requirements and allows the use of regex patterns or value lists for validation. It’s designed for multi-account environments, enabling tag restrictions for specific accounts or groups.

5. AWS Config logs its details when a user creates a resource not covered by managed rules. It triggers an AWS Lambda function via a custom rule to fetch tagging criteria from a centralized DynamoDB schema.

6. The custom Lambda function cross-references the tagging requirements outlined in the Amazon DynamoDB schema with the tags in the Resource Group (5). Based on this comparison, AWS Lambda communicates its status to the AWS Config rule (6), categorizing the resource as COMPLIANT, NON_COMPLIANT, or NOT_APPLICABLE.

Customizing validation methods tailors feedback and enforces tagging rules, like ensuring all production resources are tagged with ‘Env = Production’ and ‘CostCenter = apps’. Implement systems to handle untagged resources, such as automatic shutdowns or CI/CD pipeline triggers for compliance.

The validation engine could be implemented like this:

import os
import platform
import re
from typing import Dict, List, Iterable

import boto3
from boto3.dynamodb.conditions import Key

from exceptions import NonCompliantException, NotApplicableException
from models import Resource, TagRule

TABLE_NAME = os.environ.get("TABLE_NAME")
if 'macos' in (platform := platform.platform().lower()):
    boto3.setup_default_session(profile_name="gaborsch-Admin")
dynamodb = boto3.resource("dynamodb")
table = dynamodb.Table(TABLE_NAME)

tagging_api = boto3.client("resourcegroupstaggingapi")


def check_tags(configuration_item: Dict):
    """Entrypoint for checking tags"""
    resource_type = configuration_item.get("resourceType")
    aws_account_id = configuration_item.get('awsAccountId')
    tag_requirements = get_tag_requirements(resource_type, aws_account_id)

    if not tag_requirements:
        return

    resource_arn = configuration_item.get("ARN")
    resource_id = configuration_item.get("resourceId")
    resource = get_resource_tags(resource_arn, resource_id, resource_type)
    validate_resource_tags(tag_requirements, resource)


def get_tag_requirements(resource_type: str, aws_account_id: str) -> List:
    """Get tagging requirements"""
    resource_type_split = resource_type.split("::")
    resource_types = []

    for i in range(len(resource_type_split) - 1):
        resource_types.append("::".join(resource_type_split[0 : i + 1]) + "::*")

    resource_types.append(resource_type)
    print(resource_types)

    db_items = []
    for key in resource_types:
        response = table.query(KeyConditionExpression=Key("ResourceType").eq(key))
        rules = map(lambda x: TagRule(**x), response.get("Items", []))
        rules = filter_rules_by_account(rules=rules, aws_account_id=aws_account_id)
        db_items.extend([x for x in rules if x.Enabled])

    return db_items


def filter_rules_by_account(rules: Iterable[TagRule], aws_account_id: str) -> List[TagRule]:
    """Filter rules that are relevant for the given account"""
    result = [x for x in rules if len(x.AccountIds) == 0 or aws_account_id in x.AccountIds]
    return result


def get_resource_tags(resource_arn: str, resource_id: str, resource_type: str) -> Resource:
    """Get the tags of the resource"""
    result = {}
    response = tagging_api.get_resources(ResourceARNList=[resource_arn])
    for resource in response.get("ResourceTagMappingList", []):
        for tag in resource.get("Tags", []):
            result.update({tag.get("Key"): tag.get("Value")})
    return Resource(resource_type, resource_id, resource_arn, result)


def validate_resource_tags(tag_requirements: List[TagRule], resource: Resource):
    """Validation engine for resource tags"""
    for tag_rule in tag_requirements:
        print(f"Validating {tag_rule}...")
        if tag_rule.Required:
            validate_tag_existence(tag_rule, resource)
            if tag_rule.ValuePattern:
                validate_tag_regex(tag_rule, resource)
            else:
                validate_tag_values(tag_rule, resource)
        else:
            validate_tag_values(tag_rule, resource)


def validate_tag_regex(tag_rule: TagRule, resource: Resource):
    """Validate tags based on the ValuePattern regular expression"""
    if not tag_rule.ValuePattern:
        return

    tag_value = resource.Tags.get(tag_rule.Tag)
    value_match = re.fullmatch(tag_rule.ValuePattern, tag_value)
    if not value_match:
        raise NonCompliantException(
            f'{resource.ResourceId} ({resource.ResourceType}) tag value "{tag_value}" violates dictionary regex pattern for "{tag_rule.Tag}": {tag_rule.ValuePattern}'
        )


def validate_tag_values(tag_rule: TagRule, resource: Resource):
    """Validate tags based on the Values list"""
    tag_value = resource.Tags.get(tag_rule.Tag)
    if tag_value and tag_value not in tag_rule.Values:
        raise NonCompliantException(
            f'{resource.ResourceId} ({resource.ResourceType}) tag value "{tag_value}" violates dictionary for "{tag_rule.Tag}": {tag_rule.Values}'
        )


def validate_tag_existence(tag_rule: TagRule, resource: Resource):
    """Validate if the tag exists"""
    if tag_rule.Tag not in resource.Tags:
        raise NonCompliantException(
            f"{resource.ResourceId} ({resource.ResourceType}) missing required tag {tag_rule.Tag}"
        )

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

import os

import platform

import re

from typing import Dict, List, Iterable

import boto3

from boto3.dynamodb.conditions import Key

from exceptions import NonCompliantException, NotApplicableException

from models import Resource, TagRule

TABLE_NAME = os.environ.get("TABLE_NAME")

if 'macos' in (platform := platform.platform().lower()):

boto3.setup_default_session(profile_name="gaborsch-Admin")

dynamodb = boto3.resource("dynamodb")

table = dynamodb.Table(TABLE_NAME)

tagging_api = boto3.client("resourcegroupstaggingapi")

def check_tags(configuration_item: Dict):

"""Entrypoint for checking tags"""

resource_type = configuration_item.get("resourceType")

aws_account_id = configuration_item.get('awsAccountId')

tag_requirements = get_tag_requirements(resource_type, aws_account_id)

if not tag_requirements:

return

resource_arn = configuration_item.get("ARN")

resource_id = configuration_item.get("resourceId")

resource = get_resource_tags(resource_arn, resource_id, resource_type)

validate_resource_tags(tag_requirements, resource)

def get_tag_requirements(resource_type: str, aws_account_id: str) -> List:

"""Get tagging requirements"""

resource_type_split = resource_type.split("::")

resource_types = []

for i in range(len(resource_type_split) - 1):

resource_types.append("::".join(resource_type_split[0 : i + 1]) + "::*")

resource_types.append(resource_type)

print(resource_types)

db_items = []

for key in resource_types:

response = table.query(KeyConditionExpression=Key("ResourceType").eq(key))

rules = map(lambda x: TagRule(**x), response.get("Items", []))

rules = filter_rules_by_account(rules=rules, aws_account_id=aws_account_id)

db_items.extend([x for x in rules if x.Enabled])

return db_items

def filter_rules_by_account(rules: Iterable[TagRule], aws_account_id: str) -> List[TagRule]:

"""Filter rules that are relevant for the given account"""

result = [x for x in rules if len(x.AccountIds) == 0 or aws_account_id in x.AccountIds]

return result

def get_resource_tags(resource_arn: str, resource_id: str, resource_type: str) -> Resource:

"""Get the tags of the resource"""

result = {}

response = tagging_api.get_resources(ResourceARNList=[resource_arn])

for resource in response.get("ResourceTagMappingList", []):

for tag in resource.get("Tags", []):

result.update({tag.get("Key"): tag.get("Value")})

return Resource(resource_type, resource_id, resource_arn, result)

def validate_resource_tags(tag_requirements: List[TagRule], resource: Resource):

"""Validation engine for resource tags"""

for tag_rule in tag_requirements:

print(f"Validating {tag_rule}...")

if tag_rule.Required:

validate_tag_existence(tag_rule, resource)

if tag_rule.ValuePattern:

validate_tag_regex(tag_rule, resource)

else:

validate_tag_values(tag_rule, resource)

else:

validate_tag_values(tag_rule, resource)

def validate_tag_regex(tag_rule: TagRule, resource: Resource):

"""Validate tags based on the ValuePattern regular expression"""

if not tag_rule.ValuePattern:

return

tag_value = resource.Tags.get(tag_rule.Tag)

value_match = re.fullmatch(tag_rule.ValuePattern, tag_value)

if not value_match:

raise NonCompliantException(

f'{resource.ResourceId} ({resource.ResourceType}) tag value "{tag_value}" violates dictionary regex pattern for "{tag_rule.Tag}": {tag_rule.ValuePattern}'

)

def validate_tag_values(tag_rule: TagRule, resource: Resource):

"""Validate tags based on the Values list"""

tag_value = resource.Tags.get(tag_rule.Tag)

if tag_value and tag_value not in tag_rule.Values:

raise NonCompliantException(

f'{resource.ResourceId} ({resource.ResourceType}) tag value "{tag_value}" violates dictionary for "{tag_rule.Tag}": {tag_rule.Values}'

)

def validate_tag_existence(tag_rule: TagRule, resource: Resource):

"""Validate if the tag exists"""

if tag_rule.Tag not in resource.Tags:

raise NonCompliantException(

f"{resource.ResourceId} ({resource.ResourceType}) missing required tag {tag_rule.Tag}"

)

7. For non-compliant resources, use AWS Systems Manager Automation for automated correction, like updating AWS CloudFormation stacks or alerting via Amazon SNS. Systems Manager Automation runbooks can also support operational tasks.

Conclusion

You’ve observed how straightforward it is to initiate the creation of a tagging dictionary and develop a cost allocation strategy. This process equips management and operations teams with crucial data on cost utilization, aiding them in decision-making. The next step could involve identifying and rectifying further non-compliant resources within your AWS Organizations. This can be achieved by implementing AWS Config custom rules, which can trigger automated remediation actions through AWS Systems Manager, thereby ensuring compliance based on the outcomes of the rule evaluations.

Drop a query if you have any questions regarding Tag Management and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

Accelerated cloud migration
End-to-end view of the cloud environment

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

CloudThat is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, and many more.

To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.

FAQs

1. How can AWS Config and Custom Lambda Rules be used to manage tagging and cost allocation?

ANS: – AWS Config and Custom Lambda Rules ensure proper resource tagging in AWS Organizations. They use AWS Lambda functions to match resources with a tagging schema in Amazon DynamoDB, complying with AWS Config’s ‘required-tags’ rule. This method supports more resources than standard AWS Config rules, helping maintain consistent tagging essential for cost management.

2. What are the steps for addressing non-compliant resources in AWS Organizations?

ANS: – AWS Systems Manager automates remediation for non-compliant resources (violating tagging policies). Custom AWS Config rules detect these resources, and AWS Systems Manager Automation corrects them. This process, crucial for tagging discipline, enhances resource tracking and cost allocation. AWS Config’s dashboard also allows administrators to monitor and address rule violations.

WRITTEN BY Naman Jain

Naman works as a Research Intern at CloudThat. With a deep passion for Cloud Technology, Naman is committed to staying at the forefront of advancements in the field. Throughout his time at CloudThat, Naman has demonstrated a keen understanding of cloud computing and security, leveraging his knowledge to help clients optimize their cloud infrastructure and protect their data. His expertise in AWS Cloud and security has made him an invaluable team member, and he is constantly learning and refining his skills to stay up to date with the latest trends and technologies.