Voiced by Amazon Polly |
Introduction
In dynamic cloud environments, it’s crucial to have efficient and automated processes to manage the lifecycle of your Amazon EC2 instances, especially when leveraging warm pools for cost efficiency.
In this blog post, we’ll walk you through a comprehensive approach to automating code deployment to Amazon EC2 instances within a warm pool using AWS CodeDeploy. We’ll use AWS Lambda to manage the lifecycle of these instances, ensuring that they are in the correct state during deployment and return to a hibernated state afterward. This automation simplifies the deployment process and ensures that your applications are always up-to-date with minimal manual intervention.
Objectives
- Create resources using the AWS CloudFormation template.
- Set up AWS Lambda function that:
- Identifies Amazon EC2 instances based on their tags.
- Manages the lifecycle of instances in a warm pool.
- Triggers deployments using AWS CodeDeploy.
- Ensure instances run during deployment and return to a stopped state after deployment.
Pioneers in Cloud Consulting & Migration Services
- Reduced infrastructural costs
- Accelerated application deployment
Prerequisites
- Basic knowledge of AWS services such as Amazon EC2, AWS Lambda, AWS CloudFormation, and AWS CodeDeploy.
- AWS CLI and necessary permissions to create and manage AWS resources.
Step-by-Step Guide
Step 1: AWS CloudFormation Template
We will start by creating the necessary resources using AWS CloudFormation template. This template sets up:
- An Amazon EventBridge rule to trigger the Lambda function when an Amazon EC2 instance changes state.
- Necessary AWS IAM roles and permissions.
- The AWS Lambda function itself.
AWS CloudFormation Template
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 |
{ "AWSTemplateFormatVersion": "2010-09-09", "Description": "This CloudFormation template creates resources required for syncing new instances with the latest deployment from AWS CodeDeploy based on the CodeDeployAppname, DeploymentGroup tags on the instance", "Resources": { "EventRule": { "Type": "AWS::Events::Rule", "Properties": { "Description": "EventRule", "EventPattern": { "source": [ "aws.ec2" ], "detail-type": [ "EC2 Instance State-change Notification" ], "detail": { "state": [ "running" ] } }, "State": "ENABLED", "Targets": [ { "Arn": { "Fn::GetAtt": [ "CodeDeploySyncNewInstance", "Arn" ] }, "Id": "TargetFunctionV1" } ] } }, "PermissionForEventsToInvokeLambda": { "Type": "AWS::Lambda::Permission", "Properties": { "FunctionName": { "Ref": "CodeDeploySyncNewInstance" }, "Action": "lambda:InvokeFunction", "Principal": "events.amazonaws.com", "SourceArn": { "Fn::GetAtt": [ "EventRule", "Arn" ] } } }, "CodeDeployServiceRole": { "Type": "AWS::IAM::Role", "Properties": { "ManagedPolicyArns": [ "arn:aws:iam::aws:policy/service-role/AWSCodeDeployRole" ], "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "codedeploy.amazonaws.com" ] }, "Action": [ "sts:AssumeRole" ] } ] } } }, "CodeDeployLambdaRole": { "Type": "AWS::IAM::Role", "Properties": { "ManagedPolicyArns": [ "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole", "arn:aws:iam::aws:policy/AmazonEC2ReadOnlyAccess" ], "AssumeRolePolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { "Service": [ "ec2.amazonaws.com", "lambda.amazonaws.com" ] ], "Action": [ "sts:AssumeRole" ] } ] }, "Path": "/", "Policies": [ { "PolicyName": "Lambda-codedeploy", "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Sid": "StartContinuousAssessmentLambdaPolicyStmt", "Effect": "Allow", "Action": [ "codedeploy:Get*", "codedeploy:UpdateDeploymentGroup", "codedeploy:List*", "codedeploy:CreateDeployment" ], "Resource": [ "*" ] } ] } }, { "PolicyName": "IamPassRole", "PolicyDocument": { "Version": "2012-10-17", "Statement": [ { "Sid": "StartContinuousAssessmentLambdaPolicyStmt", "Effect": "Allow", "Action": [ "iam:PassRole" ], "Resource": [ { "Fn::GetAtt": [ "CodeDeployServiceRole", "Arn" ] } ] } ] } } ] } }, "CodeDeploySyncNewInstance": { "Type": "AWS::Lambda::Function", "Properties": { "Role": { "Fn::GetAtt": [ "CodeDeployLambdaRole", "Arn" ] }, "Environment": { "Variables": { "CodeDeployRole": { "Fn::GetAtt": [ "CodeDeployServiceRole", "Arn" ] } } }, "Code": { "ZipFile": { "Fn::Join": [ "\n", [ "def handler(event, context):", " return 'Hello from Lambda'" ] ] } }, "Runtime": "python3.8", "Timeout": 300, "Handler": "new-instance-code-sync.lambda_handler", "MemorySize": 512 } } }, "Outputs": { "NewInstanceLambdaFunction": { "Description": "The Lambda function that creates Resources for syncing new instances", "Value": { "Ref": "CodeDeploySyncNewInstance" } } } } |
Explanation
This template sets up the following:
- An Amazon EventBridge rule to trigger the Lambda function when an EC2 instance state changes to running.
- The necessary AWS IAM roles with permissions to allow the Lambda function to interact with Amazon EC2 and AWS CodeDeploy.
- The AWS Lambda function itself will handle the deployment logic.
Step 2: AWS Lambda Function for Deployment
Next, we will set up the AWS Lambda function. This function will:
- Detect when an instance in the warm pool changes to the running state.
- Use the instance’s tags to identify the application and deployment group.
- Trigger the AWS CodeDeploy deployment.
- Return the instance to the stopped state once deployment is complete.
AWS Lambda Function Code
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 |
from __future__ import print_function from dateutil.parser import parse from datetime import datetime import json import time import os import boto3 ec2client = boto3.client('ec2') client = boto3.client('codedeploy') codedeployrole = os.environ.get("CodeDeployRole") def lambda_handler(event, context): print("Received event: " + json.dumps(event, indent=2, default=str)) instance_id = event['detail']['instance-id'] try: # Ensure the instance is running start_instance(instance_id) wait_for_instance_running(instance_id) tags = get_instance_info(ec2client, instance_id) name = tags.get('Name') if not name: raise CustomException("Name tag not found on the instance") codedeploygroup = f"{name}-deployment-group-asg" appname = f"{name}-application" depresponse = deployment_group_tag(appname, codedeploygroup) print("Deployment group response: " + json.dumps(depresponse, indent=2, default=str)) ec2TagFilters = depresponse['deploymentGroupInfo'].get('ec2TagFilters', []) if not ec2TagFilters: print("Warning: No ec2TagFilters found in deployment group. Adding default tag filter based on Name.") ec2TagFilters = [tag for tag in ec2TagFilters if not (tag['Key'] == 'Name' and tag['Value'] == name)] ec2TagFilters.append({ 'Key': 'Name', 'Value': name, 'Type': 'KEY_AND_VALUE' }) update_response = deployment_group_update(appname, codedeploygroup, ec2TagFilters) print("Deployment group update response: " + json.dumps(update_response, indent=2, default=str)) deploymentId = depresponse['deploymentGroupInfo']['lastSuccessfulDeployment']['deploymentId'] print("DeploymentId is " + deploymentId) depIdResponse = get_dep(deploymentId) bucket = depIdResponse['s3Location']['bucket'] key = depIdResponse['s3Location']['key'] syncresp = deploy_now(appname, codedeploygroup, bucket, key) print(syncresp) # Stop the instance after deployment stop_instance(instance_id) return 'Done' except CustomException as ce: print("Custom exception: " + str(ce)) return 'Done' except KeyError as ke: print("Key error: " + str(ke)) return 'Done' except Exception as e: print("Unhandled exception: " + str(e)) return 'Done' def start_instance(instance_id): ec2client.start_instances(InstanceIds=[instance_id]) def stop_instance(instance_id): ec2client.stop_instances(InstanceIds=[instance_id]) def wait_for_instance_running(instance_id): waiter = ec2client.get_waiter('instance_running') waiter.wait(InstanceIds=[instance_id]) def get_instance_info(ec2client, instance_id): response = ec2client.describe_tags( Filters=[ { 'Name': 'resource-id', 'Values': [instance_id] } ] ) tagdict = {} for tag in response['Tags']: tagdict[tag['Key']] = tag['Value'] return tagdict def deployment_group_tag(appname, codedeploygroup): response = client.get_deployment_group( applicationName=appname, deploymentGroupName=codedeploygroup ) return response def deployment_group_update(appname, codedeploygroup, ec2TagFilters): response = client.update_deployment_group( applicationName=appname, currentDeploymentGroupName=codedeploygroup, ec2TagFilters=ec2TagFilters, serviceRoleArn=codedeployrole ) return response def get_dep(deploymentId): response = client.get_deployment(deploymentId=deploymentId) return response['deploymentInfo']['revision'] def deploy_now(appname, codedeploygroup, bucket, key): response = client.create_deployment( applicationName=appname, deploymentGroupName=codedeploygroup, deploymentConfigName='CodeDeployDefault.OneAtATime', description='Test', ignoreApplicationStopFailures=True, updateOutdatedInstancesOnly=True, fileExistsBehavior='OVERWRITE' ) return response class CustomException(Exception): """Raise for my specific kind of exception""" |
Explanation
- AWS Lambda Handler: Main function that processes the event, extracts the instance ID, and triggers the deployment process.
- Instance Management: Functions to start, stop, and wait for the instance state changes.
- AWS CodeDeploy Management: Functions interacting with AWS CodeDeploy, including retrieving and updating deployment groups and triggering deployments.
Conclusion
Using this setup, you can automate code deployment to instances in a warm pool, ensuring they run during the deployment process and return to a stopped state afterward. This approach leverages the power of AWS services like AWS Lambda, AWS CloudFormation, and AWS CodeDeploy to create and scalable deployment pipeline.
You can easily manage and trigger deployments by tagging instances with appropriate metadata, ensuring your applications are always up-to-date with minimal manual intervention.
Drop a query if you have any questions regarding AWS CodeDeploy and we will get back to you quickly.
Making IT Networks Enterprise-ready – Cloud Management Services
- Accelerated cloud migration
- End-to-end view of the cloud environment
About CloudThat
CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.
CloudThat is the first Indian Company to win the prestigious Microsoft Partner 2024 Award and is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 650k+ professionals in 500+ cloud certifications and completed 300+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, AWS GenAI Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, Amazon ECS Service Delivery Partner, AWS Glue Service Delivery Partner, Amazon Redshift Service Delivery Partner, AWS Control Tower Service Delivery Partner, AWS WAF Service Delivery Partner and many more.
To get started, go through our Consultancy page and Managed Services Package, CloudThat’s offerings.
FAQs
1. What are the benefits of using a warm pool in an Auto Scaling Group?
ANS: – Warm pools allow you to maintain a set of pre-initialized Amazon EC2 instances that can be quickly brought into service, reducing the time it takes to scale your application. By keeping instances in a stopped state, you save on costs, as you’re only billed for storage rather than compute. This approach is particularly beneficial for applications that experience sudden spikes in traffic and need to scale rapidly.
2. How does AWS CodeDeploy interact with instances in a warm pool?
ANS: – AWS CodeDeploy typically deploys code to instances as they enter an Auto Scaling Group. In warm pools, instances are brought from a stopped state to a running state before deployment. The AWS Lambda function handles this transition, ensuring that the instance is ready for deployment and returning it to a stopped state afterward. This integration allows you to maintain consistency in your deployments without manual intervention.
WRITTEN BY Deepak S
Deepak S works as a Research Intern at CloudThat. His expertise lies in AWS's services. Deepak is good at haunting new technologies and automobile enthusiasts.
Click to Comment