Deploying Large AWS Lambda Applications Using Docker and Amazon ECR

Introduction

AWS Lambda makes it easy to run applications without managing servers. For lightweight applications, deploying a Lambda function as a ZIP package is simple and effective. However, problems can arise when the application depends on large Python libraries, machine learning frameworks, or other heavy dependencies.

We faced this challenge while working with a Lambda based workload. Our deployment package became too large for the traditional ZIP-based approach. We explored Lambda Layers, but they did not provide a practical solution for our dependency requirements.

The solution was to move to container-based Lambda deployment using Docker and Amazon ECR.

This article explains the problem, the approaches we considered, and why container images became the right solution.

Understanding the Lambda Package Size Problem

AWS Lambda supports traditional ZIP-based deployments, but they come with deployment package size constraints.

For a small Python function with a few dependencies, this usually isn’t an issue:

Lambda Function

• Application Code
• txt
• Small Dependencies

However, applications involving AI, machine learning, or data processing can have significantly larger dependencies.

Libraries such as:

• numpy
• pandas
• scipy
• torch
• transformers
• opencv

can quickly increase the deployment size.

As dependencies grew, our ZIP package eventually reached the limits of the deployment model. Simply adding more files to the package was no longer an option.

We needed a different approach.

Why Lambda Layers Were Not Enough

Our next option was AWS Lambda Layers.

Layers allow dependencies to be separated from the main Lambda function and shared across multiple functions.

The architecture looks like:

Lambda Function

• Application Code
• Lambda Layer
• Dependencies

This is useful for organizing and sharing dependencies. However, Layers still operate within Lambda’s deployment size constraints.

For our workload, the dependencies were still too large to manage comfortably using ZIP packages and Layers.

Instead of continuously trying to reduce or split the dependencies, we reconsidered the deployment strategy.

Moving to Container Images

The solution was to package the Lambda function as a Docker container image.

The new deployment flow became:

1. Application Code
2. Dockerfile
3. Docker Image
4. Amazon ECR
5. AWS Lambda

With container-based Lambda, the application, its dependencies, and the runtime environment can be packaged together into a Docker image.

This was particularly useful for our workload, as we had larger dependencies that were difficult to manage with traditional ZIP deployments.

Creating the Docker Image

A simplified Dockerfile for a Python Lambda looks like this:

• The AWS Lambda base image provides the Lambda-compatible runtime.
• During the Docker build, the required Python dependencies are installed into the image along with the application code.
• We can then build the image using:
• docker build -t my-lambda-function
• One advantage of this approach is that the image can be tested locally before being deployed to AWS.
• The development workflow becomes:
• Write Code
• Build Docker Image
• Test Locally
• Push to ECR
• Deploy to Lambda
• This provides a consistent and reproducible deployment environment.

Using Amazon Elastic Container Registry (ECR)

After building the Docker image, it must be stored in a container registry accessible to Lambda.

Amazon Elastic Container Registry (ECR) is used for this purpose.

The image can be tagged with the ECR repository:

• docker tag my-lambda-function:latest \
• <account-id>.dkr.ecr.<region>.amazonaws.com/my-lambda-function:latest

Then it can be pushed to ECR:

• docker push \
• <account-id>.dkr.ecr.<region>.amazonaws.com/my-lambda-function:latest

AWS Lambda can then use the image stored in ECR as the deployment package.

This removes the need to manage large ZIP files and their dependencies continuously.

ZIP vs Layers vs Container Images

Each deployment method has its own use case.

ZIP deployment remains the simplest option for most small Lambda functions. Layers are useful when dependencies need to be shared.

Container images become more attractive when dependency size, system libraries, or runtime requirements make traditional packaging difficult.

The goal isn’t to use containers everywhere. It is necessary to choose the deployment model that best fits the application.

Conclusion

AWS Lambda provides multiple ways to package and deploy applications. ZIP files are excellent for lightweight functions, while Lambda Layers help organize shared dependencies.

But when an application has extensive dependencies, particularly in AI, ML, or data-processing workloads, container images can offer a more practical solution.

Our migration from ZIP → Layers → Docker → Amazon ECR → Lambda demonstrated that sometimes the best way to solve a deployment limitation is not to optimize the existing approach, but to choose a better deployment strategy.

If your Lambda deployment package is becoming difficult to manage, container-based Lambda is worth considering.

Drop a query if you have any questions regarding AWS Lambda, and we will get back to you quickly.