Handling Multi-Account and Multi-Region Workloads with AWS Lambda

Overview

AWS Lambda has become a key service for building serverless applications because it allows developers to run code without provisioning or managing servers. It automatically handles much of the underlying infrastructure and scales based on incoming workloads.

For small applications, Lambda functions are often straightforward: receive an event, perform a task, and return the result. However, the architecture becomes more challenging when the same function needs to operate across multiple AWS accounts and multiple AWS Regions.

A task that completes quickly for one account may require hundreds or thousands of AWS API calls when executed across an entire organization. Without the right architecture, this can result in long execution times, throttling, Lambda timeouts, and unnecessary costs.

Why Multi-Account and Multi-Region Processing Becomes Challenging?

Managing resources in a single AWS account is relatively straightforward. However, the complexity increases significantly when an application needs to collect information across dozens of AWS accounts and multiple Regions.

Consider a cloud management application responsible for maintaining a centralized inventory of resources across 40 AWS accounts. The application may need to discover compute instances, databases, storage resources, networking components, containers, security services, and monitoring resources from every account.

The challenge is that AWS resources are distributed differently. Some services, such as AWS IAM, are global, while services such as Amazon EC2, Amazon RDS, Amazon VPC, and Amazon EKS are Region-specific. This means the application cannot simply make one request per account. It must query multiple service APIs across multiple Regions to build a complete inventory.

For example, scanning 40 accounts across 15 Regions already creates 600 account-Region combinations. When several AWS services are queried within each combination, the number of API requests can quickly grow into the thousands.

A simple implementation may process these operations sequentially:

Account 1 → Regions → Services → Account 2 → Regions → Services → Account 3 → Regions → Services → …

In this approach, the next account starts only after processing for the previous account has completed. As the number of accounts, Regions, and services increases, the total execution time increases accordingly.

Sequential processing can introduce several problems:

• Longer execution time: Every additional account and Region adds more work to the same Lambda invocation.
• Lambda timeout risk: Large inventory operations may approach Lambda’s maximum execution duration.
• API throttling: A high volume of AWS API requests can exceed service-specific request limits.

Understanding AWS Lambda Limits

Lambda automatically scales by creating additional execution environments as requests increase. However, serverless does not mean unlimited.

Lambda functions have execution and concurrency quotas. A single invocation can run for up to 15 minutes, and functions can also be throttled when concurrency or downstream AWS service API limits are reached.

Therefore, simply increasing the Lambda timeout is not a scalable solution. A better approach is to divide a large workload into smaller independent tasks.

One effective solution for multi-account workloads is a fan-out architecture.

Instead of a single Lambda function scanning every account sequentially, a parent Lambda can identify which accounts need processing and invoke separate workers.

The flow can look like this:

Parent Lambda → Discover Accounts → Invoke Workers → Process Accounts → Store Results → Aggregate Results

For example, if an organization has 40 accounts, the parent function can create independent processing tasks for those accounts. Each worker assumes the required IAM role in its assigned account and scans the necessary Regions and services.

This provides several advantages:

• Accounts can be processed in parallel.
• Failure in one account does not necessarily stop the entire inventory process.
• Workloads can scale as the number of accounts increases.
• Individual executions remain smaller and easier to troubleshoot.
• Failed accounts can be retried independently.

AWS Lambda supports asynchronous invocation, making this pattern useful when the caller does not need to wait for every worker to complete before continuing.

Efficient Multi-Region Processing

Multi-account processing solves only part of the problem. Applications should also be careful about how they scan AWS Regions.

Not every service needs to be queried in every Region. For example, some AWS services are global, while services such as EC2 and RDS are regional.

A more efficient application should:

• Discover or configure the Regions that actually need scanning.
• Avoid calling regional APIs for global services unnecessarily.
• Reuse AWS SDK clients where appropriate.

Reducing unnecessary API calls improves both execution time and reliability.

Cross-Account Access with AWS STS

A multi-account Lambda also requires a secure mechanism to access resources in other accounts.

A common approach is to create an IAM role in each target account and allow the central Lambda function to assume that role using AWS Security Token Service (AWS STS).

The worker receives temporary credentials and uses them to query resources in the target account.

This avoids storing long-term credentials for every AWS account and provides centralized control over permissions. The assumed role should follow the principle of least privilege and include only the permissions required by the workload.

Store Results Instead of Returning Everything

Another common mistake is trying to combine all collected data into a single Lambda invocation.

For large inventories, workers can instead store their results in services such as Amazon S3 or DynamoDB.

For example:

Account Worker → Scan Resources → Store Account Result in S3

Once all workers have completed, another process can aggregate the account-level results into a consolidated inventory.

This separates data collection from data aggregation, making the architecture easier to scale and maintain.

Conclusion

AWS Lambda can effectively support multi-account and multi-region workloads, but scalability depends heavily on architecture.

A single function that loops through every account, Region, and service may work during initial development, but can become a bottleneck as the environment grows. Breaking the workload into independent workers, processing tasks asynchronously, minimizing unnecessary API calls, and storing results separately creates a more scalable design.

The key principle is simple: instead of making one Lambda function do more work, divide the work so multiple functions can do less.

With the right architecture, AWS Lambda can provide a flexible and serverless foundation for large-scale cloud inventory, governance, monitoring, and automation solutions.

Drop a query if you have any questions regarding AWS Lambda, and we will get back to you quickly.