Orchestrating Serverless Workflows with Apache Airflow and AWS Lambda

Overview

In today’s cloud-native world, serverless computing has become popular due to its scalability, reduced operational overhead, and pay-per-use pricing model. AWS Lambda sits at the heart of this paradigm by enabling event-driven function execution without provisioning servers.

However, real-world serverless applications often involve complex workflows, such as chaining multiple AWS Lambda functions, scheduling tasks, handling retries, or integrating various AWS services. This is where Apache Airflow shines as a powerful workflow orchestrator.

In this blog, we will explore how to leverage Apache Airflow to manage serverless workflows using AWS Lambda, with hands-on guidance and best practices.

Pioneers in Cloud Consulting & Migration Services

Reduced infrastructural costs
Accelerated application deployment

Get Started

Apache Airflow for Serverless Workflow Management

Apache Airflow is an open-source platform to programmatically author, schedule, and monitor workflows. It supports:

Directed Acyclic Graphs (DAGs) for defining workflows
Task-level retries, logging, and alerting
Extensibility via operators and hooks
Scalability to run workflows in cloud environments

Airflow helps overcome the limitations of AWS Lambda orchestration tools (e.g., Step Functions), providing more flexibility, Python-native scripting, and better visibility into workflows.

Architecture Overview

Here’s how Apache Airflow integrates with AWS Lambda:

airflow

Airflow runs on an Amazon EC2 instance or managed service (like MWAA), and uses the AWSLambdaOperator to invoke Lambda functions. This setup is ideal for workflows that require coordination between multiple AWS Lambda functions, conditional logic, and scheduled execution.

airflow2

Use Case Example: Data Processing Pipeline

Let’s say you have a serverless data processing workflow with the following steps:

Trigger AWS Lambda function to ingest data from an API
Run a second AWS Lambda to clean and transform the data
Invoke a third AWS Lambda to store data into Amazon S3 or DynamoDB
Send a completion notification via Amazon SNS

We will orchestrate this using Airflow.

Step-by-Step Implementation

Step 1: Setup Airflow

Install Apache Airflow with AWS provider:

pip install apache-airflow apache-airflow-providers-amazon

1	pip install apache-airflow apache-airflow-providers-amazon

Set up your Airflow environment and connection to AWS:

# In Airflow UI > Admin > Connections
Conn ID: aws_default
Conn Type: Amazon Web Services
Login: Your AWS Access Key ID
Password: Your AWS Secret Access Key

# In Airflow UI > Admin > Connections

Conn ID: aws_default

Conn Type: Amazon Web Services

Password: Your AWS Secret Access Key

Or use AWS IAM roles if running in Amazon MWAA or Amazon EC2.

Step 2: Define the DAG

Here’s a sample DAG for the serverless pipeline:

from airflow import DAG from airflow.providers.amazon.aws.operators.lambda_function import AwsLambdaInvokeFunctionOperator from airflow.utils.dates import days_ago
default_args = { 'owner': 'airflow', 'retries': 1 }
with DAG( dag_id='serverless_data_pipeline', default_args=default_args, schedule_interval='@daily', start_date=days_ago(1), catchup=False, tags=['serverless', 'lambda'], ) as dag:
ingest_data = AwsLambdaInvokeFunctionOperator(
    task_id='invoke_lambda_ingest',
    function_name='lambda_ingest_data',
    aws_conn_id='aws_default',
    payload={"source": "api"}
)

transform_data = AwsLambdaInvokeFunctionOperator(
    task_id='invoke_lambda_transform',
    function_name='lambda_transform_data',
    aws_conn_id='aws_default',
    payload={"operation": "clean_and_map"}
)

store_data = AwsLambdaInvokeFunctionOperator(
    task_id='invoke_lambda_store',
    function_name='lambda_store_data',
    aws_conn_id='aws_default',
    payload={"target": "s3"}
)

notify = AwsLambdaInvokeFunctionOperator(
    task_id='invoke_lambda_notify',
    function_name='lambda_send_notification',
    aws_conn_id='aws_default',
    payload={"status": "complete"}
)

# Defining task dependencies
ingest_data >> transform_data >> store_data >> notify

from airflow import DAG from airflow.providers.amazon.aws.operators.lambda_function import AwsLambdaInvokeFunctionOperator from airflow.utils.dates import days_ago

default_args = { 'owner': 'airflow', 'retries': 1 }

with DAG( dag_id='serverless_data_pipeline', default_args=default_args, schedule_interval='@daily', start_date=days_ago(1), catchup=False, tags=['serverless', 'lambda'], ) as dag:

ingest_data = AwsLambdaInvokeFunctionOperator(

task_id='invoke_lambda_ingest',

function_name='lambda_ingest_data',

aws_conn_id='aws_default',

payload={"source": "api"}

)

transform_data = AwsLambdaInvokeFunctionOperator(

task_id='invoke_lambda_transform',

function_name='lambda_transform_data',

aws_conn_id='aws_default',

payload={"operation": "clean_and_map"}

)

store_data = AwsLambdaInvokeFunctionOperator(

task_id='invoke_lambda_store',

function_name='lambda_store_data',

aws_conn_id='aws_default',

payload={"target": "s3"}

)

notify = AwsLambdaInvokeFunctionOperator(

task_id='invoke_lambda_notify',

function_name='lambda_send_notification',

aws_conn_id='aws_default',

payload={"status": "complete"}

)

# Defining task dependencies

ingest_data >> transform_data >> store_data >> notify

Step 3: Create the AWS Lambda Functions

Here’s an example of a basic AWS Lambda function that returns a response:

# lambda_ingest_data.py
import json

def lambda_handler(event, context):
    print("Ingesting data from source:", event['source'])
    # Simulate data ingestion logic
    return {
        'statusCode': 200,
        'body': json.dumps('Data ingested successfully')
    }

# lambda_ingest_data.py

import json

def lambda_handler(event, context):

print("Ingesting data from source:", event['source'])

# Simulate data ingestion logic

return {

'statusCode': 200,

'body': json.dumps('Data ingested successfully')

}

Create the remaining functions for transformation, storage, and notification similarly.

Step 4: Deploy and Test

Deploy your Lambda functions via the AWS Console or CLI.
Upload your DAG to the Airflow DAGs folder.
Trigger the DAG manually or wait for the scheduled time.
Monitor task logs and results in the Airflow UI.

Benefits of Using Airflow with AWS Lambda

airflow3

Best Practices

Idempotency: Ensure AWS Lambda functions can run multiple times without side effects.
Timeout Handling: Match the Airflow task timeout with AWS Lambda’s maximum execution time.
Use Tags & Metadata: Track DAG runs and function invocations using tagging.
Secure Credentials: Use AWS IAM roles and AWS Secrets Manager for authentication.

airflow4

Conclusion

Apache Airflow and AWS Lambda offer a powerful combination for managing serverless workflows. With Airflow’s orchestration capabilities and AWS Lambda’s serverless power, you can easily build flexible, scalable, and cost-efficient pipelines.

Whether data processing, notifications, or custom automation, Airflow makes managing serverless logic structured, observable, and production-ready.

Drop a query if you have any questions regarding Apache Airflow and we will get back to you quickly.

Empowering organizations to become ‘data driven’ enterprises with our Cloud experts.

Reduced infrastructure costs
Timely data-driven decisions

Get Started

About CloudThat

CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.

FAQs

1. What is the role of Apache Airflow in a serverless architecture?

ANS: – Apache Airflow orchestrates workflows by managing task dependencies, scheduling executions, and triggering AWS Lambda functions in coordination.

2. How does Airflow trigger AWS Lambda functions?

ANS: – Airflow uses the AWSLambdaOperator to invoke AWS Lambda functions directly within a DAG, enabling event-driven or scheduled executions.