Cloud Computing, Google Cloud (GCP)

4 Mins Read

A Guide to Airflow with GCP for Streamlined Data Orchestration – Part 2

Voiced by Amazon Polly


In the first part of this blog, we delved into the fundamental concepts of Airflow and its efficacy within the GCP cloud environment for streamlined data orchestration. As we venture into the second part, the focus shifts towards practical implementation, sharpening the intricacy between Airflow and GCP Composer. This installment aims to provide readers with a hands-on understanding of Airflow’s functionality within the GCP ecosystem, showcasing its practical applications and the seamless orchestration it brings to data workflows. Building on the groundwork laid in the first part, we’re set to unravel the implementation intricacies, ensuring that readers understand how Airflow becomes a powerful ally in data management on GCP.


Data is an asset, and processing it efficiently is crucial for extracting meaningful insights. In this blog post, we’ll guide you through creating a data pipeline using Apache Airflow to take Google Cloud Storage (GCS) files, transform them, and store the results in BigQuery.

By the end of this guide, you’ll have a robust and automated solution for managing your data workflows in the cloud.

Pioneers in Cloud Consulting & Migration Services

  • Reduced infrastructural costs
  • Accelerated application deployment
Get Started


Before you begin, ensure you have the following set up:

  • A GCP account with billing enabled.
  • A Google Cloud Storage bucket.
  • A BigQuery dataset.
  • Apache Airflow installed, either locally or on GCP Composer.

Step-by-Step Guide

Step 1: Set Up GCP Resources

  • Create a GCS Bucket:
  • In the GCP Console, navigate to “Storage” > “Browser.”
  • Create a new bucket and note the bucket name.


  • Set Up Google Cloud BigQuery
  • In the GCP Console, navigate to “BigQuery.”
  • Create a new dataset within your project.


Step 2: Write Python Script for Transformation

Create a Python script:

  • Write a Python script that performs the desired transformation on your data.
  • Save the script, e.g.,

Python code

Step 3: Set Up Airflow DAG:

Create a new DAG file:

  • Create a new Python file in your Airflow DAGs folder, e.g.,
  • Define the DAG structure and import the transform_data function.


Python code

Step 4: Upload DAG to Airflow

Upload the DAG file:

  • Copy the file to the DAGs folder in your Airflow instance.
  • Airflow will automatically detect the new DAG and add it to the DAG list.


Step 5: Schedule and Monitor

Set up a schedule:

  • In the file, adjust the schedule_interval parameter to your desired schedule.
  • Save the file.

Monitor in Airflow UI:

  • Access the Airflow web UI.
  • Navigate to the “DAGs” section and find the “data_pipeline_dag.”
  • Trigger the DAG manually to test the data transformation.

Step 6: Load Transformed Data into Google Cloud BigQuery

  • Write Python Script for Google Cloud BigQuery Load.
  • Write a Python script to load the transformed data into Google Cloud BigQuery.
  • Use the BigQuery client library to interact with Google Cloud BigQuery.

Python code

Step 7: Extend Airflow DAG for Google Cloud BigQuery Load:

Modify the DAG file:

  • Edit the file to include the BigQuery load task.

Python code


You’ve successfully built a data pipeline using Apache Airflow to take files from Google Cloud Storage, transform them, and store the results in Google Cloud BigQuery. This scalable and automated solution showcases the power of Airflow in orchestrating complex workflows. Feel free to customize the script and DAG to fit your requirements and explore additional Airflow features for more advanced data processing tasks. Automation like this enhances the efficiency of your data workflows, ultimately leading to more informed decision-making and insights from your data.

Drop a query if you have any questions regarding Apache Airflow and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

  • Accelerated cloud migration
  • End-to-end view of the cloud environment
Get Started

About CloudThat

CloudThat is an official AWS (Amazon Web Services) Advanced Consulting Partner and Training partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, Microsoft Gold Partner, and many more, helping people develop knowledge of the cloud and help their businesses aim for higher goals using best-in-industry cloud computing practices and expertise. We are on a mission to build a robust cloud computing ecosystem by disseminating knowledge on technological intricacies within the cloud space. Our blogs, webinars, case studies, and white papers enable all the stakeholders in the cloud computing sphere.

To get started, go through our Consultancy page and Managed Services PackageCloudThat’s offerings.


1. How can I version-control and manage the code for my Airflow DAGs when using GCP Composer?

ANS: – Version-controlling Airflow DAGs in GCP Composer is best achieved by storing your DAG definitions in a version-controlled repository like Git. You can then use GCP Composer’s environment versioning feature to update your environment with the latest version of your DAGs. This ensures that your DAGs are consistent across different environments and allows for easy rollback in case of issues. GCP Composer allows you to upload DAGs directly through the web UI or the gcloud command-line tool.

2. Can I trigger Google Cloud Functions or other GCP services from Apache Airflow DAGs?

ANS: – Absolutely! Apache Airflow has operators that enable interaction with various GCP services. For triggering Google Cloud Functions, you can use the GoogleCloudFunctionOperator. Similarly, operators for other GCP services like BigQuery, Dataflow, and more exist. You can seamlessly integrate and orchestrate workflows involving multiple GCP services within your Airflow DAGs by leveraging these operators.

WRITTEN BY Hariprasad Kulkarni



    Click to Comment

Get The Most Out Of Us

Our support doesn't end here. We have monthly newsletters, study guides, practice questions, and more to assist you in upgrading your cloud career. Subscribe to get them all!