AWS, Cloud Computing, Data Analytics

3 Mins Read

Seamless Data Lake Management with Apache Iceberg AWS Glue and Amazon Athena

Voiced by Amazon Polly

Overview

Apache Iceberg has revolutionized how we handle big data tables, enabling efficient table management with features like partition evolution, time travel, and atomic operations. Combining Iceberg with AWS Glue Catalog and Amazon Athena simplifies data lake workflows, making it accessible to modern cloud environments.

In this blog, we will explore:

  1. Registering Iceberg Tables in AWS Glue Catalog using Amazon Athena and AWS Glue
  • For partitioned and unpartitioned tables.

2. Performing UPSERT operations with AWS Glue and Amazon Athena.

3. Enabling and leveraging Time Travel in Iceberg tables.

AD

Pioneers in Cloud Consulting & Migration Services

  • Reduced infrastructural costs
  • Accelerated application deployment
Get Started

Registering Iceberg Tables in the AWS Glue Catalog

Iceberg tables can be partitioned or unpartitioned and registering them in the AWS Glue Data Catalog allows Amazon Athena and AWS Glue ETL jobs to query and manipulate these tables.

a. Registering Unpartitioned Iceberg Tables

To register an unpartitioned Iceberg table in the AWS Glue Catalog, follow these steps:

Step 1: Create the Table in Amazon Athena

Iceberg tables can be created using Amazon Athena’s SQL interface:

This command:

  • Creates an unpartitioned Iceberg table in the AWS Glue Catalog.
  • Sets the default storage type to Iceberg.

Step 2: Verify Registration

Confirm that the table appears under the appropriate database in the AWS Glue console.

Step 3: Query the Table with Amazon Athena

Test the table with simple queries in Athena:

b. Registering Partitioned Iceberg Tables

Partitioned tables allow efficient queries by reducing data scanning. To register a partitioned Iceberg table:

Step 1: Create the Partitioned Table

  • The PARTITIONED BY clause defines the partition key (age in this example).

Step 2: Load Data

Data can be inserted using Amazon Athena’s SQL:

Step 3: Verify Partitions

Use Amazon Athena to list partitions:

c. Registering Iceberg Tables via AWS Glue ETL Jobs

For AWS Glue ETL jobs to manage Iceberg tables:

  1. Use AWS Glue version 3.0 or later.
  2. Add the Iceberg connector jar (aws-glue-iceberg.jar) to the job if required.

Example PySpark Script

Performing UPSERT Operations

Iceberg tables support merge-on-read operations for upserts, combining INSERT and DELETE into one atomic operation.

Using Glue ETL for UPSERTs

Step 1: Load the Delta Data

Delta data (new or updated records) can be loaded into a Spark DataFrame.

Step 2: Perform the Merge

Iceberg uses the MERGE INTO SQL command for upserts.

  • WHEN MATCHED: Updates existing records.
  • WHEN NOT MATCHED: Inserts new records.

Using Amazon Athena for UPSERTs

  1. Create the Iceberg Table (if not already created):

2. Load the New Data into a Staging Table (Optional): If your new data comes from Amazon S3, create an external table:

3. Execute MERGE INTO to Perform UPSERT:

Using Time Travel

Time travel is a powerful Iceberg feature that allows historical snapshots of the table to be accessed.

a. Querying Historical Snapshots

  1. Version Based queries

Find the snapshot ID using the Iceberg metadata table:

Query a snapshot using its ID:

2. Time travel queries

b. Using AWS Glue ETL for Time Travel

In AWS Glue ETL jobs, time travel is configured using Iceberg properties.

Example PySpark script:

Conclusion

By leveraging Apache Iceberg with AWS Glue and Amazon Athena, you can efficiently manage your data lake, enabling advanced capabilities like partition evolution, atomic upserts, and time travel. AWS Glue’s ETL jobs provide seamless integration for processing and managing Iceberg tables, while Amazon Athena’s SQL interface simplifies querying.

Iceberg’s rich feature set and AWS’s powerful ecosystem empower modern data workflows to achieve scalability, consistency, and query performance.

Drop a query if you have any questions regarding Apache Iceberg, AWS Glue or Amazon Athena and we will get back to you quickly.

Making IT Networks Enterprise-ready – Cloud Management Services

  • Accelerated cloud migration
  • End-to-end view of the cloud environment
Get Started

About CloudThat

CloudThat is an award-winning company and the first in India to offer cloud training and consulting services worldwide. As a Microsoft Solutions Partner, AWS Advanced Tier Training Partner, and Google Cloud Platform Partner, CloudThat has empowered over 850,000 professionals through 600+ cloud certifications winning global recognition for its training excellence including 20 MCT Trainers in Microsoft’s Global Top 100 and an impressive 12 awards in the last 8 years. CloudThat specializes in Cloud Migration, Data Platforms, DevOps, IoT, and cutting-edge technologies like Gen AI & AI/ML. It has delivered over 500 consulting projects for 250+ organizations in 30+ countries as it continues to empower professionals and enterprises to thrive in the digital-first world.

FAQs

1. How do I register an Iceberg table in the AWS Glue Catalog?

ANS: – Use Amazon Athena’s CREATE TABLE command to register the table, then verify in the AWS Glue console.

2. Can I perform UPSERT operations on Iceberg tables?

ANS: – Yes, you can use the MERGE INTO command in Spark for UPSERTs or emulate it in Amazon Athena with temporary tables.

WRITTEN BY Rishi Raj Saikia

Rishi Raj Saikia is working as Sr. Research Associate - Data & AI IoT team at CloudThat.  He is a seasoned Electronics & Instrumentation engineer with a history of working in Telecom and the petroleum industry. He also possesses a deep knowledge of electronics, control theory/controller designing, and embedded systems, with PCB designing skills for relevant domains. He is keen on learning new advancements in IoT devices, IIoT technologies, and cloud-based technologies.

Share

Comments

    Click to Comment

Get The Most Out Of Us

Our support doesn't end here. We have monthly newsletters, study guides, practice questions, and more to assist you in upgrading your cloud career. Subscribe to get them all!