Mirroring Azure Databricks Unity Catalog in Microsoft Fabric

Introduction

Organizations leveraging Azure Databricks for data engineering and analytics often use the Unity Catalog for centralized governance and access control. However, integrating Unity Catalog with Microsoft Fabric can further enhance data accessibility, governance, and analytics capabilities. Mirroring Unity Catalog in Microsoft Fabric enables a seamless connection between Databricks-managed data assets and Fabric’s analytics ecosystem.

This blog will guide you through the benefits, use cases, and step-by-step instructions on mirroring Azure Databricks Unity Catalog in Microsoft Fabric.

What is Azure Databricks Unity Catalog?

Unity Catalog is a centralized governance solution for Databricks that provides:

• Fine-grained access control
• Data lineage tracking
• Schema enforcement and auditing
• Cross-cloud and multi-workspace governance

Mirrored Databases in Microsoft Fabric

Mirrored databases in Microsoft Fabric provide a seamless, end-to-end integration that simplifies analytics while promoting openness and collaboration between Fabric and Azure Databricks. This feature ensures an intuitive experience, making it easier to manage and analyze data across platforms without complex configurations.

Benefits of Mirroring Unity Catalog in Microsoft Fabric

• Unified Data Governance: Maintain a single source of truth with Unity Catalog while extending its reach into Fabric’s ecosystem.
• Enhanced Analytics: Leverage Fabric’s Power BI, AI, and Data Science capabilities for insights on Unity Catalog data.
• Real-Time Synchronization: Automatically sync Unity Catalog updates with Fabric without manual intervention.
• Reduced Data Duplication: Avoid unnecessary data replication and associated storage costs.

Use Cases

1. Enterprise Data Governance – Extend Unity Catalog’s governance framework to Fabric for cross-platform compliance.
2. Advanced Analytics & AI – Use Fabric’s Synapse, AI, and Power BI to analyze Unity Catalog data.
3. Cross-Platform Data Access – Enable seamless querying across Fabric and Databricks environments.

Mirrored catalogs are an item in Fabric Data Warehousing distinct from the Warehouse and SQL analytics endpoint.

When you mirror an Azure Databricks Unity Catalog, Fabric creates three items:

• Mirrored Azure Databricks item
• A SQL analytics endpoint on a Lakehouse
• A default semantic model

You can interact with mirrored data in the following ways:

• Each mirrored Azure Databricks item is automatically provided with an SQL analytics endpoint, enabling seamless data analysis through the mirroring process.
• Execute T-SQL queries to explore and retrieve data from the read-only SQL analytics endpoint.

Steps to Mirror Azure Databricks Unity Catalog in Microsoft Fabric

Prerequisites

1. Ensure Unity Catalog is Activated in your Azure Databricks workspace.
2. Obtain the EXTERNAL USE SCHEMA Privilege on relevant Unity Catalog schemas.
   • See Control external access to data in Unity Catalog for more details.
3. Network Configuration:
   • Ensure Azure Databricks workspaces are not behind a private endpoint.
   • Verify that storage accounts containing Unity Catalog data are not behind a firewall.

Create a Mirrored Database from Azure Databricks

1. Access Microsoft Fabric

• Navigate to Microsoft Fabric Portal.

2. Initiate the Mirroring Process

• Click on + New, then select Mirrored Azure Databricks catalog.

3. Set Up Azure Databricks Connection

• Choose an existing connection or create a new one.
• Authenticate using either your organizational account or a service principal.
• Ensure you have user or admin privileges in the Azure Databricks workspace.

4. Select Data to Mirror

• After connecting, select the desired catalog, schemas, and tables to mirror.
• By default, automatic synchronization is enabled for future catalog changes.

5. Finalize and Create

• Review your selections, finalize them, and click Create.

Conclusion

Mirroring Azure Databricks Unity Catalog in Microsoft Fabric enables organizations to leverage the best of Databricks’ governance and Fabric’s analytics capabilities. By following the outlined steps, you can establish a seamless integration between the two platforms, unlocking new possibilities for data-driven decision-making.

About CloudThat