Voiced by Amazon Polly |
In today’s data-driven world, organizations are seeking ways to harness the full potential of their data by blending powerful analytics with cutting-edge AI. Microsoft Fabric, when integrated with OpenAI, creates a compelling ecosystem that accelerates insight discovery, automates intelligence, and democratizes AI across business units. Let’s explore how this synergy works and why it matters.
Enhance Your Productivity with Microsoft Copilot
- Effortless Integration
- AI-Powered Assistance
What is Microsoft Fabric?
Microsoft Fabric is a unified data platform that brings together data engineering, data science, real-time analytics, and business intelligence in a single SaaS offering. It’s built on top of OneLake, Microsoft’s cloud-scale data lake, and combines the best of Power BI, Azure Synapse, and Data Factory under one roof. Whether you’re an analyst using low-code interfaces or a data engineer writing PySpark code, Fabric offers integrated experiences that streamline data operations and accelerate time to insights. It’s secure, scalable, and deeply connected with the Microsoft ecosystem, making it a go-to platform for enterprises looking to modernize their data stack.
Integrating Microsoft Fabric and Azure OpenAI
By leveraging Azure OpenAI, users can enhance their data workflows with powerful AI capabilities. This integration allows for automated tasks, advanced text generation, and more, making data management and analysis more efficient and insightful.
Microsoft Fabric offers several features that can be enhanced with AI:
- Data Cleaning: Automate the process of cleaning and preparing data for analysis.
- Text Generation: Create reports, summaries, and content based on data insights.
- Data Transformation: Use AI to transform and enrich data for better analysis.
- Custom AI Applications: Build applications that leverage AI to provide insights, answer questions, and identify key information.
To integrate OpenAI with Microsoft Fabric, you can use the following libraries:
- Azure OpenAI SDK: This library allows you to interact with OpenAI models hosted on Azure.
- Microsoft Fabric SDK: This library provides tools to manage and interact with data within Microsoft Fabric.
How to use Azure AI services:
Pre-Built AI Models in Microsoft Fabric (Preview)
Microsoft Fabric offers pre-built AI models currently in preview mode, enabling seamless integration with Fabric authentication and capacity-based billing. These models include capabilities from Azure OpenAI, Text Analytics, and Translator services.
Bring Your Own Key (BYOK) Support
If the required functionality isn’t covered by the pre-built models, you can provision a dedicated AI resource in Azure and integrate it within Fabric. This approach allows for greater customization and control, including BYOK support, where applicable.
Microsoft Fabric integrates Azure OpenAI through the Data Science and Data Engineering experiences, with support for:
- PySpark & notebooks using openai, azure-ai-ml, and langchain libraries.
- Power BI and Power Query can connect with OpenAI models via Azure ML endpoints.
- KQL (Kusto Query Language) in Real-Time Analytics supports calling REST APIs (e.g., OpenAI) using external call functions.
- Semantic Link in Fabric simplifies the connection between structured business data (like Power BI datasets) and natural language via LLMs.
Use case using Azure Open AI for Chat completion
The below sample code provides a detailed guide on using the Azure OpenAI service with SynapseML to scale natural language tasks using Apache Spark in Microsoft Fabric
STEP1: Microsoft Fabric subscription (its preinstalled with synapse ML packages for Azure Open AI)
Code to check the packages installed
1 2 3 4 5 6 7 8 9 |
<em>import synapse.ml.services</em> <em>print(f"SynapseML cognitive version: {synapse.ml.services.__version__}")</em> <em> </em> <em>OUTPUT:</em> <em>SynapseML cognitive version: 1.0.11.dev1</em> |
STEP2: Create a Azure Open AI service in your subscription and capture the “service name”, “deployment names” (ex:chat completion , embeddings etc). Make sure the deployment can change time to time on the updates happening.
STEP3: Create a Notebook on the fabric experience and make connections to the Azure OpenAI service created in above step
Code to connect with Azure Open AI. Key is directly hard coded and recommended to use KeyVaults for production environments.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 |
<em>import os</em> <em>from pyspark.sql import SparkSession</em> <em>from synapse.ml.core.platform import running_on_synapse, find_secret</em> <em> </em> <em># Bootstrap Spark Session</em> <em>spark = SparkSession.builder.getOrCreate()</em> <em> </em> <em>if running_on_synapse():</em> <em> from notebookutils.visualization import display</em> <em> </em> <em># Fill in the following lines with your service information</em> <em># Learn more about selecting which embedding model to choose: https://openai.com/blog/new-and-improved-embedding-model</em> <em>service_name = "</em><em>openaiXXXXX</em><em>"</em> <em>deployment_name = </em><em>"gpt-35-turbo-instruct</em><em>"</em> <em>deployment_name_embeddings = "</em><em>text-embedding-ada-002</em><em>"</em> <em> </em> <em>key = "</em><em>AGU0zJxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxv</em><em>"</em> <em> </em> |
STEP4: Creating Prompts: We will use the integrated environment to get the datasets to be used for chat completion . But for demonstration we are creating a dataset of prompts and use the OpenAICompletion object to process these prompts.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
<em>df = spark.createDataFrame(</em> <em> [</em> <em> ("Hello my name is",),</em> <em> ("The best code is code thats",),</em> <em> ("SynapseML is ",),</em> <em> ]</em> <em>).toDF("prompt")</em> <em> </em> |
STEP5: Create an OpenAICompletion object with max tokens set to 200 and creating 3 columns with prompt, error and completion.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 |
<em>from synapse.ml.cognitive import OpenAICompletion</em> <em> </em> <em>completion = (</em> <em> OpenAICompletion()</em> <em> .setSubscriptionKey(key)</em> <em> .setDeploymentName(deployment_name)</em> <em> .setCustomServiceName(service_name)</em> <em> .setMaxTokens(200)</em> <em> .setPromptCol("prompt")</em> <em> .setErrorCol("error")</em> <em> .setOutputCol("completions")</em> <em>)</em> <em> </em> |
STEP6: Transform the dataframe with the OpenAICompletion Client
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 |
<em>from pyspark.sql.functions import col</em> <em> </em> <em>completed_df = completion.transform(df).cache()</em> <em>display(</em> <em> completed_df.select(</em> <em> col("prompt"),</em> <em> col("error"),</em> <em> col("completions.choices.text").getItem(0).alias("text"),</em> <em> )</em> <em>)</em> <em> </em> |
OUTPUT SAMPLE:
Similarly you can apply the Azure OpenAI for embeddings, prompt engineering for Translation or Question Answering .
Conclusion
Whether it’s summarizing massive datasets, generating real-time narratives in Power BI, or building intelligent pipelines that understand context, Azure OpenAI in Fabric empowers data teams to do more with less code and faster deployment. As Microsoft continues to expand native support and tooling, the synergy between generative AI and modern data platforms is poised to redefine how businesses derive value from data.
Become an Azure Expert in Just 2 Months with Industry-Certified Trainers
- Career-Boosting Skills
- Hands-on Labs
- Flexible Learning
About CloudThat
CloudThat is a leading provider of Cloud Training and Consulting services with a global presence in India, the USA, Asia, Europe, and Africa. Specializing in AWS, Microsoft Azure, GCP, VMware, Databricks, and more, the company serves mid-market and enterprise clients, offering comprehensive expertise in Cloud Migration, Data Platforms, DevOps, IoT, AI/ML, and more.
CloudThat is the first Indian Company to win the prestigious Microsoft Partner 2024 Award and is recognized as a top-tier partner with AWS and Microsoft, including the prestigious ‘Think Big’ partner award from AWS and the Microsoft Superstars FY 2023 award in Asia & India. Having trained 850k+ professionals in 600+ cloud certifications and completed 500+ consulting projects globally, CloudThat is an official AWS Advanced Consulting Partner, Microsoft Gold Partner, AWS Training Partner, AWS Migration Partner, AWS Data and Analytics Partner, AWS DevOps Competency Partner, AWS GenAI Competency Partner, Amazon QuickSight Service Delivery Partner, Amazon EKS Service Delivery Partner, AWS Microsoft Workload Partners, Amazon EC2 Service Delivery Partner, Amazon ECS Service Delivery Partner, AWS Glue Service Delivery Partner, Amazon Redshift Service Delivery Partner, AWS Control Tower Service Delivery Partner, AWS WAF Service Delivery Partner, Amazon CloudFront Service Delivery Partner, Amazon OpenSearch Service Delivery Partner, AWS DMS Service Delivery Partner, AWS Systems Manager Service Delivery Partner, Amazon RDS Service Delivery Partner, AWS CloudFormation Service Delivery Partner, AWS Config, Amazon EMR and many more.
WRITTEN BY G R Deeba Lakshmi
Comments