DP-203: Data Engineering on Microsoft Azure

• Transform data by using Apache Spark
• Transform data by using Transact-SQL
• Transform data by using Data Factory
• Transform data by using Azure Synapse Pipelines
• Transform data by using Stream Analytics
• Cleanse data
• Split data
• Shred JSON
• Encode and decode data
• Configure error handling for the transformation
• Normalize and denormalize values
• Transform data by using Scala
• Perform data exploratory analysis

Design and develop a batch processing solution

• Develop batch processing solutions by using Data Factory, Data Lake, Spark, Azure Synapse Pipelines, PolyBase, and Azure Databricks
• Create data pipelines
• Design and implement incremental data loads
• Design and develop slowly changing dimensions
• Handle security and compliance requirements
• Scale resources
• Configure the batch size
• Design and create tests for data pipelines
• Integrate Jupyter/IPython notebooks into a data pipeline
• Handle duplicate data
• Handle missing data
• Handle late-arriving data
• Upsert data
• Regress to a previous state
• Design and configure exception handling
• Configure batch retention
• Design a batch processing solution
• Debug Spark jobs by using the Spark UI

Design and develop a stream processing solution

• Develop a stream processing solution by using Stream Analytics, Azure Databricks, and Azure Event Hubs
• Process data by using Spark structured streaming
• Monitor for performance and functional regressions
• Design and create windowed aggregates
• Handle schema drift
• Process time series data
• Process across partitions
• Process within one partition
• Configure checkpoints/watermarking during processing
• Scale resources
• Design and create tests for data pipelines
• Optimize pipelines for analytical or transactional purposes
• Handle interruptions
• Design and configure exception handling
• Upsert data
• Replay archived stream data
• Design a stream processing solution

Manage batches and pipelines

• Trigger batches
• Handle failed batch loads
• Validate batch loads
• Manage data pipelines in Data Factory/Synapse Pipelines
• Schedule data pipelines in Data Factory/Synapse Pipelines
• Implement version control for pipeline artifacts
• Manage Spark jobs in a pipeline

• Design data encryption for data at rest and in transit
• Design a data auditing strategy
• Design a data masking strategy
• Design for data privacy
• Design a data retention policy
• Design to purge data based on business requirements
• Design Azure role-based access control (Azure RBAC) and POSIX-like Access Control List (ACL) for Data Lake Storage Gen2
• Design row-level and column-level security

Implement data security

• Implement data masking
• Encrypt data at rest and in motion
• Implement row-level and column-level security
• Implement Azure RBAC
• Implement POSIX-like ACLs for Data Lake Storage Gen2
• Implement a data retention policy
• Implement a data auditing strategy
• Manage identities, keys, and secrets across different data platform technologies
• Implement secure endpoints (private and public)
• Implement resource tokens in Azure Databricks
• Load a DataFrame with sensitive information
• Write encrypted data to tables or Parquet files
• Manage sensitive information

• Implement logging used by Azure Monitor
• Configure monitoring services
• Measure performance of data movement
• Monitor and update statistics about data across a system
• Monitor data pipeline performance
• Measure query performance
• Monitor cluster performance
• Understand custom logging options
• Schedule and monitor pipeline tests
• Interpret Azure Monitor metrics and logs
• Interpret a Spark directed acyclic graph (DAG)

Optimize and troubleshoot data storage and data processing

• Compact small files
• Rewrite user-defined functions (UDFs)
• Handle skew in data
• Handle data spill
• Tune shuffle partitions
• Find shuffling in a pipeline
• Optimize resource management
• Tune queries by using indexers
• Tune queries by using cache
• Optimize pipelines for analytical or transactional purposes
• Optimize pipeline for descriptive versus analytical workloads
• Troubleshoot a failed spark job
• Troubleshoot a failed pipeline run