Azure Data Factory

What is ETL
 What is Data Warehouse
 What are Existing Data Warehouses
What is Data Factory
 Building Block of Data Factory
Pipelines and Activities
 Linked Services o Data Set
 Integration Runtime
 Create Storage Account
Provision SQL Server and Create Database
 Provision Data Factory

Working with Activates

 Data Factory UI
 Copy Activity on Blob Storage to SQL Database
 Copy Activity on storage account to storage account
 Create Pipeline
 Lookup Activity

for Each Activity
 Filter Activity
 Get Metadata Activity Azure
 Monitoring Pipeline
 Debug Pipeline
 Trigger pipeline manually
 Monitor pipeline
 Trigger pipeline on schedule

Real Time Scenarios and Use Cases with ADF Activates

ADF Introduction
 Important Concepts in ADF
 Creating Azure Free Account for ADF
 Integration Runtime and Types
 Integration runtime in ADF-Azure IR
 Create Pipeline in ADF
 Azure Storage Account Integration with ADF
 Copy activity multiple files from blob to blob
 Filter activity Dynamic parameters Copy Activity
 Get File Names from Folder Dynamically
 Deep dive into Copy Activity in ADF
 Copy Activity Behaviour in ADF
 Copy Activity Performance Tuning in ADF
 Validation of schema in ADF
 Get Count of files from folder in ADF
 Validate copied data between source and sink in ADF
 Azure SQL Database integration with ADF
 Azure SQL Databases – Introduction Relational databases
 Azure Functions execution in ADF pipeline
 Sending Mail during the pipeline
 Creating Your First Azure SQL Database
1) Deployment Models
2) Purchasing Modes
 Overwrite and Append Modes in Copy Activity
 Full Load in ADF
 Copy Data from Azure SQL Database to BLOB in ADF
 Copy multiple tables in Bulk with Lookup & ForEach in Data Factory
 Logging and Notification Azure Logic Apps

Log Pipeline Executions to SQL Table using ADF
 Custom Email Notifications Send Error notification with logic app
 Use For each loop up activity to copy multiple Tables- Step by Step
Explanation
 Incremental Load in ADF
 Incremental Load or Delta load from SQL to Blob Storage in ADF
 Multi Table Incremental Load or Delta load from SQL to Blob Storage
 Incrementally copy new and changed files based on Last Modified Date
 Azure Key Vault integration with ADF
 Azure Key Vault, Secure secrets, keys & certificates in Azure Data
 ADF Triggers:
o Event Based Trigger in ADF
o Tumbling window trigger dependency & parameters
o Schedule Trigger
 Self Hosted Integration Runtime
 Copying On Premise data using Azure Self Hosted integration Runtime
 Data Migration from On premise SQL Server to cloud using ADF
 Load data from on premise sql server to Azure SQL DB
 Data Migration with polybase and Bulk insert
 Copy Data from sql server to Azure SQL DW with polybase & Bulk Insert
 Data Migration from On premise File System to cloud using ADF
 Copy Data from on-premise File System to ADLS Gen2
 ToCopying data from REST API using ADF
 Loop through REST API copy data TO ADLS Gen2-Linked Service
Parameters
 AWS S3 integration with ADF
 Migrate Data from AWS S3 Buckets to ADLS Gen2
 Activities in ADF
 Switch Activity-Move and delete data
 Until Activity-Parameters & Variables
 Copy Recent Files From Blob input to Blob Output folder without LPV
 Snowflake integration with ADF
 Copy data from Snowflake to ADLS Gen2
 Copy data from ADLS Gen2 to Snowflake
 Azure CosmosDB integration with ADF
 Copy data from Azure SQLDB to CosmosDB
 Copy data from blob to cosmosDB
 Advanced Concepts in ADF
 Nested ForEach -pass parameters from Master to child pipeline
 High Availability of Self Hosted IR &Sharing IR with other ADF
 Custom Activity in Azure Data Factory

Real Time Scenarios and Use Cases with ADF Data Flows

 Data Flows Introduction
 Azure Data Flows Introduction
 Setup Integration Runtime for Data Flows
 Basics of SQL Joins for Azure Data Flows
 Joins in Data Flows
 Aggregations and Derive Column Transformations
 Joins in Azure DataFlows
 Advanced Join Transformations with filter and Conditional Split
 Data Flows – Data processing use case1
 Restart data processing from failure
 Remove Duplicate Rows &Store Summary Credit Stats
 Difference Between Join vs.Lookup Transformation& Merge Functionality
 Dimensions in Data Flows
 Slowly Changing Dimension Type1 (SCD1)
 Slowly Changing Dimension Type2 (SCD2)
 Flatten Transformation
 Rank, Dense_Rank Transformatios
 Data Flows Performance Metrics and Data Flow Parameters
 How to use pivot and unpivot Transformations
Data Quality Checks and Logging using Data Flows
 Batch Account Integration with ADF
 Custom Activity in ADF

Draft Lesson

Data Factory UI
Copy Activity on Blob Storage to SQL Database
 Copy Activity on storage account to storage account
 Create Pipeline
 Lookup Activity

Draft Lesson

ADF Introduction
Important Concepts in ADF

 Creating Azure Free Account for ADF
Integration Runtime and Types
 Integration runtime in ADF-Azure IR
 Create Pipeline in ADF
 Azure Storage Account Integration with ADF
 Copy activity multiple files from blob to blob
 Filter activity Dynamic parameters Copy Activity
 Get File Names from Folder Dynamically
 Deep dive into Copy Activity in ADF
Copy Activity Behaviour in ADF
 Copy Activity Performance Tuning in ADF
 Validation of schema in ADF
 Get Count of files from folder in ADF
Validate copied data between source and sink in ADF
 Azure SQL Database integration with ADF
 Azure SQL Databases – Introduction Relational databases
 Azure Functions execution in ADF pipeline
 Sending Mail during the pipeline
 Creating Your First Azure SQL Database
1) Deployment Models
2) Purchasing Modes
 Overwrite and Append Modes in Copy Activity
 Full Load in ADF
 Copy Data from Azure SQL Database to BLOB in ADF
 Copy multiple tables in Bulk with Lookup & ForEach in Data Factory
 Logging and Notification Azure Logic Apps

Draft Lesson

Data Flows Introduction
 Azure Data Flows Introduction
 Setup Integration Runtime for Data Flows
 Basics of SQL Joins for Azure Data Flows
 Joins in Data Flows
 Aggregations and Derive Column Transformations
 Joins in Azure DataFlows
 Advanced Join Transformations with filter and Conditional Split
 Data Flows – Data processing use case1
 Restart data processing from failure
 Remove Duplicate Rows &Store Summary Credit Stats
Difference Between Join vs.Lookup Transformation& Merge Functionality
 Dimensions in Data Flows
 Slowly Changing Dimension Type1 (SCD1)
 Slowly Changing Dimension Type2 (SCD2)
 Flatten Transformation
 Rank, Dense_Rank Transformatios
 Data Flows Performance Metrics and Data Flow Parameters
 How to use pivot and unpivot Transformations
 Data Quality Checks and Logging using Data Flows
 Batch Account Integration with ADF
 Custom Activity in ADF