In this article we are going to discuss about how to pick and delete only specific files from the ADLS storage container by passing filenames taken from a excel/csv file column value. File deletion: Recently I came across a requirement for file deletion in ADLS. Azure Data Factory’s delete activity is enough to complete this […]

Introduction In this article we will look at Dynamic Management Views and how can we leverage them to monitor the workloads in an azure synapse analytics workload. We will learn this today with a practical use case and few examples focussing on synapse workload monitoring. Dynamic Management Views Dynamic Management View or simply called DMVs are nothing […]

At an enterprise level, every project schedules and runs multiple Azure Data Factory pipelines but tracking their outcomes in ADF studio is a cumbersome process. There are companies who after for every failed pipeline activity with some error, they must track them down by drilling down each activity until they find the failed one and […]

Data security is hot topic given the data breach we hear about it every day. Though there are various specialized tools available in the market, multiple questions arise on their accessibility, Sharing and data transfers within the organization. Mostly in an organization there might be need to refresh(copy) production sensitive data to multiple nonproduction environments […]

Dynamic data masking is a feature that is available in Synapse analytics to restrict the exposure of sensitive data to the end users. We can configure data masking to hide sensitive data in the result sets that are queries by the users. Using data masking we can not only restrict also specify the amount of […]

This article provides a step-by-step guide for getting started with Azure Synapse Link for Azure SQL Database. I strongly recommend you go through my previous article which explains the basics of Synapse Link for SQL before proceeding with this (creating it) for better understanding. Configure Source Azure SQL Database Create a linked service to your […]

The newly released feature ‘Synapse link for SQL’, enables near real-time analytics into Azure Synapse analytics over operational data from both Azure SQL and SQL Server 2022. It provides seamless integration between the SQL database and Azure Synapse analytics. The rich feature it provides enables users to run analytics, machine learning or BI workloads on […]

The log analytics will monitor the synapse pipelines and provide us more insights once if the job fails. The Azure Synapse integration with Log Analytics is particularly useful in the following scenarios: You want to write complex queries on a rich set of metrics that are published by Azure Synapse to Log Analytics. Custom alerts […]

Introduction: Azure synapse analytics provides standard database templates for various industries to use and create DB model as per their company needs. These are readymade templates which can be created with rich metadata for a clear understanding that can be implemented anytime with fewer steps. Database templates are in simple terms, business and technical data […]

Introduction: In day to day operations we must have faced requirements to backup and restore or copy an Azure data factory from existing to new ones. In todays demo we will see how can we backup and restore the Azure data factory using ARM templates export/import option in azure data factory studio. Steps: I will […]

Introduction: One of the main objective of any business that is using cloud services is to optimize resources and lower the on-going costs. Most of the organizations done need access to the data warehouse layer round the clock and they will be using reporting dashboards to view the information. In such scenarios it is best […]

Introduction: Parameterization is very useful when you want a reusable code that you can use forever and get the output by executing it only by changing the parameter for all your future requirements. Traditionally while coding you will declare variables which are static(see image below) but with parameterization you can use dynamic parameters all through […]

In this article we will look into how could we run both Python and SparkSQL queries in a single notebook workspace under the built-in Apache Spark Pools to transform the data in a single window. Introduction:In Azure synapse analytics, a notebook is where you can write live code, visualize and also comment text on them. […]

We are going to see a real-time scenario on how to extract the file names from a source path and then use them for any subsequent activity based on its output. This might be useful in cases where we have to extract file names, transform or copy data from csv, excel or flat files from […]

Introduction: In this post we will discuss on how to create an external table and to store the data inside your specified azure storage parallelly using TSQL statements. What is CETAS: CETAS or ‘Create External Table as Select’ can be used with both Dedicated SQL Pool and Serverless SQL Pool to create an external table […]

Introduction: In continuation to our previous article, we will look at how could we use parameterization into datasets and pipelines. We will also implement a pipeline with a simple copy activity to see how and where we can implement parameters in azure data factory. Consider a scenario where you want to run numerous pipelines with […]

Today we will check how to create an external data source to access data stored in other resources. If you could remember, in one of our previous articles we have discussed that there will be a Logical Data Warehouse (LDW) which will work similar to a database that you could see in azure synapse analytics. […]

In continuation to our previous article on Azure Synapse Analytics, we will deep dive into the sharding patterns(distributions) that are used in the Dedicated SQL Pool. In the background, the Dedicate SQL Pool divides a work into 60 smaller queries which will be run in parallel on your compute node. You will define the distribution […]