**This is a continuation of part one, I suggest you to check that first to get a clear understanding** Once the first condition is completed let’s check the second which I named as ValidRows as it is going to capture only the non-error values. Compared to the first condition this is very simple as we […]

Azure Data Factory is a tool with tremendous capabilities when it comes to ETL operations. It has many features that would help the users to cure and transform the data that we load into it. The developers or the users face many real-time issues when performing their ETL operations one such common yet unavoidable scenario […]

Azure Synapse SQL is a technology which resides inside the Synapse workspace. Totally we have two pools which we have discussed in detail in one of our articles few weeks ago. Dedicated SQL Pool Serverless SQL Pool The built-in ‘Serverless SQL Pool’ gets created automatically when you create the workspace and the ‘Dedicated SQL Pool’ […]

In line with our previous articles, today we will see how to create, schedule and monitor a pipeline in synapse using synapse analytics studio. Pipeline is ETL with workflow where we will execute and extract the results. A pipeline can be a single or group of activities to be run. Activity is a task to […]

This article is a continuation from Part1 which I posted earlier. I strongly recommend you to go through part 1 before you go through this article. The demo we are going to see will use apache Spark serverless pool model where we will be loading a parquet sample data file into spark database (yes, we […]

This is the part one article of the two part series with demo which explains analyzing data with spark pool in azure synapse analytics. Since the topic touches apache spark heavily, I have decided to write a dedicated article to explain apache spark in azure -hence this part one. Pls make sure to read the […]

We are all aware that SQL is commonly used to query structured data but in Synapse Analytics we can use SQL to query unstructured data saved in files like csv, parquet etc., using OPENROWSET function and it is one of the many features that can be done using synapse analytics. In this week’s article we […]

In continuation to our previous article in this article we will investigate how to create our first synapse workspace. I strongly recommend you have a look at my previous article where we have discussed the basics of azure synapse analytics and what can be done through it. To get started with azure synapse you must […]

In Azure Synapse Analytics you will be frequently crossing over a term called SQL pools. Its good to know the difference and the working functionalities of both of them. No requirement will be similar to the one before and the end users may need different types of usage for each project. Microsoft has kept that in […]

Azure Synapse Analytics is a single solution for all data needs like ingesting, processing, and serving the data. It delivers unified experience of data integration, data warehousing and big data analytics in a single workspace environment. Azure Synapse analytics can be easily integrated with other services provided by azure like Azure Machine Learning, CosmosDB and […]

Introduction: In this article we will check how we can copy new and changed files based on last modification date. The steps have been given below with explanation and screenshots.  As of this writing Azure Data Factory supports only the following file formats, but we can be sure that more formats will be added in […]

Introduction: In this blog, we will look into Azure Data Factory Triggers which is an important feature to scheduling the pipeline to run without manual intervention each time. Apart from regular advantage to schedule the pipeline for future runs (which is very common), the azure data factory trigger has a special feature to pick and process data from […]

Introduction: The linked services in azure data factory have the option to parameterize and pass dynamic values at run time. There might be requirement where we want to connect different databases from the same logical server or different database servers itself. Traditionally we would create separate linked services for each database or database servers but […]

Introduction In this article we will look at our first hands-on exercise in Azure Data Factory by carrying out simple file copy from our local to blob storage. The steps has been given below with explanation and screenshots. Create a storage account After creating storage account, create container which will hold the data that we […]

As we all know that data is the new oil in the world, but it is more than that. The data projection and insights generated can make or break a company’s prospects. Every organization will face challenges in some form in any or all the below actions. Acquiring / data procurement Storing and archiving the […]

Introduction A lot of consumer data is being posted on social media every minute and social media analysis has become a critical component in audience analysis, competitive research, and product research. Social media analytics and its tools are helping organizations around the world understand currently trending topics. Trending topics are those subjects and attitudes that […]

Obviously you should have an active Azure subscription. If you are testing out this feature you can create a free account for $200 free credit to explore azure and 12 months of popular free services. Creating resource group All resources are deployed and managed from a resource group. A resource group is a logical collection […]

This article is an quickstart demo of how one can send or receive events from Azure Event Hub using python script. If you are new to Event Hubs please check my previous post which explains the basics before you continue. We will be using two python scripts, ‘send.py’ and ‘recv.py’ for sending and receiving test […]

Azure Stream Analytics is a fully managed PaaS (Platform-as-a-Service) and a real-time streaming service provided by Microsoft. It consists of a complex event processing engine designed to analyze and process vast volumes of real-time data like stock trading, credit card fraud detection, Web clickstream analysis, social media feeds & other applications. For quicker analysis of […]

Azure Event Hubs is a highly scalable publish-subscribe PaaS service that can ingest millions of events per second with low latency and stream them into other applications. We can consider Event Hub as the starting point in an event processing pipeline often it represents the “front door” for an event pipeline. Event Hubs provides a […]