I successfully designed and implemented a data integration solution using Azure Data Factory (ADF) to streamline data processing and analysis. The primary objective was to populate custom Azure SQL Database tables with data from multiple views, ensuring efficient data organization and retrieval. To achieve this, I created a robust data pipeline in ADF that extracted data from the views and loaded it into the corresponding Azure SQL Database tables. This pipeline automated the data transfer process, eliminating manual intervention and ensuring data consistency.
Additionally, I leveraged ADF to develop a second pipeline, this time to handle JSON data stored in an Azure storage account and ingest it into Azure Cosmos DB containers. This allowed for seamless data synchronization and ensured that the data stored in Cosmos DB was up-to-date and aligned with the source data.
To enable data-driven insights and visualizations, I utilized Power BI. By connecting the data sources from Azure SQL Database and Azure Cosmos DB to Power BI, I created an interactive dashboard with insightful visuals.
Ultimately, this project exemplified my expertise in designing end-to-end data solutions, from data extraction to visualization, contributing to enhanced data-driven decision-making within an organization.