Imagine...

There are many offerings out there that target to support you in your analytical data estate. Some will give you a database, others offer ETL tools, computes or AI capabilities and storage and again others offer dashboarding and visualization or data governance tools. Every time for each capability you need to subscribe to a separate service or buy a license. BUT! None of the offerings can give you a 360 degree offering that spans over all the components potentially needed.

How would you think about a platform that offers you MPP databases as-a-service that scale to your needs along with a serverless SQL engine that would directly talk to your Data Lake processing the files there? Let me go further: what would you say, if the same platform would offer you an autoscaling Spark engine and all these compute engines would even exchange their metadata with each other?

I have mentioned the storage. So let's go on imagining. What about if these computes would seamlessly integrate with your Data Lake and at the same time would link other data services and components into the environment in a way that you have a growing list of data services that store data anywhere in your Azure environment just a click with your mouse away from ad-hoc analysis or data integration?

Did I say data integration? What if the same environment could give you an ETL/ELT tool with access to a more than significant amount of connectors to access all typical suspects when it comes to data sources?

And what would you think if this platform would even seamlessly integrate with a Machine Learning environment that supports you in all kinds of Data Science projects and will help you to build responsible AI? 

I have mentioned dashboarding and reporting too. You might already call me a lunatic for the following question when I ask the next what-if. But what if the environment I am imagining here would integrate report authoring into all this to enable you to visualize your data in one and the same development experience. A development environment that lets you access your data sources using SQL or Spark with just the click of your mouse.

Ah yes the development experience! Now when we would have all these components mentioned above in one environment sure you would want to have a development interface that will connect all the capabilities with each other. One that would enable you to span your development over all the necessary computes using what ever makes most sense for a certain use case and combine the pieces and orchestrate them in a single place? 

Sure you would want to integrate your work with a DevOps platform to enable and automate CI/CD from Dev to Test to Prod.

And to just put the cherry on top of the cake of imagination! What, if the whole platform would integrate with a Data Governance suite that would catalogue your data estate, give you a searchable interface and even provide lineage over the ETL/ELT up to the data sets in your reports?

Impossible?

Nope!

We're calling this state of the art in Synapse Analytics, Azure Machine Learning and Azure Purview! 

You can find some reading here in the documentation:

If this topic has caught your attention and if you maybe want to know more, I would love to read your feedback. And maybe you want to follow me into some articles that will look into Synapse Analytics in more detail. Stay tuned.

Thank you for reading and stay safe

Patrik Borosch

Other parts of the series: