Data Quality in the Era of AI

Today - in the tech-world, every hour, AI is evolving, technology is advancing, and data is increasing. With growing data, grows the need to govern it and also manage and maintain its quality to make right use of data. We need to quickly adapt newer and better ways to manage, discover and access data.

In my previous blog (click here to read), I discussed about various data quality qualms, how to address them to build a better relationship with data quality in an organization, what are some of the measures to take and most importantly, how we can think about DQ with the new lens of AI.

In this blog, I would like to show and discuss about Microsoft Purview where the data governance, data quality has been implemented keeping AI at the core of it all. In this article, you will see a glimpse of UI with AI assistance to create and manage data domains, create data quality rules, understand the data quality related health actions etc.

Microsoft Purview has been envisioned to help customers to get accelerated data value creation with - a responsible offense and secure data with an essential defense.

It helps rapidly advance business innovation by enabling business units across the organization to govern their data. Let's take a sneak peek at the UI of Microsoft Purview.

Data Domain and Data Product Creation

With the introduction of domain and data product, Microsoft Purview empowers you to structure data according to your business context and the specific terminology you use. It seamlessly classifies data for discovery, ensures data quality management, and facilitates data access governance, all with the assistance of Copilot.

Now comes the best part - developer productivity enhancement for data quality by leveraging on the AI assistance.

On the selected data product, you can ask Copilot to assist you to understand it better. You may ask questions around source of data aka lineage details, data asset usage, data quality scoring etc.

On a selected product, you may create rules manually.

There are 7 DCAM rules that we can create as shown below.

You can ask copilot to suggest the DQ rules and copilot will recommend the DQ rules based on the content of the data product.

After the DQ rules are selected/adjusted/modified per the need of the customer, we can initiate a DQ scan post which we can see the DQ trend, DQ scoring and other details within the same UI.

The DQ score is highlighted in the Data product page too for a bird's eye view of data product, its quality, it's subscriber details etc.

Conclusion

This blog is an attempt to show you just the tip of the iceberg when it comes to Microsoft Purview Data Quality, use of AI in Data Quality. In DQ itself there are several features that are not discussed in this blog like the data profiling, data remediation etc. DQ is just one arm of Purview. We have general Data Governance, Data Privacy, Policy and other aspects. There are copilots to assist each of these arms like copilot to help create/recommend glossary term, simple UI to create/manage access policies etc. I will discuss more about it in my subsequent blogs. Please watch this space.

Please share your thoughts, comments, feedback in the comments section and if you find this article useful, please give it a thumbs up.

#Microsoft #MicrosoftPurview #DataInTheEraOfAI #DataQuality #DataManagement #CloudScaleAnalytics #DataDomain #DataProduct #Copilot