Design Guidelines for Data Mesh and Decentralized Data Organizations
0 likes133 views
This document outlines best practices for implementing a decentralized data organization using a data mesh architecture. It discusses principles of data mesh including domain ownership, data as a product, and federated governance. Implementation best practices are provided for developing data products, roles and responsibilities, and deploying changes. Self-service data infrastructure is enabled through a data catalog, search, and governance tools.
More Related Content
What's hot (20)
Similar to Design Guidelines for Data Mesh and Decentralized Data Organizations (20)
More from Denodo (20)
Recently uploaded (20)
Design Guidelines for Data Mesh and Decentralized Data Organizations
- 1. Design Guidelines for Data Mesh and Decentralized Data Organizations Thursday, November 17th 2pm CET | 1pm GMT Daniel Tenreiro Sales Engineer | Denodo
- 2. Daniel Tenreiro Arcos Sales Engineer | Denodo Design Guidelines for Data Mesh and Decentralized Data Organizations
- 3. AGENDA 1. Introduction to Decentralized Organizations a. Motivation b. Principles c. Data Product Characteristics d. Data Product Types 2. Implementation Best Practices 3. Closing remarks
- 5. ‹#› ▪ Lack of business understanding in the data team ▪ Lack of flexibility of centralized data platforms ▪ Slow data provisioning and response to changes Motivation Introduction to Decentralized Organizations
- 6. ‹#› ▪ Domain Ownership ▪ Data as a Product ▪ Self-Service Data Infrastructure as Platform ▪ Federated Computational Governance Data Mesh Principles Introduction to Decentralized Organizations
- 7. ‹#› ▪ Domain as Organizational Unit ▪ Independent ▪ Removes bottlenecks ▪ Accelerate changes ▪ Domains have more flexibility Data Mesh Principles - Domain Ownership Introduction to Decentralized Organizations
- 8. ‹#› ▪ Publishing Data Products ▪ Easy discoverable and usable ▪ Understandable and documented ▪ Secure ▪ Interoperable by other Domains ▪ Avoid Data Silos Data Mesh Principles - Data as a Product Introduction to Decentralized Organizations
- 9. ‹#› ▪ Operated by central IT team ▪ Domains manage their own Data Products ▪ Avoid complexity and duplication of efforts Data Mesh Principles - Self-Service Data Platform Introduction to Decentralized Organizations
- 10. ‹#› ▪ Ensure Interoperability ▪ Common Semantics and Conventions ▪ Global Security and Governance Policies Data Mesh Principles - Federated Computational Governance Introduction to Decentralized Organizations
- 11. ‹#› ▪ Source-Domain Data Products ▪ Aggregate-Domain Data Products ▪ Consumer-Aligned Data Products Data Product Types Introduction to Decentralized Organizations
- 13. ‹#› Data as a Product - Implementation Implementation Best Practices Marketing Domain
- 14. ‹#› Marketing Domain Data as a Product - Implementation Implementation Best Practices Data Sources
- 15. ‹#› Marketing Domain Data Sources Data as a Product - Implementation Implementation Best Practices Base Views
- 16. ‹#› Marketing Domain Data Sources Data as a Product - Implementation Implementation Best Practices Base Views Integration Views
- 17. ‹#› Data as a Product - Implementation Implementation Best Practices Base Views Integration Views Interface View Data Sources Marketing Domain
- 18. ‹#› Data as a Product - Implementation Implementation Best Practices Marketing Domain Campaigns Source-Domain Data Product Base Views Integration Views Interface View Data Sources
- 19. ‹#› Data as a Product - Combination Implementation Best Practices Marketing Domain Sales Customer Source-Domain Data Products Campaigns Sales Domain
- 20. ‹#› Data as a Product - Combination Implementation Best Practices Marketing Domain Sales by Customer Sales Customer Source-Domain Data Products Aggregate-Domain Data Products Campaigns Sales by Campaign Sales Domain
- 21. ‹#› Marketing Domain Sales by Customer Sales Customer Source-Domain Data Products Aggregate-Domain Data Products Campaigns Sales by Campaign Sales Domain Data as a Product - Combination Implementation Best Practices Consumer-Aligned Data Products Marketing Report
- 22. ‹#› Domain oriented decentralized data ownership and architecture Implementation Best Practices Campaigns Customer Sales Connection layer Integration layer Data Products JDBC ODBC REST GraphQL OData Source-Domain Data Products Aggregate-Domain Data Products Consumer-Aligned Domain Data Products
- 23. ‹#› Domain oriented decentralized data ownership and architecture Implementation Best Practices Campaigns Customer Sales Connection layer Data Products JDBC ODBC REST GraphQL OData
- 24. ‹#› Domain oriented decentralized data ownership and architecture Implementation Best Practices Campaigns Customer Sales Connection layer Integration layer Data Products JDBC ODBC REST GraphQL OData Source-Domain Data Products Aggregate-Domain Data Products Consumer-Aligned Domain Data Products
- 25. ‹#› Domain oriented decentralized data ownership and architecture Implementation Best Practices Campaigns Customer Sales Connection layer Integration layer Data Products JDBC ODBC REST GraphQL OData Source-Domain Data Products Aggregate-Domain Data Products Consumer-Aligned Domain Data Products Data Products layer
- 26. ‹#› Domain oriented decentralized data ownership and architecture Implementation Best Practices Campaigns Customer Sales Connection layer Integration layer JDBC ODBC REST GraphQL OData Source-Domain Data Products Aggregate-Domain Data Products Consumer-Aligned Domain Data Products Data Products layer Data Products
- 27. ‹#› Domain oriented decentralized data ownership and architecture - Roles Implementation Best Practices Domains Domain Data Product Owner Domain Data Product Owner Domain Data Product Owner Domain Data Product Developers Domain Data Product Developers Domain Data Product Developers Platform Product Owner Technical Validator Central IT
- 28. ‹#› Domain oriented decentralized data ownership and architecture - Roles Implementation Best Practices Responsible for creating, serving and evangelizing Data Products. ▪ Local Administrator ▪ assignprivileges Role ▪ Can connect to Domain specific Data Sources ▪ Can tag and secure Data Products Domain Data Product Owner
- 29. ‹#› Domain oriented decentralized data ownership and architecture - Roles Implementation Best Practices Responsible for building and maintaining the transformation logic that generates the Data Products. ▪ Builds the integration views that implement Data Products ▪ Applies optimizations, caching strategies… ▪ Has permissions over the Domain VDBs Domain Data Product Developer
- 30. ‹#› Domain oriented decentralized data ownership and architecture - Roles Implementation Best Practices Responsible for facilitating the prioritization of the platform services to design and build the experience of the users of the platform. ▪ serveradmin Role ▪ Responsible of the Platform Configuration like VCS ▪ Can create connections to Data Sources in the shared connectivity layer Platform Product Owner
- 31. ‹#› Domain oriented decentralized data ownership and architecture - Roles Implementation Best Practices Technical Validator Responsible for reviewing the views created by the Domains from a technical perspective and apply optimizations or refactoring if necessary. ▪ Technical background. ▪ Local administrator of Domain VDBs.
- 32. ‹#› Domain oriented decentralized data ownership and architecture - Development Implementation Best Practices Main Domain VDB Domain Data Product Owner Data Sources Domain Data Product Owner is the owner of the Domain Virtual Database 1
- 33. ‹#› Domain oriented decentralized data ownership and architecture - Development Implementation Best Practices Main Domain VDB Domain Data Product Owner Data Sources Copy Domain VDB Domain Data Product Developer Domain Data Product Owner is the owner of the Domain Virtual Database Each Developer has a copy of the Domain VDB connected to the VCS with its own user 1 2
- 34. ‹#› Domain oriented decentralized data ownership and architecture - Development Implementation Best Practices Main Domain VDB Domain Data Product Owner Data Sources Copy Domain VDB Domain Data Product Developer Domain Data Product Owner is the owner of the Domain Virtual Database Each Developer has a copy of the Domain VDB connected to the VCS with its own user There will be an independent repository for each Domain VDB 1 2 3
- 35. ‹#› Domain oriented decentralized data ownership and architecture - Deployments Implementation Best Practices SOLUTION MANAGER Revision Validated Revision Deployment Target Environment
- 36. ‹#› SOLUTION MANAGER Domain oriented decentralized data ownership and architecture - Deployments Implementation Best Practices Domain Data Product Developer Revision Validated Revision Deployment Target Environment Domain developers can create and validate Revisions 1
- 37. ‹#› SOLUTION MANAGER Domain oriented decentralized data ownership and architecture - Deployments Implementation Best Practices Domain Data Product Developer Revision Validated Revision Deployment Domain Data Product Owner Target Environment Domain developers can create and validate Revisions 1 Domain Admin can create, validate and deploy revisions 2
- 38. ‹#› SOLUTION MANAGER Domain oriented decentralized data ownership and architecture - Deployments Implementation Best Practices Domain Data Product Developer Revision Validated Revision Deployment Domain Data Product Owner Target Environment Domain developers can create and validate Revisions 1 Domain Admin can create, validate and deploy revisions 2 Deployment is done by Solution Manager through a specific deployment strategy 3
- 39. ‹#› SOLUTION MANAGER Domain oriented decentralized data ownership and architecture - Deployments Implementation Best Practices Domain Data Product Developer Revision Validated Revision Deployment Domain Data Product Owner Target Environment Domain developers can create and validate Revisions 1 Domain Admin can create, validate and deploy revisions 2 Deployment is done by Solution Manager through a specific deployment strategy 3 VCS can be used to save a backup of the deployment 4
- 40. ‹#› Self-service data infrastructure as a platform Implementation Best Practices Sales by Customer Sales Customer Data Catalog Sales by Campaign Campaigns Virtual DataPort
- 41. ‹#› Self-service data infrastructure as a platform Implementation Best Practices Sales by Customer Sales Customer Data Catalog Sales by Campaign Campaigns Metadata Search Virtual DataPort
- 42. ‹#› Self-service data infrastructure as a platform Implementation Best Practices Sales by Customer Sales Customer Data Catalog Sales by Campaign Campaigns Virtual DataPort Metadata Search Content Search
- 43. ‹#› Self-service data infrastructure as a platform Implementation Best Practices #aggregatedataproduct Metadata Search Content Search Browse Databases, Tags and Categories Sales by Customer Sales Customer #dataproduct #dataproduct Data Catalog Sales Category Sales by Campaign Marketing Category #aggregatedataproduct #dataproduct #productionready Campaigns Virtual DataPort
- 44. ‹#› Self-service data infrastructure as a platform Implementation Best Practices #aggregatedataproduct Metadata Search Content Search Browse Databases, Tags and Categories Sales by Customer Sales Customer #dataproduct #dataproduct Data Catalog Sales Category Sales by Campaign Marketing Category #dataproduct #productionready Campaigns Virtual DataPort Browse Relationships #aggregatedataproduct
- 45. ‹#› Self-service data infrastructure as a platform Implementation Best Practices #aggregatedataproduct Metadata Search Content Search Browse Databases, Tags and Categories Sales by Customer Sales Customer #dataproduct #dataproduct Data Catalog Sales Category Sales by Campaign Marketing Category #dataproduct #productionready Campaigns Virtual DataPort Browse Relationships Add Endorsements, Warnings, Custom Properties… #aggregatedataproduct
- 46. ‹#› Self-service data infrastructure as a platform Implementation Best Practices Metadata Search Content Search Browse Databases, Tags and Categories Browse Relationships Governance Data Catalog Marketing Category Virtual DataPort Add Endorsements, Warnings, Custom Properties… Campaigns
- 47. ‹#› Self-service data infrastructure as a platform Implementation Best Practices Metadata Search Content Search Browse Databases, Tags and Categories Browse Relationships Data Catalog Virtual DataPort Governance Add Endorsements, Warnings, Custom Properties… Execute & export queries Campaigns SELECT * FROM Campaigns Deploy Saved Queries Export to File
- 48. ‹#› Federated Computational Data Governance - Interoperability Implementation Best Practices Marketing Sales by Customer Sales Customer Campaigns Sales by Campaign Marketing Report Sales JDBC ODBC REST GraphQL OData
- 49. ‹#› Federated Computational Data Governance - Global Security & Governance Policies Implementation Best Practices Campaigns Customer Sales #dataproduct #marketing #dataproduct #dataproduct Global Security Policies Marketing Sales Role Based Security
- 50. Closing remarks
- 51. CLOSING REMARKS ▪ Decentralized Data Organizations like Data Mesh are new data management paradigms ▪ Each Domain manages its own data ▪ Improves data quality ▪ Speeds up the creation and availability of Data Products ▪ Through Data Virtualization we can implement Data Mesh principles ▪ Each Domain can work independently over the same platform ▪ Data Virtualization abstraction layer simplifies Data Products creation and their interoperability ▪ Provides with different integration capabilities ▪ Governance and Security ▪ Self-Service infrastructure through Data Catalog
- 52. ‹#› Resources Closing remarks ▪ KB Article: Denodo Design Guidelines for Data Mesh and Decentralized Data Organizations
- 53. Q&A
- 54. Thanks! www.denodo.com info@denodo.com © Copyright Denodo Technologies. All rights reserved Unless otherwise specified, no part of this PDF file may be reproduced or utilized in any for or by any means, electronic or mechanical, including photocopying and microfilm, without prior the written authorization from Denodo Technologies.