Key challenges in the building data warehouse for large corporate

Lack of strategic focus to build Enterprise Data Warehouse (EDW)

Building EDW is a strategic initiative since it requires a shift in culture, a longer timescale & more importantly it is an expensive affair. Hence, it should be one of the top agendas of the CXOs and they need to closely monitor the progress and also need to provide executive support to break any unwanted barriers.

1.  The need for considerable Time, Effort & Cost
2. Lack of cross-divisional collaboration
3. Technological complexity
4. Changing business data requirements & understanding of business requirements
5. Lack of clarity on the true source of data
6. Lack of ability to manage data quality issues
7. The vested interest of vendors in promoting their own solution
8. The comfort of using divisional data marts
9. Subordinate use of data warehouse

1.The need for considerable Time, Effort & Cost

The typical time taken for a global Corp to build an EDW varies from a couple of years to 5 years. It also requires substantial effort & eventually a huge amount of money to build a data warehouse. Also, Evidence of successful ROI is very opaque in the existing data warehouse implementation.

 2.Lack of cross-divisional collaboration

Building EDW requires constructive collaboration from various teams like multiple business divisions, source system teams, architecture & design teams, project teams, and vendor teams. 

 3.Technological complexity

Mostly, source data is kept in multiple operating systems & multiple database technologies. There are plenty of tools for data sourcing, data quality management, data integration, data warehousing, reporting & analytics. Choosing appropriate technology is not so simple and is complicated by various emerging techniques like data virtualization, self-service BI, in-database analytics, columnar database, NoSQL database, massively parallel processing, in-memory computing and etc,. Also, a traditional data warehouse is required to be integrated with big data technologies & the Internet of Things for gaining business insights.

 4.Changing business data requirements & understanding of business requirements

Most of the time business finds difficulty in defining the data requirements since data requirements keep evolving as the use of data increases. However, the technical team wants finalized data requirements from the business before designing & building a data warehouse.

 5.Lack of clarity on the true source of data

Most of the large Corps has a great legacy behind them and have been growing over the decades through mergers & acquisitions. They have a wider footprint across geographies and various customer segments. In this process, they have acquired many systems that are poorly integrated, less documented, and data is scattered across multiple systems. It is a nightmare for these Corps to identify the true source of their data.

 6.Lack of ability to manage data quality issues

Since data is an organizational asset it needs to be acquired & maintained well. Many front office/customer-facing systems don’t capture quality data at its origination. There is no unified data capturing process across organizations.

For example, the last name of a personal customer would not have been captured in a front office system, since it is not a mandatory field, whereas it may be a mandatory field for another system. Sometimes there is a lack of well-defined processes & technologies to curtail the data quality issues.

 7.The vested interest of vendors in promoting their own solution

Most of the top data warehousing vendors have their own suite of solutions/products in the entire data warehousing ecosystem. These vendors tend to promote their own solutions rather than advocating what is best suited for the customer.

8. The comfort of using divisional data marts

Reporting is an indispensable activity of Coping. Many Corps have built divisional data marts for fulfilling their own divisional needs. Though divisional marts do not provide an enterprise-wide view, many business users are comfortable in using divisional data mart assuming that “Known devil is better than unknown angel”.

9.Subordinate use of data warehouse

Business users from various divisions need to use the data warehouses for reporting, business intelligence, data analytics & advanced analytics to unleash the full potential of the enterprise data asset. Under utilized data warehouse will not grow & will not yield the desired return on investment (ROI)