The Journey from Data to Insights

It has been a long fragmented journey over the last 14 years and the toughest part is, to keep it simple and succinct since each of the topics could be a book in itself. This article is an honest effort to talk about the journey on a high level, especially as a first step for a greenfield setup or simply a scribble from a binary mind.

Evolution

Data and processes have always been two key parts of business. Processes are easier to capture, showcase and improve upon. Capturing data has been a challenge and ‘unclean data’ has been an easy escape. So, lets discuss how do we find a light at the end of the tunnel.

Data Storage

In past, we had to store only high-value data because storage was costly and cloud adoption was in a nascent stage. However, with time, cheaper cloud storage became a reality and provided flexibility while storing data. Improvements in technology made it possible to store unstructured data, and transform the data while it is being used. More and more tools have become available for data governance.

Dirty (Unclean) Data

The fact of garbage in garbage out is still true but advanced analytics and data profiling using AI makes it easier to spot where things are going wrong and fix it.  

As a result, there are many organizations across APAC that have a good technical setup for data but the business strategy and a plan to harness the full potential of data is still missing.

Current Picture

In a recent survey, it was found that 97.2% of companies say they’re investing in big data and AI projects but only 29.2% report achieving transformational business outcomes. If we ask businesses about their vision for data then it's either unclear or built on the fly. Some organizations are planning to use AI but unless the models are clearly scribed, weighted for correctness and security then feeding data blindly could be a dangerous endeavour.  

Simple Solution to a Complex Problem

A structured simple solution often works for complex problems such as making the most of data. We'll talk about it on a very light level for the sake of simplicity.,

1.      Understand current and potentially future data

• Business: Understand the data from business stakeholders. Not all data makes its way to digital systems. Some of it might still be in excel sheets and notepads. Understand which data that is not captured but could be captured as potentially future data using simple means such as power apps. The operating model must be data-driven.
• Technology: Understand how digital systems are capturing data, especially with a focus on the data that is not entered by the business but could still be useful, such as datestamp when a record was updated.
• External: Understand the data that you provide to external entities. Also, capture which external data you capture or if you don’t then potentially future data. Lots of good data is open source such as weather, data from government bodies, and micro-mobility transport data.

2.      Build the capability

It is important that there is a centre of excellence for Data. It can either be in business empowered by technology teams or even better if it is in technology teams led by a techno-functional person with an excellent understanding of business insights. It can be complimented quite well by Data Champions in every department.

Technical limitations should never hinder the journey of creating insights. It should be on the cloud, easy to integrate with, able to manage multiple sources, data should be refreshed at the required frequency and easy to access. The front end used (e.g. Power BI/Tableau) should be visually appealing.

3.      Establish Governance

Data security, lineage, and access control should be at the forefront of the governance.

Data security is a vast topic but at the minimum, we should know what kind of data we have, it should be labelled based on the sensitivity, PI (Personally Indefinable) information etc. who can access it and how we control access.

Data lineage is critical for understanding where the data is originating from and how it influences its use. It also makes integration and change management easy.

Access control is a critical part. Ideally, the access must be role-based so that it inherently takes care of an important part of security. I wanted to use role-based access control (RBAC) offered by platforms such as Azure but haven’t had an opportunity to implement it yet.

The centre of excellence will define how the user roles and access are to be set up. In my opinion, the technical teams should make the data/datasets available to businesses for building reports and insights. The data champions in business who understand business needs quite well, can build the report/dashboards and share those internally.

4.      External collaboration

There are 2 parts of external collaboration. One is with agencies/ third parties and the other is with the community. In the first part of the collaboration, data can be provided to external agencies such as regulatory agencies, or it can be imported from entities like the weather department.

In the second part, some of the key data can be shared with the community. It could be a web template that can accommodate commentary and visual dashboards such as PowerBI/ Tableau.

5.      Journey from Data to Insights

These terms are used quite loosely in the industry however for me, there is a clear difference. Data points tell you ‘What is it’ and insights tell you ‘What you can do with it’. There are 3 stages of using the data for business

a. Single Source – Single source helps us understand various data points that could influence decision-making. There is still value in understanding how the sales in the last few months were and how the trajectory could go in the next few months. My recommendation is to make sure you’ve sizeable data to make a decision i.e. if you want a trajectory then you should have at least 12 months of data. It is still not possible to understand the effects of external events but at least you can see the impact.

b. Multiple Sources – This will help you understand the effect of one source on another for example if you are looking at retail sales data but you also have the weather data at your disposal then it might give you an insight on how the weather data could impact sales. Or you could combine weather data with travel data and understand how it influences commuter’s mode of transport.

c. Insights – This one takes the crown. In the first two stages, our focus was more on the qualitative data which can be automated but to create insights we need experts who understand the impact of qualitative data and events. Combining qualitative and quantitative data not only makes it powerful but also relevant for fact-based decision-making.