Analytics Platform: D.I.Y. or BUY - Considerations. Gotchas. Recommendations

Considerations

Building a modern end-to-end data platform is a time consuming and an expensive option. There is no easy way around this. One may informally enquire from the engineering teams at Talend, Trifacta, Looker and other point tool providers what it took them to get them to enterprise ready product; let us not even speak about end-to-end unified analytics platform just yet. And, not only the expense of teams and time to build and support this code complexity, but the need to continuously evolve components and platform capabilities to support - evolution of underlying technologies, organizational policy changes, evolving end-user needs, is not a job for the faint hearted.

D.I.Y.

Building a data analytics platform from scratch may seem like an attractive option upfront and may give the team a sense of control over their future. In fact creating a basic end-to-end workflow to handle few analytics scenarios is often easy, and done frequently by most organizations of today. There are plenty of open-source tools and frameworks available for download, and you may also procure some commercially available tools or cloud services for the same. These analytics cases with limited data sources/sets are often not too hard to build, and work well for a few quarters.

However, more often than not, it does not get you where you need to be with regard to scale, roadmap features and business timelines. And most organizations who have done so are struggling to justify these costs and their related business value (ROI, monetization, market share, cost reduction, others).

Reality of the D.I.Y. solution

A vast majority of enterprises who have undertaken such journeys, realize that home grown solutions have limitations and are not able to scale to support increasing analytics scenarios and use cases. This is often for lack of engineering vision, product roadmap, product engineering capability and/or simply not enough right type of knowhow/skill in the team. You are stuck with what you have built and sunk cost fallacy stares at your face. And sometimes your competition gets ahead while you're caught in your data challenges.

D.I.Y. Recommendation:

Build yourself when you have ideally all, or most of these factors in your favor:

- Deep pockets to invest in product design and engineering over a multi-year time frame and an appetite for risk associated with such efforts should they fail

- Sufficient in-house engineering and product leadership able to build a multi-year roadmap and continually oversee and guide through the journey. Ideally you have a history of building enterprise solutions

- You have a generous supply of deep engineering talent. Building and maintaining an end-to-end platform is akin to building a product company of sorts minus the marketing/sales

- You have some specific reasons that require you to alter/modify data platform capability frequently and don't want external dependencies on an outside vendor

- And most importantly, this product engineering exercise will not distract you from your core business

Buy

While it sounds simple, enterprise software procurement too can have its share of challenges. You have to evaluate product for both flexibility, stability and scale, in your specific environment, not easiest thing to do. You may end up procuring a tool that does not provide you the flexibility needed in your data journey, or its upgrades may cause entire data chain to be break. So careful consideration is needed in making this decision.

The pros here are rather straightfoward and well understood - some peace of mind with regards to readily available feature/capablity and support when you need it; the core team can focus on building its own business - retail, healthcare, logistics etc; and the internal cost and effort to actually build this can be avoided.

Gotchas w.r.t. Buy Decision

Sooner or later, most enterprises will evolve to need sophistication and end-to-end capability in your data chain. It's not an if, just a matter of when. Almost never buy just based on a few impressive features and capabilities. e.g. a product may have strong Data Science capabilities but not enough capabilities for fault tolerance or flexibility to handle evolving data sources; another tool may be a leading visualization product but may have minimal capabilities in other areas. Integrating tools across vendors is not only expensive and time consuming, the maintenance and upgrades can be disruptive for the entire data chain. Instead, look at the entire data chain from ingestion to real-time recommendations and governed dashboards, and many things in between like lineage, security, governance and others. Enterprises are advised to be cautious with the fact that the cost to own and operate enterprise software is not limited to just license.

Buy Recommendation:

Consider a buy in one or more of these scenarios

- You do not have robust architectural and product engineering leadership needed to build, scale and evolve an enterprise grade product

- You lack the deep pockets needed to invest in resources for a multi-year product development journey

- Your appetite for failure is low, in case your development efforts don't bear fruit

- You core business consumes most of your time, and the time/effort in planning, building and managing will be a distraction and drain on management time

Before you Buy

Key Recommendations to consider:

- Invest in some rigorous POCs with indicative use cases, for perhaps 2-3 scenarios which are at least 1-2 years forward looking. It is ok to invest time and $ for something this critical. It may save you tremendous heartache at a later stage

- Evaluation of vendor journey and product roadmap, and its mapping with enterprise's own data journey. Ideally chosen product vendor capabilities are in line, or perhaps ahead of your roadmap

- Rigorous cost estimations - a TCO analysis for a minimum 2-3 years and ideally 5 years is needed, where many miss the boat by comparing license and resource only (should include infra, license, engineering resources, usage/compute, training, ongoing support, and upgrades). Strongly suggest that you work closely with your platform vendor to help establish this indicative TCO (perfect numbers are not feasible upfront) to the extent possible (its also a function of how well your use cases and business and technical usage is defined). You may find it interesting that many vendors will go in circles and avoid this under any pretext - those with less transparent pricing will particularly struggle with this. And this exercise itself will filter out the non-compatible vendors.

- Avoid falling for freebie credits and major upfront discounts in Year 1 and Year 2. These locking strategies come with a lot of fine print and associated expenses for future years. Ask the experience of any Fortune 500 CIO in 2023, as they struggle to lower their escalating cloud spend.

Hope this framework will assist you with your decision making.

Until the next post.