Hadoop vs The Trough

I was at an analytics conference recently where the key note speaker talked about how Hadoop is entering the Gartner Trough of Disillusionment.  This is part of the Gartner Hype Cycle that tracks new technology from its initial rapid rise in the Innovation Trigger, to the Peak of Inflated Expectations, then an equally rapid descent through the Trough of Disillusionment.  From there, it’s a steady climb up the Slope of Enlightenment to the Plateau of Productivity.  It is a nice simple lifecycle that actually tracks new technology surprisingly well.

Hearing about Hadoop’s entry into the trough raised a smile to my face and a little cheer within.  This is not, I hasten to add, because I don’t like Hadoop.  Quite the opposite.  I have produced the internal white paper for our company explaining what Hadoop is and why we need it.  From there I have created the conceptual architecture that integrates Hadoop into our existing data warehouse estate.  Put simply, I get it.  The reason for my internal cheer was a hope that we, as data professionals, will start to apply good data architecture practice to the field.  Let’s look at an example.

Schema-on-Write vs. Schema-on-Read

Several of the use cases I have put forward for Hadoop rest on the schema-on-write vs schema-on-read argument.  Schema-on-write is what we do in the traditional data warehouse environments.  It entails understanding our data sources, transforming them into a consistent format, syntactically and semantically, then loading them into a data schema.  This data schema is defined prior to data loading, and is supported by a data model.

Schema-on-read says “let’s get hold of the data and just load it, from there we can apply data retrieval methods and see what’s in there”.  I have massively simplified the method here, but you get the idea.  No requirement to transform the data before loading.  No requirement to have a fixed data schema or model before loading.  Schema-on-read is a powerful, useful and quite reckless tool depending on its use.

The value of Schema-on-Read

There are times where business cases struggle to articulate the value of putting data into corporate data warehouses.  Requirements may not be fully understood, the cost of integrating the data can seem high when compared to value understood at that point in time are just two examples.  Schema-on-read can be used to capture data into relatively low cost storage with minimal effort.  From there, it can be explored iteratively with the aim of looking for value.  For me, this brings us back to the spirit of what people like Bill Inmon originally intended.  Low cost iterative development in a safe environment, done until we find data products of value.  This, if done correctly, is what schema-on-read can give us, and I love it.

But we’re only half way there

So going back to my point about good data architecture practice.  There is a danger that once we have found value in the data, and even producing semi-live reports and analysis, that we can consider our job done.  But doing this too many times will result in the data spaghetti that I often see in organisations.  So the next step is back to the business case.

If we have found value in the data, and that value can be sustained over time, then we should complete the business case for this data to be moved into the corporate data warehouse (by the way, when I talk about the corporate data warehouse I make no assumption on the technology, be that relational, NoSQL, etc. , it’s just the trusted source of data for the organisation).  But this time we are going back with a proven value and a complete understanding of the new data source and business requirement.  This is where the good data architecture comes into play - understanding our business requirement, the data needed to satisfy this and the methods to align the delivery of the data to the requirement in a secure and repeatable manner.

So don’t be too disillusioned by the trough.  This is good news for Big Data, Hadoop and the data professionals that work in this area.  By applying good data architecture practice to the field of Big Data we will see sustainable business value being driven from that data.  I can already feel myself climbing the Slope of Enlightenment.