How can a data scientist bring value?

I often get asked this wonderful question by aspiring minds: “how can a data scientist bring value?” , and often a more explicit variant “isn’t data science quickly getting commoditised?”

Summing up my experience so far, the answer is like an onion.

Outermost layer of the onion: You are given a standard problem. You use off-the-shelf tools to churn out standard analytics and run-of-the-mill reports with minimal coding or algorithmic intervention. This layer is already commoditised, and as a DS you add little value if your work is restricted to this layer. Although, this isn’t a bad place to start your career as a DS.

Second layer: You are given a unique data-science problem along with the data necessary to solve it. You unleash your creativity and build a solution for it after iterative experimentation, good dose of learning, common sense, perseverance and disciplined coding. You may draw upon your existing technical skills, or you may pick up anew along the way. The end result may be an algorithm that goes into a product, or may be an insight that forms core of a sought-after report. The core value you bring is through deep technical experiments in developing the algorithm or extracting the valuable insights or both. Majority of data science projects would fall in this category. Upwork, Elance, Experfy and their kinds have began commoditising this layer in last few years.

Third layer: You are given a business problem, or often you discover a business problem yourself after talking with customers and domain experts. You translate the business problem to a DS problem. You gather, clean, format and prepare the data (this is almost always the longest step in the process). You build solution for the problem using data science wizardry. You translate the solution / insights back to the business context. You communicate the solution with the right people in the right way, which help make your solution go live. You follow through until it delivers the impact it was designed to deliver to the end user. In this layer, in addition to your tech wizardry (as in Layer 2), you draw upon your domain knowledge, communication skills, attention to detail and discipline. The core value you bring here is in translating from and to the business context, data munging, and following the solution through till the end.  This layer isn’t at risk of getting commoditised any time soon.

Fourth layer: You may have been given a business problem, or you discover it yourself. You may have gotten shining clean data on a silver platter, or you toil through the data gathering and preparation exercise as usual. You may know very well the data science technique that need to be applied, or you may pick them up on the fly.

Those are mere details.

What makes this layer most valuable is the fact that the insights / outcomes of your work clash in powerful ways with your stakeholders' expectations / world view. Example: Your company culture takes pride in doing things a certain way, but your experiments show conclusively that an alternative is more beneficial to the company. Your client has plotted the manufacturing strategy while implicitly betting on success of material/process A but looking at the data you find that the alternative B works much better. You are a statistician auditor hired by a Big Pharma to get an external evaluation of effectiveness a new pill that they have invented, but when you crunch the numbers its proves no more effective than a placebo. You are told to quantify impact of a proposed acquisition that the CEO is very gung-ho about, but your forecast indicates that the acquisition may be a bad move.

The opportunity for you is not only in communicating the troublesome findings, but in ensuring that they are embraced by the people at the helm and result in appropriate action. To achieve that you need to draw upon not only your technical and communication skills, but also on your courage, compassion, perseverance and faith in your ability to bring good. The very qualities that make us human. Here you deliver your value more by emotional labor and less by technical / skilled labor. Also, while most people and companies want to be data driven, they are usually driven by stories crafted around the data than the data itself. If you just show the data, smart people will quickly craft a story around it that fits their existing world view. So you need to tell the right stories at the right time and in the right way to drive the message home. Of course this is art not science and of course you may often fail in driving the message home. The inherent conflict, friction, uncertainty, risk, emotional labor is what makes this opportunity least pursued and yet this layer the most valuable. I will assert that this layer is *never* getting commoditised -- at least not until robots have the same EQ as humans.