Technical competence and how I got it back

A year ago my calendar was full of meetings. Too much projects, too much micromanaging, too much legacy... and the technical gap between me and the world got bigger day by day. I hadn't had time to narrow the gap: cloud solutions, machine learning, different distributed environments, tools around hadoop, new languages, containers... and the big picture around all these: How to build company's bigdata analytics platform roadmaps? And more specifically how to justify those decisions?

I just didn't have enough technical competence to do the decisions anymore. On the other hand that was just my opinion: collegues saw me still as a professional who knew a lot and made justified decisions. But I knew I need to learn more. A lot more.

So I decided to left Elisa and thought I will learn new things as a data engineer/architect consultant. My vision had a lot of different environments, a lot of different data related tasks and not meetings.... and I got what I wanted.

Within a year I

• learnt new cloud environments (Azure, AWS, private cloud) and their own solution stacks
• designed and implemented real time data pipelines (with cloud native solution stacks, with open source tools and with edge solutions).
• improved software engineering skills (programming in general, devops, ansible, ci/cd-pipes, spark, kafka, nosql-db's, hive, object storages...)
• used different programming languages: python, C#, scala
• compared native cloud tool stacks to open source tools (cloud solutions vs kafka/flink/spark/hadoop/hive/etc.)
• understood different kinds of IoT possibilities
• learnt new BI tools (Tableau, PowerBI)
• understood how machine learning pipelines are done
• understood about the container-based approach for data analytics (docker, kubernetes, scaling with those, suitable usecases..)
• got certified in Azure
• got really good feedback

... and much more.

But I missed the decision making. And I missed the domain knowledge. I wanted to know the datasets and be responsible of them. Also I missed the small enhancements "that just needs to be done" without billing for hours.

... and then I missed the "data science". Although my data science had been mostly data investigation, but I knew that I can learn more and wanted to learn more. I took some courses and understood that I know the maths behind them already from university years.

Then after almost a year as a consultant I was offered to come back to Elisa.

Within last few days I have: gathered most important changes for data scientists from upcoming solution stack upgrade, optimized&fixed java/spark/hbase/oozie-code, tuned our deployment standards, had discussions about future of advanced analytics with several people, tested and gathered experiences of local distributed deep learning environment, used PowerBI as substitute of drill explorer for quick data investigation, learnt more and more about ml/dl and how to build production-level solutions with those tools within our environments...

... so it seems that the gap is mostly cleared. Now I know again what I'm doing and I'm good at it! And damn happy. 

PS. Big thanks for Bigdatapump/CGIAA and Bitfactor: talented collegues in both companies were a crucial part of this journey.