What does it take to have a successful career in Data Engineering?

Hi everyone, I’m Connor Baker, one of the newest members of DS Group and I head the Data Engineering recruitment here. As I’m settling into my new role, I’ve thought it was best to reach out to industry experts to see what they think a junior Data Engineer should consider in order to have a successful career within the industry and it benefits me as I get a stronger understanding of my market. My intentions are to also help junior Data Engineers understand the needs of companies as they begin their careers in the Software Engineering field.

The first person I spoke to was Dylan Lentini – Data Engineering Manager at Treatwell

“There’s been a shift in Data Engineering in the last 5 years and having people up to date with the latest evolvement in the industry is essential. Solutions are different from what they were 5 years ago, so there is a need for adaptation and change. 

With that being said, to be successful in Data engineering it is important to be as knowledgeable as possible, particularly with Python, SQL, and the cloud. It is also important to have the willingness to learn and be a great listener to excel in the field of Data Engineering.” 

I also had the pleasure of speaking to Andrei Nita – Director of Data and Analytics at Builder.ai

“When hiring for Juniors, the focus should be on strong fundamentals and dedication. You are looking for potential, people you can equip for success in their future career.

A profile will stand out with the following: distinctions, publications, repositories with relevant code and a Computer Science degree.

Experience in Data is not required, but the most common technologies that should be at an Advanced level include Python, SQL and knowledge of any Cloud provider.

Marco Zanchi (Head of Engineering at BridgeU) also provided some great insight.

“First and foremost, I’d hope they matched the values of the company and that they match the skills required. 

They’d need to get along with the team and be okay with the fact that they may not be suitable, it’s important not to let it get them down.” 

Robert Farrow - Head of Engineering at Profusion

“Tenacity more than anything else, a good attitude, crack their head against a wall and learn! Attitude is the main part of success in Data Engineering. 

In terms of tech, I look for Python, AWS, good software engineering principles, and the ability to write good code.”

Rene-Jene Corneille - Head of Data Engineering at Builder.ai

“Fundamentals are crucial like SQL and Python.

Levelling up is about being curious to learn and being aware of what tech to use. Curiosity is very important.”

Robert Sams – Founder and CEO of Clearmatics

“A willingness to get your hands dirty on the Data Engineering side. We've found that there's an assumption amongst younger candidates that the craft is mostly about applying ML and other techniques they learned in school. But infrastructure and data wrangling problems are often the bulk of the work, and there's a supply/demand imbalance with these skill sets.

So, to finalise I'd say low-level data engineering skills, grit, strong work ethic and attention to detail are a must. These low-level skills are often lacking with candidates – if you have them, you'll stand out from the crowd!”

I hope these insights have provided you with a stronger insight in how to be a successful data engineer, as I’ve definitely learnt a thing or two! It’s clear that the things to focus on as a junior data engineer are having is keeping up to date with the industry, a wanting to learn and a solid understanding of your code/data.

A huge thank you to everyone I spoke to:

Dylan Lentini – Data Engineering Manager at Treatwell

Andrei Nita – Director of Data and Analytics at Builder.ai

Marco Zanchi - Head of Engineering at BridgeU

Robert Farrow - Head of Engineering at Profusion

René-Jean Corneille - Head of Data Engineering at Builder.ai

Robert Sams – Founder and CEO of Clearmatics