Tech Talk #1 - Forget Chasing Superstars: How to Hire Good Developers

As part of our Tech Talk series we spoke with Ming T.Chow who is a technology leader within Rakuten (Kobo) in Asia.

Ming has worked in Ireland, Toronto and more recently Singapore for Rakuten and has gone from a hard-core coding position to a very client orientated global strategy position. In summary, Ming is an awesome coder turned tech leader and he has a proven recipe on how to build high performing tech teams successfully.

Over to Ming.......

"We always talk about finding superstar developers and all the great things they can do for companies, however, most of our teams do not consist of superstar developers. By definition, superstar developers are like unicorns, they are rare and are well-sought after. Due to factors such as time, talent pool, budget, project requirements, and team fit, it is often not feasible to hire and keep superstar developers. Instead, being able to consistently hire good developers will provide your organization much more value in the long run than being a star-hunter.

I would place good developers in a separate category than superstar developers because comparing them on the same spectrum may imply that good developers are not as good as the superstars. In reality, they have different strengths and serve different purposes. Good developers blend into teams well and are usually reviewed by peers and managers as steady and reliable. They may not win coding competitions or even be interested in participating, but they are the ones you will not hesitate to put into any project because you know they will be performant with few surprises. For these reasons, good developers often fly under the radar and picking them out from the pool of candidates can be more difficult than finding a shiny superstar.

In this post, I wish to discuss the technical test structure my colleagues and I developed over the years to sift through candidates to identify good developers (and also the not-so-good developers). This structure has been battle-tested through hiring for multiple startups, building new branch offices, and assembling remote teams.

First of all, the general qualities of a good developer that we look for are:

• Can deliver solutions that satisfy the requirements in a timely fashion
• Can make sensible tradeoff decisions based on real-world parameters
• Easy-going and cooperative personality who can constructively criticize and defend solutions
• Can inherit existing code and improve on it without redoing the whole thing his/her own way

To test for these qualities, we are concentrating on improving the technical test part of the interview (HR, senior management, team-fit interviews are not included within this scope). 

The technical test is three hours long: two hours for the coding test, one hour to discuss the solutions provided on the test.

The Coding Test:

The candidates are given a laptop and a sheet of paper containing three questions. They are encouraged to read through the questions carefully first and figure out how much time they should spend on each question before starting. We stress to them that time-management is an important part of this test. The three questions take the form of:

1. The “Are you in the ballpark?” question:

This first question is a straightforward question where the candidate has to read through a passage and extract the requirements to implement the solution. For example, if it is for a payment processing company, the passage can explain the validation logic for 16-digit credit card numbers. The task would be to code the validation logic and output whether a given input is valid and the classification of card type.

Perhaps not surprising to tech hiring managers, this basic reading comprehension and coding skills question was too difficult for many candidates. If this question cannot be completed to a satisfactory level, we try to minimize the time we spend afterwards with this candidate and do so in a respectful manor.

2. The “Are you witty or can you muscle through?” question:

This question is closest thing to a traditional brain teaser within this test. We ask the candidate to implement an algorithm where the description contains some information that greatly reduces the complexity of the problem but if the candidate was not able to identify it, then it is still possible to solve the problem in a more general way but it will require more time and effort. For example, we can ask:

"Given two lists of arbitrary length containing integer values of 1 to 10, produce an efficient algorithm to find the most frequent matching integer between the two lists. Do not use any API functions for manipulating lists or collections.”

While this question sounds innocuous enough, we are always surprised at some the wild solutions that were submitted by less confident or less focused developers. The point here is to allow multiple valid ways to complete the question to avoid inducing the feeling of being stumped. Those who can identify the key gets an easier pass and is appreciated for their attention to detail. Those who did not notice the key but was able to implement an efficient solution within the time constraints is demonstrating a good amount of fundamental skills. Those who burn up too much time trying to re-invent the wheel and/or cannot produce the correct output do not qualify for our definition of a good developer.

  3. And finally, the “How good are you with constraints?” question:

This is the last and most important question. It is another take on testing how the candidate handles tradeoffs in real-world situations. 

We prepared an IDE with the code to a simple program that can run perfectly except that the code reeks of every bad smell in the book with a few more atrocities peppered in from personal experiences. We ask the candidate to refactor the code as they see fit. 

This question tests the candidate on some important topics:

1. Did the candidate violate the number one rule of refactoring? The functionality of the program cannot change (and of course, it must still compile / run without errors)
2. How deep can the candidate dive? Changing bad variable names is a start, but is he/she confident enough to touch the underlying data structures and apply a couple of design patterns?
3. What is the candidate’s priority for refactoring? Is it for speed and efficiency? Code readability? Scalability?
4. Given the time constraints, did the candidate choose the most effective items to refactor? 

Within this question, we found that the candidates who try to rewrite the whole program from scratch inevitably fail due to the time constraints. As well, this gave us great insights on the candidate’s idea on what good code is. The same cannot be as well-explored if we asked the candidate to write a program from scratch within the same time constraints.

As a final benefit, given that this is an open-ended question that can easily consume the full two-hours by itself, by the time we re-enter the room to announce the coding test is over (we usually give five more minutes for them to finish up), the more successful candidates will be in a state of intense work and enthusiasm which leads to very engaging discussions afterwards. 

The Discussion:

After reviewing the code, we talk through each of the questions and get feedback from the candidate about their thought process through each one and what they could have done better.  We spend most of our time discussing the last question because there’s always more to do with the refactoring and we would like to find out what tradeoff decisions the candidate had made. Here, we can ask very specific questions about design and architecture. Additionally, we can try to attack the candidate’s decisions a bit to see how he/she reacts.

We then challenge the candidates to verbally provide solutions for extended scenarios such as scaling the program to be accessible by millions of users and/or the underlying data structure needs to be concurrently modifiable through a network. Without establishing the mutual deep familiarity with the code and functionality of the program, these extended scenario questions will not be as effective to ask. 

Through these intense discussions, we gain a good idea about the candidate’s team fit because we have a very good sample of the candidate’s work attitude and demeanor. As well, the candidate who can perform satisfactorily on these questions and discussion would have demonstrated all of the qualities we are seeking in a good developer.

Observations & Conclusion:

Here are some of our observations and learnings while refining this test structure:

• Candidates do not like being stumped by brain teasers. Indeed, feeling stumped may put candidates in a weaker and/or more defensive position which is not conducive to productive discussions afterwards. 
• Time-management and tradeoff decision-making skills are difficult to measure with questions that have a narrow solution set. 
• Good developers feel energized and are very eager to discuss the test even after two hours of high-pressure work. We can often see it in their eyes.
• There is less reason to ask textbook questions unless it is within the context of a broader design discussion. It is not that they provide no value, but asking a series of one-off textbook questions will not affect the hire or no hire decision nearly as much as the results of the coding test.
• Talking about past experiences and interesting items on the resume is important, but only after there is confidence that the candidate has met the technical expectations from the coding test and discussion. 
• Many candidates like to write unit tests and comments gratuitously for questions they cannot answer but do not do the same for questions they can answer.
• When half of the candidates cannot provide a satisfactory solution to the first question, ask HR and/or recruiters to send through better quality candidates. This one is a blog post by itself…

So far, this test structure has been very successful at hiring good developers (between my colleagues and I, we hired at least 50 people this way). Some of my colleagues have been replicating and tweaking this test for their own hiring after they’ve moved to other teams and companies.

I hope you found this information useful and can adopt some of the ideas here when you’re searching for good developers"

Ming T.Chow