Free trial

Scenario-based Coding Challenges

  • The secret to identifying developers that deliver

Julian Ghionoiu
Coding challenge designer
Seasoned developer and entrepreneur

Abstract (TL;DR)

Developer assessment methods have traditionally focused on what a developer can produce (results oriented) rather than how they approach a problem (process oriented).

However, in order to hire developers that repeatedly deliver quality code, you need to focus more on the development technique, rather than solely assess code samples.

Scenario based coding challenges provide predictive insights into all the areas you really need to focus on when assessing developers

Table of Contents

In this paper, we will look at:

  • The pros and cons of current assessment methods and score them based on their predictive power
  • Scenario-based coding challenge - a new way to assess developers
  • How to structure an optimal hiring process

Assessing developers is challenging

Problem

As an engineering manager, your assessment efforts can be summarised by this simple question:

“How do you identify developers that deliver – and be sure that you are making the right hiring decision?”

Top Developers are scarce and often “hidden” amongst a bunch of CV’s. Building your team involves a constant race against your competition. Too many hoops can scare your best candidates away.

Simple coding challenges may weed out your worst candidates, yet never reveal who’s best.

The result is hours of fruitless interviews, phone calls and other time-wasting drudgery. More hours stolen from your high-value senior Developers.

And when it comes to making an offer, still not sure that it is the right call.

Add up all of your risks, costs and potential mistakes, and it’s clear:

“Finding quality, reliable Developers for your team is a hugely expensive and risky undertaking.”

What insights do Engineering Managers want?

As an engineering manager, your assessment efforts can be summarised by this simple question:

  • development technique
  • delivery pace
  • code quality
  • testing strategy
  • experience with key technologies (frameworks, languages, tools)

This provides them with a high-level of confidence to invite the candidate for a F2F interview.

Traditional assessment methods

What predictive insights do current assessment methods provide?

Firstly, “CV filtering" is not an assessment method as it provides no actionable insights.

The main assessment methods used by companies are:

  • traditional take-home exercises
  • online programming puzzles
  • pairing sessions
  • questions via an online questionnaire
  • questions during the face to face interviews
  • whiteboarding sessions

The following section provides an overview of each, their pros and cons and how they stack up against the four predictive insights that Engineering Managers require to make a confident decision to fast-track them to interview.

Traditional take-home exercise

What is it:

You send a list of requirements to the candidate (usually as a PDF) and ask them to write the best software that they can produce that meets those requirements.

The candidate will then attempt to model the requirements and then send back their source code for review.

Positives:
  • it is an actual proof of work
Negatives:
  • as the candidate’s codes “offline”, you have no idea how long it took them to complete. It could be 1hr or 100 hrs! This makes it impossible to make confident decisions or be able to compare their solutions versus other candidates
  • takes one of your internal developers at least 30 minutes to review the submission
  • the evaluation is subjective and it’s hard to achieve consistency across your team
Predictive insight ratings:
Development technique
UNKNOWN
Delivery pace
UNKNOWN
Code quality
HIGH
Testing strategy
MEDIUM
Key technologies
MEDIUM

Online programming puzzle

What is it:

You send the candidate a link to a remote system where they are asked to implement the solution to a simple problem (usually algorithmic). It is predominately used for screening.

Positives:
  • you get to see a code sample
  • the system automatically validates their submission, is objective
Negatives:
  • very limited insights. You are unable to prioritise the “best” candidates on this data
  • some Developers will not perform at their best using an online environment
Predictive insight ratings:
Development technique
UNKNOWN
Delivery pace
LOW
Code quality
LOW
Testing strategy
UNKNOWN
Key technologies
LOW

Hands-on pairing session

What is it:

You meet with a candidate (either F2F or online) and you work together on a piece of code.

Positives:
  • enables you to interact socially with the candidate
Negatives:
  • due to the time constraints, not much code is being written, so you cannot gauge the quality of their code or development speed
  • the validation is highly subjective and biased towards people who can interact well socially
Predictive insight ratings:
Development technique
MEDIUM
Delivery pace
LOW
Code quality
LOW
Testing strategy
MEDIUM
Key technologies
LOW

Questions during the face to face interview

What is it:

During the face to face interview, you ask the candidate targeted questions that are aimed to uncover more about the candidates' experience with a particular technology.

Positives:
  • you can really get into the minute details of one particular technology
  • the interviewer can control how much time it is spent covering a particular area
Negatives:
  • you need to ask the right questions otherwise you risk making an assessment on things that don't really matter for the success of the hire
  • guidance needs to be provided to interviewers in regard to what the acceptable answers are - and why
  • requires an experienced interviewer - being a social situation, the conversation can easily drift into areas of limited insights
  • it is difficult to achieve a repeatable and consistent outcome
Predictive insight ratings:
Development technique
LOW
Delivery pace
UNKNOWN
Code quality
UNKNOWN
Testing strategy
LOW
Key technologies
MEDIUM

Question via an online questionnaire

What is it:

The candidate is given a set of multiple questions targeting a particular technology or framework.

Sometimes the questions are programming puzzles where you are asked to figure out an answer.

Usually employed as a screening tool, pass or fail.

Positives:
  • can really get into the minute details of one particular technology
Negatives:
  • answers are often easily available online and they are usually part of the documentation for that particular piece of technology
  • the programming puzzles rely on very specific details about the technology and are not a predictor for programming ability
Predictive insight ratings:
Development technique
LOW
Delivery pace
UNKNOWN
Code quality
UNKNOWN
Testing strategy
LOW
Key technologies
LOW

Whiteboarding session

What is it:

During the face to face interview, together with the candidate, you sit in front of a whiteboard and sketch the design of a system.

It could be a production system, a CI pipeline, anything really.

The idea is to have meaningful conversations about software architectures and design trade-offs.

Positives:
  • can really dig deep into the candidates' architectural experience
  • provides insights into how they approach operational considerations
  • provides insights into the candidate's ability to explain complicated concepts
  • the interviewer can control how much time it is spent covering a particular area
Negatives:
  • It’s a high-risk strategy to do this with candidates if you have not already seen some of their code
  • due to time constraints, the conversation might drift into areas of limited insights
  • requires an experienced interviewer - being a social situation, the conversation can easily drift into areas of limited insights
Predictive insight ratings:
Development technique
MEDIUM
Delivery pace
UNKNOWN
Code quality
UNKNOWN
Testing strategy
MEDIUM
Key technologies
UNKNOWN

A new approach to assessing Developers

Rationale

The traditional way of assessing developers involves looking at a sample of their code. This shows you what they can produce, the end result.

With the advent of Extreme Programming and Test Driven Development, people started focusing more and more on how the code is produced rather than the end result.

The ideal coding assessment should provide insights into both what the candidate can produce and how.

Scenario based coding challenge

What is it:

Simulates a real-world project in a compressed format.

The candidate does this in their own time, using their laptop and tools.

The candidate is given a set of requirements that are changing over time.

Starts like a take-home assignment, but a system is put in place so that when one set of requirements is completed, another one is sent to the candidate. The system will also validate the submissions.

Once the challenge is complete, a developer will be asked to look at the candidate report which also contains the source code.

Positives:
  • identifies if the candidate consistently delivers quality code and good development technique
  • identifies if the candidate is a fast developer
  • provides insight into their testing strategy
  • enables you to accurately compare candidates
  • exposes the candidate's development technique
  • provides insights into the candidate's delivery speed
  • mimics real world development constraints
  • candidates perform at their best as they are using their own IDE and tools
Negatives:
  • requires candidates to have a full working development environment. If you are hiring juniors, they might struggle with this
  • format less suitable for testing experience with a particular framework
Predictive insight ratings:
Development technique
HIGH
Delivery pace
HIGH
Code quality
HIGH
Testing strategy
HIGH
Key technologies
LOW

Reengineering your hiring process

Optimise the process as a whole

The structure of a hiring process can be boiled down to this key question:

"What tests do you need and in which order, so that the process is optimal and so you can make confident hiring decisions?"

A common mistake is to add assessments steps on top of an existing process.
Don’t do that!

Each assessment step lengthens the recruitment process and you risk losing candidates.

When you make changes to your assessment you should always optimise the whole process to make sure that you obtain high quality predictive insights with the least number of assessment steps.

At the same time, the candidate experience has to be kept to a high level as we operate in a candidate-driven market.

Let’s look at such a process!

Hiring process
Engagement call
1. Sell the role, rather than trying to screen out candidates
Scenario-based challenge
2. Real-world challenge with deep insights into delivery capability
Face to face interview
3. Highly confident they can deliver, use the time to explore other areas
Offer

Step 1: Candidate Engagement Call

This is NOT a phone screening call*.

This is where a lot of great companies get it wrong.

When hiring top-notch engineers, you are not buying, you are selling. Your goal is to get the candidate excited about the role and your company.

This call “drives” the subsequent candidate behaviour. Candidates are often dubious that an actual role exists. By taking the time to explain the projects that they will be working on, the tech that they will be using and the company culture, you will have built trust and they will be far more engaged and incentivised to complete the scenario-based challenge.

*The call does provide you with the opportunity to assess a candidate's soft-skills.

Step 2: Scenario Based Coding Challenge

The candidate does this in their own time, using their own laptop and tools.

It is an actual proof of work and the purpose of this is to give you insights into their:

  • Development technique
  • Delivery pace
  • Quality of code
  • Testing strategy

The longer the coding challenge the more accurate the insights will be.

The sweet spot is between 2 to 4 hours. Long enough to get good quality insights, but short enough to be done during an evening.

Step 3: Non-technical F2F Interview

This is where everything comes together.

The insights provided by the scenario-based coding challenge will have already addressed a significant amount of the technical skills.

This frees you up to cover the other skills which can also make a difference for development roles.

By using a combination of:

  1. Code review the coding challenge
  2. Questions about key technologies
  3. Whiteboard session focused on software architecture

You can assess the candidates’:

  • ability to explain and present complex information
  • experience with key technologies or frameworks
  • understanding of infrastructure and deployments
  • architectural principles

For example, one approach is to carry out a whiteboarding exercise, where the candidate works together with you to design a system.

This approach gauges the candidate's systems thinking and engineering experience.

Step 4: The offer

By this point you will have a complete picture of the candidates’ skillset and have the confidence to make offers to who you now know are objectively the best candidates.

Conclusion

Scenario based coding challenges deliver high quality predictive insights within a short period of time and will dramatically improve the quality of your hiring decisions.

When deployed as part of a hiring process optimised for maximum insights in minimum time, i.e. removing unnecessary upfront screening calls, you can:

  • Quickly and objectively identify developers who have proven they can deliver
  • Expedite these developers to interview before your competitors
  • Significantly increase your interview to offer ratio
  • Reduce the risk of a bad hire due to poor coding skills

In case you are wondering, our Developer Insights challenge is a scenario based coding challenge.

If you want to learn more, we recommend that you read our case studies to find out about the phenomenal results our clients are achieving.