4 Considerations When Building a Best-In-Class Technical Assessment

As organizations across industries expand their need for talented software developers, recruiting teams require a scalable way to evaluate technical talent. One of the most effective, validated ways to do this is the technical assessment.

Technical assessments provide a way to evaluate a candidate’s ability to code, consistently and objectively. They are a crucial tool to ensure organizations hire candidates with the right skills to succeed.

In this article, I’ll examine the best practices behind world-class technical assessments, particularly those that expertly assess candidates from all learning backgrounds.

1) Simple vs. Complex Coding Challenges

A common question we often receive when building the coding challenges that make up technical assessments is “Should hiring teams use simple challenges or complex challenges?”

Our recommended approach, which is light touch and high-value for screening candidates, is to use a relatively simple coding challenge in a specific programming language that you know is a requirement for the job.

Keep in mind that a “simple” coding challenge might look different depending on if developer candidates are experienced or entry level. Entry-level developers, because they need to have the basic skills and ability to learn on the job, should be evaluated with the simplest challenges. You don't need to send all your candidates down a wormhole into a three-hour programming task to ensure that they can be an effective coder.

A better approach is to use the coding challenge to confirm that they have some base competency in the relevant programming language. Using more complex challenges becomes a slippery slope because it can favor candidates that have a formal Computer Science education over those that are self-taught, have fantastic practical skills, or have stellar soft skills and the ability to ramp quickly.

A side note for hiring experienced developers: More experienced developers can have a more complex coding challenge and follow-up questions (more on this in consideration #3), but it shouldn’t take too long, or it may lead to dropout. Experienced candidates generally expect to have a conversation first, to understand what to look for and how they can prepare. An invitation to take a technical assessment should not be the first communication they receive. Depending on the scenario, it may be more beneficial to skip the technical assessments, and go straight to a live coding session with the technical hiring team.

2) Gauge Candidates’ Understanding and Soft Skills with Video Interview Questions

If you’re using HireVue’s technical hiring solution, you can go beyond this. Using a follow-up recorded video response to a question turns the coding challenge into a conversation that gets candidates talking about:

• The choices they made;
• The approaches they took in solving the challenge;
• The complications they overcame; and
• The solution alternatives that they considered.

These are far more interesting than optimizing an academically obscure algorithm as a way to confirm that they attended a formal computer science program (as opposed to solving problems in the real world).

You can also use video interview questions to evaluate candidates’ job-relevant soft skills. Situational judgment, scenario-based, and other behavioral-type questions yield insight into:

• How candidates work in teams;
• How candidates respond to feedback;
• How candidates explain their understanding and thought processes to managers; and
• Other critical soft skills.

Most developers spend less than half their day coding. If you aren’t testing for soft skills like teamwork, communication, and collaboration, your technical assessment isn’t assessing for the reality of modern dev work.

3) Validate Knowledge with Additional Scored Questions

Coding challenges and recorded interview questions are not the only type of question to use in an interview for an assessment of the candidate's domain of knowledge.

Additional scored, multiple-choice questions can validate candidates’ knowledge of frameworks, tool kits, or architectural topics that are relevant to the job. These typically are difficult to assess in a coding challenge, and would greatly lengthen the assessment if they were.

Combining multiple choice-type knowledge questions with validation of basic programming competency is a powerful combination that vets relevant knowledge. Knowledge questions provide a way to vet the high-level proficiency of both entry-level and experienced developers without requiring a significant time investment on their part.

4) Deterring and Detecting Cheaters

Computer programmers are constantly solving a vast array of technical problems. Often, there are multitudes of solutions posted online for any given programming problem and no particular solution is the best for all potential applications of that code.

Unfortunately, this means that coding challenges and their respective solutions are frequently posted online. Technical hiring teams could be presented with a slate of the best cheaters, not the best developers.

Best-in-class technical assessments offer several ways to check the uniqueness and authenticity of a candidates’ code.

How HireVue’s Technical Assessments Empower Hiring Teams, While Detecting Cheaters

HireVue’s CodeVue technical assessment solution is built to empower technical recruiters and hiring teams and detect cheaters. We built it so non-technical recruiters could make informed, qualified screening decisions on technical candidates.

This is our approach:

Deter Cheating Before It Happens

One of the best ways to avoid cheating is to deter it before it happens. CodeVue offers several functionalities that empower hiring teams to deter cheaters:

• Question Rotation. Several different challenges are available for challenges at each level of proficiency. CodeVue automatically rotates through these, so candidates get a different challenge than other candidates in the pool.
• Custom Challenges. When a custom CodeVue challenge is created, it is specific to an individual organization. This makes it more difficult for candidates to find the exact challenge solution on the internet because it has only been solved by the small number of candidates who are applying to the specific role at the same organization.
• Special Instructions. Special instructions can be given to the candidate during their CodeVue interview to help evaluators understand the process used to reach the solution. These instructions can include things like asking the candidate to build the code within the HireVue platform or discouraging copying and pasting. The paste function within the HireVue platform can be disabled during the CodeVue challenge, and the playback tool can be used to see if a candidate deviated from the instructions provided.

Automated Cheating Detection

CodeVue uses Measure of Software Similarity Scoring (“MOSS”) algorithm scoring to automatically flag candidates who likely cheated.

MOSS algorithm scoring is a widely recognized way to compare electronic documents and identify which ones have an unusual amount of similarity between them. CodeVue uses MOSS scoring to identify when a submission is similar to other candidate submissions for the same challenge. Although high code similarity is an indication of suspicious activity, coupling this with the code editor playback tools and follow up video questions are a good way to really understand if the candidate has cheated.

Comprehensive Data Around a Candidates’ Thought Process

CodeVue also provides additional data points which can be used to gauge candidates’ thought processes.

If you want to take a deeper look at how a shortlist of candidates came to their solutions - and vet if they have cheated - the Code Editor Playback functionality allows you to play back a recording of the candidates coding session. You can watch a candidate’s keystrokes to see how they worked out the code and observe exactly how they developed their solution in the code editor. You can also speed up the playback up to 32x the actual speed, and skip through periods of inactivity, so you don’t have to sit through lengthy solutions as they are coded.

Code Editor Events are displayed as part of the playback to help you see key actions that the candidate took during their coding session:

• Paste. Paste allows you to jump to events where candidates have pasted code. It also outlines how many characters have been pasted in that event.
• Loss of Browser Focus. This allows the reviewer to see when candidates have navigated to a different browser or application (eg, away from the code editor).
• Inactivity Detection. This identifies periods of time where the candidate is not actively coding, which is a great way to skip times when the candidate is not actively working on the solution
• Compile Errors. This event marks times during the candidate’s session when they ran the code and it failed to execute due to syntax or other language errors.
• Test Runs. Test Runs denotes when during the coding session the tests were run against their code and what the results were.

Another method to identify potential cheaters is via Time Tracking. With time tracking, we capture how much time was taken for the candidate to produce their answer.

It's a pretty simple litmus test. Ask yourself when evaluating the candidate's submission: “Is this a realistic amount of time for them to have completed this on their own?” If a typical candidate solves the challenge in 15 minutes and one candidate submits a solution in under 5 minutes, there is a high probability they have cheated.

Another data point we alluded to previously: the ability to create Follow-Up Video Questions. Adding a question to the interview immediately following the coding challenge is a great way to get the candidate to talk about their solution. A three minute video response from the candidate is plenty of time to validate their understanding.

All of these tools are powerful ways to understand if a candidate’s solution is suspicious. And because you do not need to know code in order to identify suspicious activity, these resources allow non-technical recruiters to quickly disposition candidates who have cheated and flag suspicious activity before they pass the candidates over to the technical team for a deeper review.

Empower Non-Technical Recruiting Teams

Altogether, these functionalities empower recruiters to make qualified screening decisions on technical candidates. Rather than submit candidates to hiring managers based on a resume review and phone screen, recruiters gain deep insight into candidates’ coding proficiency. They can give the most qualified candidates an expedited, high-touch hiring experience, while freeing up time for managers to focus on the best candidates for their teams.

Learn how Sabre nearly doubled the quality of their early-level technical hires.

Read How Sabre Doubled Quality of Hire for Technical Roles

Read their blog.

About the Author:

Scott Keller is HireVue's CodeVue Product Champion. As product champion for the CodeVue technical hiring solution, he consults Recruiters, Technical Hiring Managers, and HireVue’s Product Engineering Group on how to recognize and retain talented software developers. Prior to working at HireVue, Scott was Senior Architect and Consultant at Navitaire, where he worked with development teams across the globe on building custom back office solutions, earned data processing patents in 3 countries, and invented the Temporal Tables model of the ANSI-SQL 2011 standard. Find him on LinkedIn.