How SHL Handles Uncompilable Code in Coding Skill Assessments

If you have ever taken a coding test for a job on a machine, you probably frown if you could not make your code compile. Your program might be almost right, but due to some silly bug, unidentified in a small-time frame, you will get a ZERO score. According to SHL’s Tech Talent report, on average, around 25% of candidates in a hiring pool have strong coding skills, but often get sorted out due to the limited capability of traditional coding skill assessment platforms—a missed opportunity.

But fret not! SHL’s AI team has created a technology that can detect how good the program’s algorithm is, even if it does not compile. Our coding skill assessment platform can understand uncompilable programs using Artificial Intelligence and provides semantic feedback to improve.

 
Uncompilable code is not equivalent to a zero score in the coding skill assessment

How do we do it? First, we can fix some of the codes using Artificial Intelligence. By looking at patterns in good compilable codes, our algorithms minimally modify existing programs to make them compilable. And by using this approach, we can compile about 40% of uncompilable codes. Once compilable, our patented machine learning-based algorithm can generate a grade that mimics human raters.

Fancy as it may seem, we had a harder problem to solve.

What about the codes that do not compile? The remaining 60% of the codes go through smart static analysis and derive features, and signatures of the logic of the program automatically. With these features and a customized form of our machine learning algorithm, we provide grades as accurate as you could think!

Where does it make a difference? Let’s look at some real data. On a set of programs attempted for a job in a large e-commerce player in the USA, we find that 46% of codes were not compiling but were not blank.

Our AI-based algorithm found that 6% of these codes, for 596 candidates, had nearly correct logic. Another 29% of candidates, with a little bit of guidance, would have reached the right logic. All these candidates deserved a shot with the company!

By looking at patterns in good compilable codes, our algorithms minimally modify existing programs to make them compilable. And by using this approach, we can compile about 40% of uncompilable codes.

This system was also used by a large IT company in India for the large-scale hiring of entry-level software engineers. The company hires thousands of software engineers every year and struggles with filling all open positions. By way of grading non-compilable codes, they have been able to improve hiring throughput by 30-40%.

More recently for a technology giant in China, the system helped in selecting around 26% more candidates for the interviews. These candidates had written a logically meaningful code, but the code did not compile. Out of these candidates, around 19% of candidates were finally hired (detailed analysis in Table 1). This is a big win as many worthy candidates who would have been missed out by traditional program grading systems were hired.

 What is more? Our AI algorithm provides feedback to all candidates whose code does not compile. To some, we can tell how to fix their programs and make them compilable. To all, we can give them feedback on their algorithmic approach, tips to reach the correct logic, and provide feedback on the stylistic and maintainability issues in their code.

Disappointed with coding skill assessment platforms which give everyone a poor score and no feedback… We have corrected this for all times to come!

Stay tuned for more to come as SHL’s AI team continues to work on advancing the state-of-the-art coding skill assessment and AI. 

If you want to see SHL’s coding skill assessment in action, please book a demo with one of our product specialists. Also, check out our case studies to learn how we helped our clients!