In our experience, software engineers spend less than 5% of their time using anything that resembles your typical white-boarding programming challenge. In this post we make the case that programming challenges are an anti-pattern and that they became standard practice not because they are particularly good at assessing engineering talent but because they shift the burden from the interviewer/company to the candidate.

What is a programming challenge?

A programming challenge is an exercise meant to assess programming competence. An example might be “Write a program that prints out a multiplication table of the first 10 prime numbers”.

They range in complexity from trivially easy to wildly obtuse and abstract, which makes them all the more difficult to plan for. Most of them have little to do with real world programming and they are almost universally hated by most everyone.

Wait, what does a Software Engineer do?

I’ve asserted that Software Engineers spend less than 5% of their time programming. It is probably valuable to discuss then what Software Engineers are doing with the other 95% of their time. The best way I can come up with to convey this distribution of skills is with a couple examples.

Example 1

Lucy works for Widgets, Inc. as a Software Engineer. She is on the team currently building the first version of Widgets Marketplace, a new section of the Widgets, Inc. website that allows suppliers to sell widgety things to consumers, much like Etsy or Ebay. The team consists of a Product Manager, Designer, and two other Engineers.

Lucy collaborates with the other engineers, PM, and Designer to develop potential customer flows: onboarding, inventory management, and order fulfillment. The engineers are present to make sure the workflows described can be completed in a timely fashion. Lucy helps expand those workflows into actionable tasks for the engineering team. With a queue full of high level tasks, the engineering and design teams set out to build.

While engineers are sorting through their tasks and planning how to build, Lucy is setting up the initial codebase. She has generated a Ruby on Rails application and setup the database schema so the other engineers can hit the ground running. She sets up the test suite and creates several simple example tests so other engineers can easily copy them.

Lucy and the Designer review the experience as they build to make sure the experience is seamless for customers. Engineers consult Lucy on a particularly nasty database query. Lucy conducts code reviews to make sure they can easily maintain what they build. When her team surfaces a roadblock, Lucy recommends and adds to the project a background processing library that circumvents the problem. When Lucy discovers a nasty bit of code for communicating with an API, she takes it upon herself to write a clean client library that can be extracted out of the code and maintained separately so as to keep the Marketplace project modular.

Lucy’s exposure to a variety of problems of the years has given her a set of solutions and patterns for solving common programming problems.

Example 2

One month after the launch of the Widgets Marketplace, a number of feature requests and bugs are beginning to pile up. The team works together to prioritize the tasks.

Lucy consults the other engineers and they decide that fixing the bugs should take 2–4 weeks. Feature requests are moved to the backlog and the bugs are queued up for the next sprint. During the next sprint, Lucy debugs a bad database query while the two other engineers work through some javascript bugs in the UI. After Lucy finishes, she helps one engineer find the source of the UI bug — a javascript variable that was scoped incorrectly.

Solutions to the bugs are peer-reviewed by the team and merged/deployed in a way that allows easy rollback in case there were any unforeseen consequences.

If you’re following along then you’re probably developing a bit of an incongruence in your head. The typical software engineer workflow doesn’t sound anything at all like the programming challenge example listed above. At all.

The programming challenge is an anti-pattern, a failed attempt at distilling “engineering” into a 30–45 minute exercise.

Software Engineering consists of understanding a problem, building meaningful abstractions, designing clean interfaces, requirements management, test discipline, maintenance planning, environment constraints, and respectful collaboration with peers, most of which is not assessed during a programming challenge.

Our advice? Drop the programming challenge and instead pair one of your engineers with the candidate for 3 hours on a real bug or feature. Introduce them to stakeholders if necessary. Assess how they communicate their strategy and understanding of the problem. Watch how they begin to architect a solution. Do they start with set of tests or do they dive right in?

And it will take more time, but better filtering at the top of funnel should free up time for a longer assessment. This should give you a much better idea of their talent as a Software Engineer.

For business insights, news, events, and product announcements...