Writing a good technical test

Having previously defended technical tests, I started thinking about what makes a good tech test.  It's also made me review the tech tests I issue to follow these guidelines. Writing a good technical test comes down to the following:

• Clear directions
• Expectation of time
• Unrelated to existing business issues
• Degrees of difficulty
• Dogfooding

Clear Directions

The instructions for a technical test are a specification. You need to invest the time into creating a specification that is clear to someone picking it up.  Look at your technical test and see if it's something you would give to a member of your team to work on.  If it's not, change it.

If you're expecting someone to turn in some professional looking code, make sure they have a professional looking brief.  That's not to say it can't still be fun, or even slightly tongue-in-cheek, but be clear on the instructions.

These will also need to include any instructions to get the test up and running, if you require the candidate to build on a specific tech stack, or work with specific external software.

Expectations of time

Let the candidate know how long they are expected to spend on the task.  If I can see, up front, that I am expected to spend 20 hours on it, I'm not going to try.  If I know I'm only expected to spend an hour or two on it, great.

Time is a valuable commodity, and asking someone to spend half a working week (2.5 days) on a task is hugely unreasonable.  I'd go as far as to suggest that anything beyond half a day (4 hours) is unreasonable.  Would you bring them into the office for a full day interview session, where half of it is sat at a machine coding?  Probably not, so work back from there.

Unrelated to existing business issues

It's great if a candidate can get exposure to the code they are likely to be working on in advance, but don't send them your current code base with a real-life bug to fix. Doing so will make them feel that they are being used as free labour to fix an issue you have.

If you want to ask them how they would implement something within your current product, ask them in an actual interview.  Ideally for something you already have in place rather than something which is being planned.

Should you absolutely have to give them something to work on which is in the business domain, and based on existing code, make sure it works before you send it to them. Ideally it should have unit tests ready so they will know if something isn't working as expected. As you will already know the desired outcome, build the unit tests for the code they are required to write, so they can check that they have a working solution before returning it.

Degrees of Difficulty

A technical test is there to establish that someone can do what they say they can, but also give you a chance to evaluate their skills and abilities.  If you make the test too easy, some people may look better than they actually are.  If you make it too difficult, some good candidates may give up or fail before you've had a chance to evaluate them (this happens a lot).

Have a single test where you have basic requirements, more difficult requirements, and some advanced concepts. Each should be attainable at different ability levels within the same time scale, so as not to require a junior developer to only spend 30 minutes on it, but needing a senior developer to spend a day. The reality is the amount of time people are willing to spend on a technical test grows smaller and smaller as they progress through their career.

What a junior developer can achieve in 2 hours might only be 20 or 30 minutes to a senior developer, so they can spend time looking at the more challenging elements to showcase their abilities.

By having a technical test with different levels of difficulty, you can see at just what level someone is working to.  This can help further down the line when it comes to offering a remuneration package. Has someone applying for a junior role managed to get most of the way through what you'd expect a mid-level developer to be able to do?  Get them in and offer them more than they are expecting.  Are they applying for a mid-level role, but not quite getting to the level you expect?  Offer them less if they show the right potential, but let them know where they need to improve to get a higher salary.

Dogfooding

Now you've written your technical test and given it an expectation of time and difficulty, try it out.  Run through it yourself to see how long it takes you.  You know what you are looking for from the candidate, so take those 2-4 hours you're expecting them to take, and see if you can do it in that time.  All of it, not just the easy part.

Once you've managed to do that, and refined the test so you can do it with time to spare, try it on your team.  Give them the test to see what they can achieve in the same time you're expecting a candidate to perform it in.  Does this match up to their level in the organisation?  If not, you might need to change it again.  If they surpass your expectations, then maybe they are due a pay rise.