What I’ve Learned About the Pitfalls of Hiring After Holding 1000 Interviews (2023)

Setting up and managing a great interview process that outputs strong talent is crucial.

This is essential for your startup or projects because it is how you get smart, talented people to join your team.

This article is the second of a two-part series where I share my experience after holding over 1000 interviews in startups and big companies. If you have already read the first part, thank you! Hopefully, you will find the second part valuable. You can find the first blog post here if you haven’t read it.

Short recap, part 1 was focused on:

• the interview process
• what to look for in candidates and
• things to keep in mind before, during, and after the interview.

In part 2, the focus is on the following:

• common pitfalls
• what was my biggest challenge, and most likely yours and
• the million-dollar question: “How do I know if a hire will work out or not?”

Let’s get to it.

Let’s go through a non-exhaustive collection of pitfalls I also ran into and observed in interview panels or while shadowing interviewers.

Bias

There are multiple biases that can occur during the interview process. The most common ones that you have to keep an eye out for are:

• Affinity bias

We tend to have a preference for people with characteristics or attributes like ourselves. These could be past jobs, companies, education, and so on. If somebody just graduated from a prestigious school, it does not mean they would be a good fit. The reverse is also valid.

• Confirmation bias

This translates into deciding that the candidate is a good fit for the job before the interview and then trying to confirm your assumptions during and after. For example, maybe you know that the candidate went to company X, so the candidate has to be a stellar performer, or that you know somebody that has worked with them in a previous job that’s also a great hire.

• Halo effect

This could mean focusing too much on just one good/bad example or situation. You will have to look for some consistency in the candidate’s past performance or through the interview. If they missed one answer from a list of 10, it should not cast a shadow on the entire performance. The other way around is that they get most answers wrong but get the last one right.

• Anchoring effect

This one ties in a bit with the Halo effect, anchoring on just one piece of information to make a decision, be it a Yes or a No.

• Follow the crowd

This happens mostly in debriefing interview panel sessions, where you change your opinion based on what the room says. That is why it is very important to take notes, document your decision and submit them before talking to other interviewers from the panel.

• Ignoring situational circumstances

Ignoring circumstances of why the candidate did X. Usually, I see this working more on the negative side; if a candidate didn’t do X that they should have done in accordance with industry-wide engineering best practices, then they are not a good fit. Not getting the full context before making a judgment may land you turning down good candidates.

How to reduce biases

The best thing you can do is to be aware of the possible biases and refresh them just before the interview. I look over a quick checklist I have in a draft interview template and try to do a mental refresh as much as possible.

If you have a prepared list of questions and are familiar with all the biases, document and take notes of the interview so you can review it afterward to reduce the chances of being biased. If you can set up an interview panel even better, just set some ground rules for everyone: take notes, make a decision as close to the interview as possible, submit it before the debrief, make everyone aware of these common biases, etc. This will keep the biases in the group to a minimum and land you with better hiring decisions.

Not leading the interview

While candidates can speak for hours on a specific topic, they may not speak on things that interest you or give you the answers you are searching for. It is on you to guide the interview and get the answers you need to decide whether it’s a Yes or a No.

If after the interview, the answer is maybe, you need to get better at asking the right questions, guiding candidates through the interview, and sometimes interrupting them when they go off course or on details that are not of interest for your assessment.

Probing for exact technology knowledge

Hiring for a specific technology or job can backfire. There are some exceptions to this, but usually, you want a generalist that can easily learn new technologies. The technology space evolves very quickly. What you use today may change tomorrow; you will want this flexibility and skill in your workforce.

Expecting to demonstrate specific behaviors

Let’s say you work in a big technology product company where you are looking for a tech lead that can push back to product and avoid tech debt as much as possible. Somebody that never will cross the boundaries of releasing something that is not 100% unit test code coverage in production.

You may ask them the following question: have you ever been in a situation where you had to push back to product on a specific launch? (most likely, you are looking for an answer where descoping, sequencing the project, or other tactics were employed). You have to be careful with the candidate’s previous context. If the candidate was in an outsourcing company, they most likely didn’t have this luxury, and everybody did whatever they could to keep the client happy and the contracts flowing so they wouldn’t lose their jobs.

If somebody didn’t do something in the past, it does not mean that they will not do it in the future if given the opportunity.

Not understanding how to hire on potential

The most common mistake I have seen in this category is in how to assess potential. Many just probe if a candidate has potential but don’t look at the whole picture: Does the candidate have potential, given the opportunity you have? For me, hiring on potential means somebody smart who wants to learn and advance their career with the opportunity I can offer them.

The candidate may be smart, but if they are not a fit for the opportunity you have and they are not motivated to grow, it will not work out. It will be a shame because it will be wasted potential at that point.

To successfully hire on potential, you need an intelligent candidate who has proven they can learn fast and is motivated by the opportunity you give them. I have seen smart and determined junior engineers that, in a few years, outperformed more senior individuals. Hiring on potential is excellent when done right.

Not balancing seniority on teams

It depends on your project and company and how your current team looks, but you should strive for a diverse team from a seniority perspective.

In most cases, when it comes to a delivery perspective, a team with junior engineers hungry for knowledge and some strong senior folks outperforms a team solely comprised of senior engineers.

Hiring for culture fit and not culture add

We covered this already in part 1. Looking for just culture fit may land you in a place where you don’t have a very diverse team. Keep an eye open for this; a good start is contemplating the question:

“Am I hiring just for culture fit and not for culture add?”

Not accepting any errors in the interview

Candidates that pass the interview do not always perform their best in all modules. One of the reasons why there are multiple interview modules that probe for overlapping skills is that candidates may miss answering questions in one module but demonstrate that skill in another. The entire scope of the interview is for the candidate to demonstrate if they possess specific skills.

I usually probe for how the candidate approaches problems, what questions they ask, whether they know the fundamentals, etc.

The candidate may not fully solve each problem I give them, but if they advanced enough, they might still get a Yes from me.

Being very harsh or having unrealistic expectations

I have seen new interviewers being very harsh in their feedback, mainly because they want to do a good job and hold the bar high but are not yet calibrated.

• An interviewer at debrief: I asked question X, and the candidate answered eventually but took too much time to come up with the solution.
• Me: How much time?
• Interviewer: About 2 minutes, but it was something they should have known from university. It should have been an instant answer..”
• Me: Yes, but the candidate has 8 years of industry experience. They might not have the red/black trees algo that fresh in their memory…”

Usually, it takes a couple of sessions for new interviewers to get calibrated. I suggest having some interview sessions where new interviewers shadow more senior ones or more senior interviewers that shadow them during interviews. This way, they will learn how to conduct the interview, what expectations to have, and how to grade candidates while not passing on some good people in the pipeline.

Not trying to help candidates out

Sometimes candidates get stuck. It’s a reality. In an interview situation, candidates lose a couple of IQ points due to stress.

It happened to me once, back in my university years. I was interviewing with a company and was asked an algorithm question. Right after that, my neighbor decided it was a good idea to start remodeling and started drilling in the wall. Thank you, neighbor! I was thrown off my game. I was stuck. My interviewer tried to unblock me once or twice but eventually gave up. After the interview finished, 2 minutes later, I had the solution to the problem.

You should offer candidates hints and continue to guide the interview. For example, if they are nervous, try to calm them down, get them back on track, offer hints, and see how the interaction progresses.

There is a saying:

Leave no candidate behind

Cutting the interview short

If you have X time with the candidate, you have to stay with the candidate for the full X period. Under no circumstance should you cut the interview short (there are a couple of exceptions here, the candidate being very aggressive and posing a threat to themselves or you).

If you cut the interview short, this may result in a terrible candidate experience. You might also miss some good candidates that were just a bit stuck and needed more hinting (wink to my former interviewer; I hope you are reading this.)

Letting them know how they have done at the end

This is more problematic if there is an interview panel but also if you are the only one holding the interview and the decision lies with only you. You should say to the candidate that you will follow up with the results in the next X days. You can and should give more visibility on the next steps and how the decision will be made but don’t give them an answer on the spot.

Having more time after the interview to reflect on your notes and check your biases will help you make a better decision than in mid-interview.

Very strange, I thought that my biggest challenge was starting up in the interview process. Things that I thought were tough were things like what questions to ask, how to structure the interview, how to grade candidates, and so on. After several interviews, I thought things would be more straightforward, and no new challenges would arise. I was wrong.

My biggest challenge was keeping my biases in check once I reached the 100-interview mark.

Our brain tries to automate decisions for us as much as possible. So after 100 interviews, or a decent amount of them, your brain will look for patterns, and you can become cocky and think you know it all. You believe you can decide if a candidate is good or not in the first 15 minutes or, even worse, 30 seconds.

If you are not careful, you may end up with a not-so-diverse team where everyone shares your strengths but also your weaknesses. Another issue is that you will start making the mistakes you avoided in the early days when you were paying more attention to everything and your mind was not on autopilot.

You have to battle your inner bias demons and your brain’s nature. Each candidate has 1 hour to convince you if they are a yes or a no. Try to give them the full hour. Stick with preparing an interview question list before, take notes during the interview, and be aware of these biases before every interview. Be humble! This should keep you out of trouble.

The answer is you don’t, and that’s the tricky part. This is the million-dollar question, and it really can be a million-dollar question because it can make or break your startup, team, or projects.

Hiring is not an exact science, but there are some principles that you can use to guide yourself.

1. If it’s not a Hell Yes! and it’s a Maybe, request another interview and probe for the missing pieces that you need. It is best to avoid a bad hire for both parties.
2. Be honest with the candidate. Try to present the candidate with how things are; if sometimes long hours are needed before a release (the case for most startups), tell them that. Hiring needs to be a fit from both sides.
3. If the person is not a great fit for the company or is a smartass, you should pass. I had my share of passes on very strong engineers that seemed very hard to work with, and that is ok; you should do the same.

Ultimately, what matters to me is the answer to my question:

Is the candidate smart, gets things done, and a good culture add?

If the answer is yes, I try to set the candidate up for success the best I can.

If I have missing information, I may request another interview session. If I have all the information I need and I am not certain, most likely, my answer is no.