An Elegant Puzzle- Systems of Engineering Management

by Will Larson

  Almost every unkind interviewer I’ve worked with has been either suffering from interview burnout after doing many interviews per week for many months or has been busy with other work to the extent that they have started to view interviews as a burden rather than a contribution. To fix that, give them an interview sabbatical for a month or two, and make sure that their overall workload is sustainable before moving them back into the interview rotation.

  Identifying interview burnout is also one of the areas where having a strong relationship with open communication between engineering managers and recruiters is important. Having two sets of eyes looking for these signals helps.

  6.2.2 What role is this, anyway?

  The second critical step toward an effective interview loop is ensuring that everyone agrees on the role they are interviewing for, and what extent of which skills that role will require.

  For some roles—especially roles that vary significantly between companies, like engineering managers, product managers, or architects—this is the primary failure mode for interviews, and preventing it requires reinforcing expectations during every candidate debrief in order to ensure interviewers are “calibrated.”

  I’ve found that agreeing on the expected skills for a given role can be far harder than anyone anticipates, and it can require spending significant time with your interviewers to agree on what the role requires. (This is often in the context of what extent and kind of programming experience is needed in engineering management, DevOps, and data science roles.)

  6.2.3 Finding signal

  After you’ve broken the role down into a certain set of skills and requirements, the next step is to break your interview loop into a series of interview slots that together cover all of those signals. Typically, each skill is covered by two different interviewers to create some redundancy in signal detection, in case one of the interviews doesn’t go cleanly.

  Just identifying the signals you want is only half of the battle, though; you also need to make sure that the interviewer and the interview format actually expose that signal. It really depends on the signal you’re looking for, but a few of the interview formats that I’ve found very effective are:

  Prepared presentations on a topic. Instead of asking the candidate to explain some architecture on the spur of the moment, give them a warning before the interview that you’ll ask them to talk about a given topic for 30 minutes, which is a closer approximation of what they’d be doing on the job.

  Debugging or extending an existing codebase on a laptop (ideally on their laptop). This is much more akin to the day-to-day work of development than writing an algorithm on the board. A great problem can involve algorithmic components without coming across as a pointless algorithmic question. (One company I spoke with had me implement a full-stack auto-suggest feature for a search inbox, which required implementing a prefix tree, but the interviewer avoided framing it as yet another algo question.)

  Giving demos of an existing product or feature (ideally the one they’d be working on.) This task helps them learn more about your product and get a sense of whether they have interest around what you’re doing, and it helps you get a sense of how they deliver feedback and criticism.

  Roleplays (operating off a script that describes the situation.) This option can be pretty effective if you can get the interviewers to buy into it, allowing you to get the candidate to create more realistic behavior (architecting a system together, giving feedback on poor performance, running a client meeting, and so on).

  More than saying that you should specifically try these four approaches (but you should!), the key point is to keep trying new and different approaches that improve your chance of finding signal from different candidates.

  6.2.4 Be prepared

  If you know the role you’re interviewing for and know the signal which your interview slot is listening for, then the next step is showing up prepared to find that signal. Being unprepared is, in my opinion, the cardinal interview sin, because it shows a disinterest in the candidate’s time, your team’s time, and your own time. When I recall the fortunately rare situations when I’ve been interviewed by someone who was both rude and unprepared, I still remember the unprepared part first and the rude part second.

  I’ve also come to believe that interview preparedness is much more company-dependent than it is individual-dependent. Companies that train interviewers (more on that below), prioritize interviewing, and maintain a survivable interview-per-week load tend to do very well, and other companies just don’t.

  Following from this, if you find that your interviewers are typically unprepared, it’s probably a structural problem for you to iterate on improving and not a personal failing of your interviewers.

  6.2.5 Deliberately express interest

  Make sure your candidates know that you’re excited about them.

  I first encountered this idea reading Rands’s “Wanted” article,4 and he does an excellent job of covering it there. The remarkable thing is how few companies and teams do this intentionally: in my last interviewing process, three of the companies I spoke with expressed interest exceptionally well, and those three companies ended up being the ones I engaged with seriously.

  Whenever you extend an offer to a candidate, have every interviewer send a note to them saying that they enjoyed the interview. (Compliment rules apply: more detailed explanations are much more meaningful.) At that point, as an interviewer, it can be easy to want to get back to your “real job,” but resist the temptation to quit closing just before you close: it’s a very powerful human experience to receive a dozen positive emails when you’re pondering if you should accept a job offer.

  6.2.6 Feedback loops

  Interviewing is not a natural experience for anyone involved. With intentional practice you’ll slowly get better, but it’s also easy to pick up poor interviewing habits (asking brainteaser questions) or keep using older techniques (whiteboard coding). As mentioned earlier, even great interviewers can become poor when experiencing interview burnout or when they are overloaded with other work.

  The fix for all these issues is to ensure that you build feedback loops into your process, both for the interviewers and for the designer of the interview process. Analytics (discussed in the next section) are powerful for identifying broad issues, but pair interviews, practice interviews, and weekly sync-ups between everyone strategically involved in recruiting (depending on your company’s structure, this might be recruiters and engineering managers, or something else) work best to actively improve your process.

  For pair interviews, have a new interviewer (even if they are experienced somewhere else!) start by observing a more experienced interviewer for a few sessions, and then gradually take on more of the interview until eventually the more senior candidate is doing the observing. Since your goal is to create a consistent experience for your candidates, this is as important for new hires who are experienced interviewing elsewhere as it is for a new college grad.

  To get the full benefit of calibration and feedback, after the interview have each interviewer write up their candidate feedback independently before the two discuss the interview and candidate together. Generally, I’m against kibitzing about a candidate before the group debrief, in order to reduce bias in later interviews based on an earlier one, but I think this is a reasonable exception since you’ve experienced the same interview together. Also, in a certain sense, calibrating on interviewing at your company is about having a consistent bias in how you view candidates, independently of who on your team interviews them.

  Beyond the interviewers getting feedback, it’s also critical that the person who owns or designs the interview loop get feedback. The best place to get that is from the candidate and from the interview debrief session.

  For direct feedback from candidates, I’ve started to ask every candidate during my “manager interview” sessions how the process has been and what we could do to improve. The feedback is typically surprisingly candid, although man
y candidates aren’t really prepared to answer the question after five hours of interviews. (It’s easy to get into the mode of surviving the interviews rather than thinking critically about the process that is being used to evaluate you.) The other, more common, mechanism is to have the recruiters do a casual debrief with each candidate at the end of the day.

  Both of these mechanisms are tricky because candidates are often exhausted and the power dynamics of interviewing work against honest feedback. Maybe we should start proactively asking every candidate to fill out an anonymous Glassdoor review on their interview experience. That said, this is definitely a place where starting to collect some feedback is more important than trying to get it perfect in the first pass, so start collecting something and go from there.

  6.2.7 Optimize the funnel

  Once you have the basics down, the final step of building a process that remains healthy for the long haul is instrumenting the process at each phase (sourcing, phone screens, take-home tests, onsites, offers, and so on) and monitoring those metrics over time. If your ratio of sourced referrals to direct ones goes down, then you probably have a problem (specifically, probably a morale problem in your existing team). And if your acceptance rate goes down, then perhaps your offers are not high enough, but it also might be that your best interviewer has burned out on interviewing and is pushing people away.

  Keep watching these numbers and listening to candidate post-process feedback, and you can go to sleep at night knowing that the process is still on the rails.

  As a side note: I’ve put optimizing your funnel—and by this I include the entire process of building explicit analytics around your process—as the last priority in building a great interviewing process. From a typical optimization perspective, you should always measure first and optimize second, and here I’m giving the opposite advice.

  Doing this first instead of last is definitely reasonable. In fact, I considered making this the first priority, and when I was setting up my last hiring process it was the first thing I did. In the end, I think you’ll find that your process cannot thrive without handling the first six priorities, and that your analytics will direct you toward fixing those issues. Plus, the underlying data is very often poor, and it can be easy to get lost spending your cycles on instrumenting your process instead of improving it.

  Altogether, the most important aspect to interviewing well is to budget enough time and to maintain a healthy skepticism about the efficiency of your current process. Keep iterating forward and your process will end up being a good one.

  Figure 6.2

  The phases of a hiring funnel.

  6.3 Cold sourcing: hire someone you don’t know

  The three biggest sources of candidates are referrals from your existing team, inbound applicants who apply on your jobs page, and sourced candidates whom you proactively bring into your funnel.

  Small companies tend to rely on referrals, and large companies tend to rely on sourcing candidates, using dedicated recruiting sourcers for whom this is their full-time job (often the first rung on a recruiter career ladder). Medium-sized companies are somewhere on the continuum between those extremes. (Slack, in particular, has done some interesting work to encourage inbound applicants, and while they’re now getting to the size at which most companies depend more heavily on sourced and direct applicants, I suspect that inbound applicants are not their largest source of candidates.)

  Hiring and recruiting teams tend to prefer referrals because they often have higher pass and accept rates, and most early-stage companies, especially those without dedicated recruiting functions, end up being primarily composed of referrals. (An interesting caveat: lately I’m seeing more of a second category of referral candidate: those who are running their own extremely systematic interviewing process, with the aim of getting offers from three or more companies. Consequently, these folks tend to have much lower acceptance rates.)

  Figure 6.3

  Small companies rely most on referrals, and larger companies on sourced candidates.

  Referrals come with two major drawbacks.

  The first is that your personal network will always be quite small, especially when you consider the total candidate pool. This is especially true early in your career, but it’s easy to work for a long time without building up a large personal network if you work in a smaller market or at a series of small companies. (One of the side benefits of working at a large company early in your career, beyond name recognition, is kickstarting your personal network.)

  The other issue is that folks tend to have relatively uniform networks, composed of the individuals they went to school with or worked with. By hiring within those circles, it’s easy to end up with a company whose employees think, believe, and sometimes even look similar.

  6.3.1 Moving beyond your personal networks

  Many hiring managers freeze up when their referral network starts to dry up, or as they look to bring a wider set of backgrounds onto their teams. However the good news is that there is a simple answer: cold sourcing. Cold sourcing, a technique that’s also common in some kinds of sales, is reaching out directly to people you don’t know.

  Figure 6.4

  Small personal networks relative to total pool.

  If you’re introverted, this will probably be an extremely unsettling experience at first, with questions like “What if they’re annoyed by my email?” and “What if I’m wasting their time?” ringing in your head. These are important questions, and we have an obligation to be thoughtful about how we inject ourselves into others’ lives. I was personally paralyzed by this concern initially, but ultimately I think it was unfounded: a concise, thoughtful invitation to discuss a job opportunity is an opportunity, not an infringement, especially for those who are on career networking sites like LinkedIn. Most folks will ignore you (which is great), others will politely demur (also great), a few will actually respond (even greater), and a surprising number will ignore you for six months and then pop up mentioning that they’re starting a new job search. I’ve never had someone respond unkindly.

  The other great thing about cold sourcing is that it’s pretty straightforward. I’ll share the approach I’ve used, with the caveat that I believe there are a tremendous number of different approaches that are probably more effective. Take this as a good starting point, track your results, and then experiment!

  Figure 6.5

  Three-degree network.

  6.3.2 Your first cold sourcing recipe

  My standard approach to cold sourcing is:

  Join LinkedIn. I suspect variations of this technique would work on other networks (e.g., GitHub), but the challenge is that folks on other networks are generally not looking to engage about employment opportunities, and intent increases response rate significantly! A good parallel here is search versus display advertising, where search advertising has click-through rates that are an order of magnitude higher, as candidates are actually searching for what is being advertised.

  Build out your network by following individuals you actually do know. Add everyone that you went to school with, have worked with, have interacted with on Twitter, etc. It’s important to seed your network with some people you know because it’ll increase the reach of your second-degree network, and it’ll also reduce the rate at which folks mark you as someone they don’t know (which is an input to being penalized as a spammer).

  Be patient. If your initial network is small, it’s very likely that you’ll get throttled pretty frequently. Once you’re throttled (you’ll get a message along the lines of “You’ve exceeded your search capacity for this month”), you’ll have to wait for a few days, potentially until the next month, to unthrottle. Alternatively, you could sign up for their premium products, which would accelerate this quite a bit. It may take weeks or months of occasional effort (schedule an hour each week) to get your network large enough that you’re able to perform more than a few searches without rate limiting. Anecdotally, the number of connections at which things se
em to get easier is around 600 or so.

  Use the search function to identify second-degree connections to connect to. Start out by searching in your second-degree network by job title—software engineer or engineering manager—and as your network expands, consider switching from title to company. (Consider the various lists of great companies5 in order to find companies to search by.) Build a broad church of connections! Even people whom you don’t reach out to now are folks who might reach out to you later, or whom you might reach out to in a few months as your hiring priorities change. If you’re not sure, just go ahead and do it.

  When someone accepts your connection request, grab their email address from their profile and send them a short, polite note inviting them to coffee or a phone call, and sharing with them a link to your job description. Experiment with varying degrees of customization. (If you’re having trouble finding their email, make sure you click Show more in their Contact and Personal Info section, and that they’re a first-degree connection. A few folks don’t share their email at all, and in that case I’d recommend moving on. Alternatively, you could send them a LinkedIn message directly.) I’ve personally found that customization matters less than I assumed, because people mostly choose to respond based on their circumstances, not on the quality of your note. (As a caveat, it’s possible to write bad notes that discourage folks from responding. Iterating on your reach-out notes and getting a few other individuals with different perspectives to review your note is a quick and high-leverage thing to do.)