An Elegant Puzzle- Systems of Engineering Management
Page 18
Roles too “trivial” to value. Many roles start by taking on work that is viewed as uninteresting by the role shedding that responsibility, and consequently individuals in that role tend to view that work as trivial and unimportant. This often translates into the new roles struggling to have their impact be recognized.
Roles too “trivial” to promote. Similar to the above, you’ll often find that the work done by new roles is valued very highly in terms of impact, but not viewed as sufficiently “strategic” to merit promotion, particularly beyond career level.19 This can lead to individuals being obligated to change roles if they want to attain higher tiers of achievement.
Head count obstacles. Companies eventually develop arcane rituals by which a series of emails, meetings, and incantations is translated into an annual head count plan. These systems are, quite reasonably, designed around supporting the needs of large, existing roles. Consequently, they tend to make it pretty challenging to get head count for a new role, particularly if it’s in tension with existing functions that need to expand.
Recruiting rare humans. For entirely great reasons, people want the first hires they make into a new role to be strong role models for the entire function. This often leads to a proliferation of requirements until it’s impossible for any candidate to pass the bar.
Inability to evaluate. This is almost the opposite of the above challenge: sometimes the existing organization has so little experience with the new function they wish to create that they simply don’t have a usable means by which to evaluate candidates. This can lead to evaluations focused on qualities that are largely independent of what the candidates would be doing if they accepted the job.
6.7.2 Facilitating success
If we do want to create a new role, it’s important to take stock of what we’ll need to do to make employees in the role successful. The ingredients that I’ve found necessary for a new role to bootstrap successfully are:
Executive sponsor. Not necessarily an executive, but you’ll need to find an authorized, senior leader who is committed to the success of the new function. Especially for the first performance20 and head count cycles, there will be a number of rough edges that will require significant organizational support to navigate. The need to find a sponsor who’ll be able to provide the necessary support is the most obvious constraint on creating new roles. If you can’t find a sponsor, it’s usually important feedback that leadership doesn’t believe the new role will have a good return on invested energy.
Recruiting partner. A new role will require significant support from your recruiting organization in order for it to succeed. Every role being recruited against has a high fixed cost, and adding new roles can make it difficult for recruiters to hit the targets that their performance is measured against. Make sure that your recruiting team is able to support a new role! If they aren’t, the first step may be working with your executive sponsor to direct more head count toward recruiting.
Self-sustaining mission. New roles are frequently described in terms of how they’ll impact other functions, rather than in terms of what they’ll accomplish. For example, you might describe technical program managers (TPMs) as offloading project management responsibilities from engineering managers. This approach frames the role as an auxiliary, support function, which makes it difficult to recognize the work’s impact. You must be able to frame the role’s work without referencing other existing roles in order for it to succeed long-term.
Career ladder. In pretty much all cases, new roles should have a career ladder from the beginning. The career ladder is the foundation of a successful performance management system, and it’s not possible for a role to be valued or evaluated coherently without a thoughtful career ladder. Sometimes folks rush ahead to hire before writing the ladder, but the work required to design an effective interview loop is roughly equivalent to writing a career ladder, so I’ve found that skipping this step is an act of false economy.
Role model. Who will be the external and internal role models for this role? A good role model is a human embodiment of the intent codified in your career ladder. You want to have a person you can point to.
Dedicated calibrations. Most performance management systems rely on a calibration system to ensure that performance designations are being assigned in a consistent fashion across teams and roles. Sometimes managers try to perform calibration with mixed roles in a single session, which leads to smaller roles being treated as afterthoughts. Often the designations will be approved without much thought, or they’ll be pushed toward the center. Neither of those scenarios creates a useful feedback loop for the individual in the new role. It’s best to consider them separately in a dedicated calibration session for the one role. If that’s not possible, the second-best option is to consider all smaller roles together, so that various forms of specialized contributions can be considered thoughtfully.
6.7.3 Advantages
If creating a new role was all costs and challenges, it would be easy to decide not to move forward, but there are pretty significant advantages to be had:
Efficiency. This is the flip side of brittle organization: people in specialized roles are able to spend more time doing a smaller set of tasks, which leads to great expertise in that area. For areas where existing roles are spending significant amounts of time, this specialized efficiency can translate into significantly more overall throughput without increasing head count. I think this is the most important advantage, and it’s especially valuable for teams or companies in which financial resources are the limit on growth (which is most teams at most companies).
Efficiently solve constraints. This is an extension of the efficiency point, but subtly different: with specialists, you can add exactly the kind of capacity you are missing, which is very powerful for efficiently solving constraints. If your organization is low on project management bandwidth, you could add five new managers who can each take on a bit of it, or you might be able to add a single project manager who individually adds as much relevant bandwidth as the five managers combined.
Recognition. Simply creating a new role will absolutely not cause folks to suddenly start valuing work that they previously dismissed, but it can be a useful component in that shift. In particular, it will provide additional structural mechanisms to support recognition, such as distinct career ladders, calibration sessions, and even compensation structures.
Evaluating for strengths. It’s often challenging to interview specialists effectively, because you’ll typically evaluate them based on how they’d perform for your generalist position, while missing out on their peculiar areas of excellence. Creating a new role makes it possible to target the interview process on the areas that are most important.
Increased hiring pool. You’re now able to consider a new pool of candidates in your hiring funnel,21 and that potentially expands the total number of candidates you can consider.
Specialized compensation. In some cases, the market compensation for specialists is significantly higher than that for generalists, and in that case it’s often quite a challenge to hire specialists without specialized compensation bands.
6.7.4 What to do?
Once you’re familiar with the challenges and the costs of provisioning a new role, all that’s left is to consider the advantages and make a quick judgment call. Ah, well, actually it’s still a pretty hard decision to make!
Some questions that can guide this decision are:
Is there a less extreme way to address the recognition gap? Maybe you could adjust the existing career ladder instead.
Do you have a plan for changing how the company values the work? Creating a new role won’t inherently change how the company values this work. You’ll still need to do the hard work of expanding your company’s values.
If you have a plan to change company values, could you do a trial run of that plan before introducing the new role? This helps de-risk the experiment for the folks you are trying to help, and it’s much easier to boot.
Will this increase short-term recognition of performance but ultimately hurt the career growth of employees who change roles? Creating a new role is absolutely the kind of thing that can initially feel like progress but ultimately sets folks back significantly, requiring them to transition to other roles later.
Is the number of people impacted sufficiently high and is the recognition gap of value significantly large, to cover the sizable costs of creating and nurturing a new role?
Who will pay the maintenance costs for the new role? If the answer is that you’ll personally pay them, who will take up the torch if you leave?
As you think through those questions, hopefully the right approach for your situation gets a bit clearer. As a rule of thumb, I would always create a new role if it immediately covered 20 individuals; would reluctantly create a new role if it would cover 20 individuals within two years; and would be skeptical of creating a new role that couldn’t meet one of those two conditions!
6.8 Designing an interview loop
Anyone who has flipped through Cracking the Coding Interview by Gayle Laakmann McDowell22 knows that evaluating candidates for a new role is a coarse science. Most interviewers are skeptical of the accuracy of their interviews, and it’s hardly the rare interview retrospective where interviewers aren’t sure they got enough signal on a candidate to hire with confidence.
Although a dose of caution about interview accuracy is well-founded, I’m increasingly convinced that a bit of structure and creativity leads to interview loops that give a clear signal on whether a candidate will succeed in the role, and whether they can do so in a fair and consistent fashion.
The approach I’ve started using to design interview loops is:
Metrics first. Do not start designing a new interview loop until you’ve instrumented your hiring funnel.23 It’s pretty hard to evaluate your loop or improve it without this data, so don’t undertake an improvement project without it.
Understand the current loop’s performance. Take time to identify what you think does and doesn’t work well in your current process. Three sources that are useful are:
Funnel performance. Where do people drop off in your funnel? How does your funnel benchmark against your peers at similar companies?
Employee trajectory. For all the candidates you have hired, understand how their work performance relates to their interview performance. What elements correlate heavily with success, and which appear to filter on things that end up not contributing to performance in the job?
Candidate debriefs. Try to schedule calls with everyone who goes through your process, especially with those who drop out.
Learn from peers. Chat with folks who have interviewed at other companies for the kind of role you’re hoping to evaluate. You’re not looking to copy elements verbatim, but rather to get a survey of existing ideas.
Find role models. Write up a list of ideal candidates for the role and write down what makes them ideal. Be deliberate about ensuring that your list of role models includes women and underrepresented minorities, helping avoid matching on correlating traits in a less diverse set of role models.
Identify skills. From your role models and your career ladder,24 identify a list of skills that are essential to the candidate’s success in the role they’ll be interviewing for. Take a bit of time to rank those skills from most to least important.
Test for each skill. For each skill, design a test to evaluate the candidates’ strengths. Whenever possible, prefer tests that have a candidate show their strengths, avoiding formats in which they tell you about it. For example, we took an interview in which folks described their experience building a healthy team, and replaced it with an interview in which folks are given results from an organization health survey and asked to identify issues and propose how they’d address them.
Avoid testing for polish. Many interviews accidentally test for the candidate’s polish as opposed to testing for any particular skill. This is especially true for experiential interviews in which folks are asked to describe their work, and less common in interviews that ask them to demonstrate skills. That isn’t to say that you shouldn’t deliberately test for polish—it’s quite useful—just that you shouldn’t do it inadvertently.
For each test, a rubric. Once you’ve identified your tests, write a rubric to assess performance on each test. Good rubrics include explicit scores and criteria for reaching each score. If you find it difficult to identify a useful rubric for a test, then you should look for a different test that is easier to assess.
Avoid Boolean rubrics. Some tests tend to return Boolean results: for example, whether someone has experience managing someone is a good example of a common Boolean filter. These are inefficient tests because you pretty quickly get a sense of whether someone has or hasn’t ticked this box, and the rest of the interview doesn’t lend more signal. Likewise, you can almost always answer Boolean questions from an applicant’s resume or in a pre-interview screen.
Group tests into interviews. Once you’ve identified the tests, group them into sets that can be performed together in a single 45-minute interview, or whatever duration you prefer. The more cohesive the format and subjects of the tests in a given interview, the more natural it’ll feel for the candidate.
Run the loop. At this point you have an entire loop ready, and it’s time to start using it. Especially early on, you should be asking candidates what did or didn’t work well—but, really, you should never stop doing this! Each debrief will uncover some opportunities to improve your rubric or tests.
Review the hiring funnel. After you’ve run the new loop 10 to 20 times, review the funnel metrics to see how it’s working out. Are some of the interviews yielding too many passes? Are others too hard? Review the results in batch.
Schedule an annual refresh. As the initial rate of iteration slows down, schedule a review for a year out, and at that point you get to ask yourself if the loop has proven effective for your needs, or if you should restart this process!
At this point, you have a complete interview loop and the systems to guide the loop toward improvement. Beyond this approach, here are a few more general pieces of guidance:
Try to avoid design by committee. These almost always lead to incremental change. Prefer a working group of one or two people that is then tested against a bunch of candidates for feedback!
Don’t hire for potential. Hiring for potential is a major vector for bias, and you should try to avoid it. If you do decide to include potential, then spend time developing an objective rubric for potential, and ensure that the signals it indexes on are consistently discoverable.
Use your career ladder. Writing a great interview loop is almost identical to writing a great career ladder.25 If you’ve already written expectations for the role, reuse those as much as possible.
Iterate on the interview a little. When you first create an interview, you should spend time iterating on the interview format, but the rate of change should drop to near-zero after you’ve given it approximately 10 times.
Iterate on the rubric a lot. As you attempt to apply the interview rubrics, you’ll continue to find edge cases and ambiguities. Continuing to incorporate those into the interview rubrics is an essential way to reduce bias creeping in.
A/B testing loops. There is a platonic ideal of testing new interviews using the standard mechanisms we use to test other changes, such as A/B testing. At a certain scale, I think that is almost certainly the best approach, but so far I’ve been part of a company with enough volume to usefully conduct such tests. In particular, these tests are quite expensive, because you need to control for interviewers and have those interviewers trained on both sets of interviews, which ends up taking a long time to reach confidence that a new loop is better.
Hiring committees. As an alternative to A/B testi
ng, I have found a centralized hiring committee that reviews every candidate’s interview experience to be quite valuable for identifying trends across new loops. More generally, this approach helps guide the overall process toward consistency and fairness.
Wrapping all my learnings on designing interview loops together into something pithy: avoid reusing stuff that you know doesn’t work, and instead approach the matter from first principles with creativity. Then keep iterating based on how it works for candidates!
Figure 6.13
Steps of designing an interview loop.
7
Chapter 7
Appendix
Figure 7.1
Organization processes added to support operating it as it scales.
7.1 Tools for operating a growing organization
One tension in management is staying far enough out of the details to let folks innovate, yet staying near enough to keep the work well-aligned with your company’s value structures. Seeing this challenge from a variety of perspectives with different teams, I’ve collected a playbook of tools to facilitate the balancing act, and a loose framework for rolling those tools out.
A few caveats about process rollouts: