An Elegant Puzzle- Systems of Engineering Management

by Will Larson

  I believe that the balancing of positive and negative freedoms is a fundamental task of managers and management. When we’ve lucked upon (or perhaps nurtured, if you’re much more talented than I) a phenomenal culture and a great team who are executing well along a worthy roadmap, then, like a central bank reducing interest rates to avoid a bubble, or like a jogger reducing their pace to lower their heart rate, we can carefully ramp toward negative freedom and away from positive freedom. This is one of our essential tools for facilitating and prolonging success.

  Further down the road, if the structure loses its luster, the economy shifts around us, or entropy’s endless march throws a wrench into the machinery, then once again we shift toward positive freedoms, which gives the organization a greater chance to successfully adapt to its new circumstances.

  Using the two together, management slowly decelerates to keep the good times rolling, and accelerates to help push through challenging periods.

  Freedom is a loaded term, so it’s easy to deteriorate into a moral discussion, but in times and topics of great sensitivity, I believe looking through the lens of system dynamics12 is a valuable approach. Companies are vastly complicated systems with dozens of feedback loops, and managing the kind and quality of freedom is simply another mechanism to be adjusted, albeit with immense care and consideration.

  A few closing tangents. First, Tom DeMarco’s Slack13 has an excellent suggestion for a good starting state between positive and negative freedoms for engineering teams: generally follow the standard operating procedure (i.e., keep doing what you’re already doing, the way you’re doing it), but always change exactly one thing for each new project. Perhaps use a new database, a new web server, a different templating language, a static JavaScript front-end, whatever—but always change exactly one thing.

  Second, I’m always terrified of getting on the wrong side of history, so I’ve spent some time considering how this discussion of freedoms relates to Ben Horowitz’s recent post on “Can Do vs. Can’t Do Cultures.”14 I read that article as describing how young companies that are focused on innovation differ from mature companies that are trapped in “the innovator’s dilemma.”15 Older companies can (and do) foster sheltered pockets of innovation, as in the example of Larry Page investing in good ideas that he encounters within Google, but maintaining a market position is fundamentally distinct from creating new markets. I think that the more complete argument is to use both cultures (and, in parallel, to place emphasis on positive and negative freedoms) in the appropriate circumstances.

  5.6 Kill your heroes, stop doing it harder

  The project launch is 18 months late, company revenue is dropping significantly, key people are leaving and being replaced by new hires. What to do? Well, work harder!

  Does it work?

  Of course not. Unless your problem is that people aren’t trying hard, the “work harder” mantra only breeds hero programmers whose heroic ways make it difficult for nonheroes to contribute meaningfully. Later, as your new heroes finish martyring themselves on burnout, you’re left with three exceptionally challenging problems:

  You’ve bred a cadre of dissatisfied and burned-out heroes.

  You and your heroes have alienated everyone else.

  Your project is still totally screwed.

  This is a recurring pattern that many growing companies fall into, and it also happens to projects within larger companies. Anywhere you find managerial desperation and a hardworking team, “Do It Harder” may be visiting.

  5.6.1 The fall and rise of a hero

  One rainy day, you walk into the office and your boss wants to talk to you. He needs you to finish your project, but he also wants you to finish your coworker’s project, without the coworker feeling bad about it, because your coworker is still going to own the project, you’re just going to do it (along with your other work, remember).

  A few weeks later, the site starts crashing every few days, and the company really needs to launch the new version of the site. The boss pops in to let you know he deeply trusts you, and needs you to take over both efforts. You’re a good guy and it sounds like a good opportunity, and you’re pretty sure that you can do a better job than the guys already working on it, so you say yes.

  Congratulations! You’re a hero programmer.

  You’re now working on five disparate projects, trying not to piss too many people off, but you’re having trouble getting everyone involved. It seems like they’re not really working as hard as you are, and it’s a bit of a drag, since you’re pulling 70-hour weeks and getting paged every Saturday night.

  The other developers are glad that you’re taking a lead on the problems that were terrifying them too, but all is not well. A couple are quietly bitter because of your newly elevated status, but most just don’t know how to contribute anymore, because you and your hero peers are rewriting the existing system, debugging the outages, and cherry-picking the easy wins. What’s left for them to do?

  Day after day and week after week, the frustration between heroes and nonheroes grows stronger, tumbling toward inevitable disaster.

  5.6.2 Kill the hero programmer

  When it comes to solving the problem of the hero programmer, your options are limited: either kill the environment that breeds hero programmers, or kill the hero programmer by burnout.

  They’re truly unmaintainable beings, as their presence limits the effectiveness of those around them in exchange for a short-term burst of productivity fueled by long hours and minimized communication costs (minimized because most other people aren’t able to do much).

  These long hours burn your heroes out, and then they either quit or you shove them into a corner, where they’ll glower at you while remembering how their hard work and critical contributions culminated in them glaring at you from that corner.

  You can rehabilitate heroes, but it’s touch-and-go from the beginning, and healing takes time. They might hate you for a while, and they probably should, because you created them with your ham-fisted attempt to fix your current problem.

  5.6.3 A long time coming, a long time going

  One of the observations from systems thinking16 is that, though humans are prone to interpreting events as causal, often problems are better described in terms of a series of stockpiles that grow and shrink based on incoming and outgoing flows. The Dust Bowl17 wasn’t caused by one farmer or one year of overfarming, but by years of systemic abuse.

  Stocks and flows are especially valuable in understanding the failure of projects and teams. Projects fall behind one sprint at a time. Technical debt strangles projects over months.

  Projects fail slowly—and fixing them takes time, too.

  Working at a frenetic pace for a couple of weeks or a month may feel like a major outpouring of effort and energy, but it’s near impossible to quickly counteract a deficit created over months of poor implementation or management choices.

  If hard work and breeding heroes doesn’t work, what does?

  5.6.4 Resetting broken systems

  Your options for addressing a broken system depend on whether you’re in a position to set policy. If you set the original direction and have the leverage to change directions, then resetting is as simple as standing up and taking the bullet for the fiasco you’re embroiled in.

  Taking the blame is painful, and it only plays well with the crowds a couple of times. After that, people won’t trust you to lead them toward success, which makes some sense, since at that point you’ve led them off the rails multiple times. Fair’s fair.

  If taking the loss doesn’t leave you looking shiny, at least attaining closure is healing, and the team will have the opportunity to start healing as the schedule is reset and goals are adjusted. Without the leverage to change policy—and this doesn’t have to be direct authority, for influence is a powerful thing—you can’t start the healing, but you can help reach the reset point more quickly. (This is similar to the protagonists’ goal in Isaac Asimov’s Foundation’s series, as t
hey struggle to accelerate and minimize the collapse of the Galactic Empire.18)

  Without policy, your tool is stepping back and allowing the brokenness to collapse under its own weight. A deeply flawed system can’t be saved by band-aids, but it can easily absorb your happiness to slightly extend its viability. If you step back, you conserve your energy and avoid creating rifts by pushing others away in hero mode, and you will be ready to be a part of a new—hopefully more functional—system after the reset does occur.

  This is a very uncomfortable process, and if you’re a hardworking, loyal person, then it probably goes deeply against your nature. It certainly goes against my nature, but I believe that this is one case in which following my nature is a detriment to both myself and those around me.

  Projects fail all the time, people screw up all the time. Usually it’s by failing to acknowledge missteps that we exacerbate them. If we acknowledge errors quickly, and cut our losses on bad decisions before burning ourselves out, then we can learn from our mistakes and improve.

  Kill your heroes and stop doing it harder. Don’t trap yourself in your mistakes, learn from them and move forward.

  6

  Chapter 6

  Careers

  Figure 6.1

  Relationship of practitioner experience with policies that raise floor or raise ceiling of expected outcomes.

  Careers

  Luck plays such an extraordinary role in each individual’s career progression that sometimes the entire concept of career planning seems dubious. However, as managers, we have an outsize influence in reducing the role of luck in the careers of others. That potential to influence calls us to hold ourselves accountable for creating fair and effective processes for interviewing, promoting, and growing the folks we work with.

  This chapter explores how we can design effective interviewing and hiring processes, as well as how we can steer our own careers while standing on a bedrock of constant change.

  6.1 Roles over rocket ships, and why hypergrowth is a weak predictor of personal growth

  There’s a pervasive trope that people who’ve worked for an extended period in a large company will struggle to adapt to working in a smaller company. Work at a company too long, the theory goes, and you’re too specialized to hire elsewhere. This belief is reinforced by both age bias1 and the reality that few companies continue to win the rounds of reinvention necessary to maintain excellence over time. The pool of once-phenomenal companies is quite large: Yahoo!, Oracle, and VMware, to name a few.

  If long tenure holds a stigma, what of short tenure? Yep, that’s stigmatized, too.2 Although, it’s certainly less stigmatized than it used to be. A fairly consistent belief across companies is that multiple stints below one year are a bad sign, but, generally as long as you’ve worked a few years somewhere, then having a career that is otherwise entirely composed of one-year stints raises few red flags. While I’m certain that these beliefs are, at best, deeply flawed, they’ve accidentally become a rule of thumb in my own career: stay everywhere at least two years, and look for a place where I could spend four years to serve as the counterweight to my series of two-year stints. I’ve followed this rule very literally,3 staying at least two years (well, we can all count here, exactly two years) at each company I’ve worked at, and starting each job with the hope that this one would be the place I’d make it to four.

  To the surprise of no one, it turns out that’s been a pretty terrible way to think about career planning, and lately I’ve been trying to find a more useful framework.

  Working at a company isn’t a single continuous experience. Rather it’s a mix of stable eras and periods of rapid change that bridge between eras. Thriving requires both finding a way to succeed in each new era and successfully navigating the transitional periods. You yourself trigger some transitions, like switching companies. Others happen on their own schedule: a treasured coworker leaves, your manager moves on, or the company runs out of funding.

  Discard the discussion of tenure. Let’s talk about eras and transitions.

  6.1.1 Your new career narrative

  Start by building out a map of your past year. Each time there was a change that meaningfully changed how you work, mark that down as a transition. These could be your direct manager changing, your team’s mission being redefined, a major reorganization, whatever—what counts is if how you work changed. What skills did you rely on to navigate the transition? What skills did the transition give you an opportunity to develop?

  Next, think about the eras that followed those transitions. How did the values and expectations change? Did operational toil become considered critical work? Did work around inclusion and diversity become first-shift work that got mentioned in performance reviews? What skills did you depend on the most, and which of your existing skills fell out of use?

  What you just wrote down is your new career narrative, and it’s much richer than just another year at your company.

  6.1.2 Opportunities for growth

  The good news is that both the stable eras and the transitions are great opportunities for growing yourself. Transitions are opportunities to raise the floor by building competency in new skills, and in stable periods you can raise the ceiling by developing mastery in the skills that the new era values. As the cycle repeats, your elevated floor will allow you to weather most transitions, and you’ll thrive in most eras by leveraging your expanding masteries.

  It’s suggested that mature companies have more stable periods while startups have a greater propensity for change, but it’s been my experience that what matters most is the particular team you join. I’ve seen extremely static startups, and very dynamic teams within larger organizations. I particularly want to challenge this old refrain:

  “If you’re offered a seat on a rocket ship, don’t ask what seat! Just get on.”

  —Sheryl Sandberg

  Even hypergrowth companies tend to have teams that are largely sheltered from change by either their management or because they’re too far away from the company’s primary constraints to get attention.

  By tracking your eras and transitions, you can avoid lingering in any era beyond the point when you’re developing new masteries. This will allow you to continue your personal growth even if you’re working in what some would describe as a boring, mature company. The same advice applies if you’re within a quickly growing company or startup: don’t treat growth as a foregone conclusion. Growth only comes from change, and that is something you can influence.

  6.2 Running a humane interview process

  No matter how many times you’ve done it, changing companies is stressful because of the requisite job search and interviewing. Having conducted hundreds of interviews across a number of companies, I feel a bit more prepared to interview each time I do it, but being back on the interviewee’s side of the table always leaves me humbled.

  I’m confident that the state of interviewing is improving: many processes now involve a prepared presentation on a technical topic instead of an impromptu presentation (this more closely replicates a real work task), and many have replaced the whiteboard algorithm problem with a period of collaborative pair programming on a laptop with your editor of choice.

  Looking back on my early interviewing experiences, when I was once asked to do calculus on the whiteboard, I’m amazed at how far things have improved.

  That said, it’s certainly not the case that interviewing has improved uniformly. There is still a lot of whiteboard programming out there, and a disproportionate number of the most desirable companies continue with the practice due to the combined power of inertia (it was the state of play when many engineers and managers—including myself—entered the profession) and coarse-grained analytics (if you’re hitting your hiring goals—and with enough dedicated sourcers, any process will hit your hiring goals—then it can be hard to prioritize improving your process).

  Reflecting on the interviews I’ve run over the past few years and those I got to experience recently,
I believe that, while interviewing well is far from easy, it is fairly simple.

  Be kind to the candidate.

  Ensure that all interviewers agree on the role’s requirements.

  Understand the signal your interview is checking for (and how to search that signal out).

  Come to your interview prepared to interview.

  Deliberately express interest in candidates.

  Create feedback loops for interviewers and the loop’s designer.

  Instrument and optimize as you would any conversion funnel.

  You don’t have to do all of these to be effective! Start from being nice and slowly work your way through to the analytics.

  6.2.1 Be kind

  A good interview experience starts with being kind to your candidate.

  Being kind comes through in the interview process in a hundred different ways. When an interview runs overtime before you get to the candidate’s questions, the kind thing to do is to allow the candidate a few minutes to ask questions, instead of running on to the next interview to catch up. Likewise, in that scenario the kind thing is to then negotiate new staggered start times, rather than kicking off a cascade of poor interviewer time management as each person tries to fractionally catch up to the original schedule.

  My experience is that you can’t conduct a kind, candidate-centric interview process if your interviewers are tightly time-constrained. Conversely, if an interviewer is unkind to a candidate (and these unkindnesses are typically of the “with a whisper not a bang” variety), I believe it is very often a structural problem with your interviewing process, and not something you can reliably dismiss as an issue with that specific interviewer.