An Elegant Puzzle- Systems of Engineering Management

by Will Larson

  The three rules for speaking with the media:

  Answer the question you want to be asked. If someone asks a very difficult or challenging question, reframe it into one that you’re comfortable answering. Don’t accept a question’s implicit framing, but instead take the opportunity to frame it yourself. Don’t Think of An Elephant by George Lakoff 32 is a phenomenal, compact guide to framing issues.

  Stay positive. Negative stories can be very compelling. They are quite risky, too! As an interviewee, find a positive framing and stick to it. This is especially true when it comes to competitors and controversy.

  Speak in threes. Narrow your message down to three concise points, make them your refrain, and continue to refer back to your three speaking points.

  That was it! Concise, compact, and I’m still using and learning from that advice a decade later.

  3.11 Model, document, and share

  Early on in my career, I spent a lot of time trying to find my leadership style. Recently, I think it’s more useful to think about growing yourself as a leader by developing a range of styles and applying them thoughtfully to your circumstances. Confining yourself to one style is just too hard.

  One of the trickiest, and most common, leadership scenarios is leading without authority, and I’ve written about one of the styles that I’ve found surprisingly effective in those conditions. I call it Model, Document, Share.

  Figure 3.7

  The Model, Document, Share-Mandate approach.

  3.11.1 How it works

  Imagine that you’ve started a new job as an engineering manager, and the teams around you are too busy to use a planning process. You’ve mentioned to your peers a few times that you’ve seen kanban33 work effectively, but folks tried it two years ago and are still upset whenever the word is mentioned: it just doesn’t work here.

  Your first reaction might be to confront this head-on, but it takes a while to build credibility after starting a new job. Sure, you’ve been hired for your experience, so they respect your judgment, but it’s a hard sell to convince someone that your personal experience should invalidate their personal experience.

  I’ve been trying something different.

  Model. Start measuring your team’s health (maybe using short, monthly surveys) and your team’s throughput (do some lightweight form of task sizing, even if you just do it informally with a senior engineer on the team or with yourself), which will allow you to establish the baseline before your change.

  Then just start running kanban. Don’t publicize it, don’t make a big deal about it, just start doing it with your team. Frame it as a short experiment with the team, and start trying it. Keep iterating on it until you’re confident it works. Have the courage to keep at it for a while, and also the courage to stop doing it if it doesn’t work after a month or two. Use the team’s health and throughput metrics to ground your decision about whether it’s working.

  Document. After you’ve discovered an effective approach, document the problem you set out to solve, the learning process you went through, and the details of how another team would adopt the practice for themselves. Be as detailed as possible: make a canonical document, and even get a few folks on other teams to check that it’s readable from their perspective.

  Share. The final step is to share your documented approach, along with your experience doing the rollout, in a short email. Don’t ask everyone to adopt the practice, don’t lobby for change, just present the approach and your experience following it.

  You’ll spend the majority of your time refining approaches that work effectively for your team, a bit of your time documenting how you did it, and almost no time trying to convince folks to change their approach.

  Strangely, in my experience, this has often led to more adoption than top-down mandates have.

  3.11.2 Where it works

  When considering how the Model, Document, Share approach works, it’s interesting to compare it with the top-down mandate.

  Mandates assume:

  It’s better to adopt a good-enough approach quickly.

  Teams have the bandwidth to adopt a new approach.

  The organization has available resources to coordinate a rollout.

  You want to centralize decision-making on this topic.

  Consistency is important; all teams need to approach this problem in the same way.

  It’s important to make this change quickly.

  Model, Document, Share assumes:

  It’s better to adopt a great approach slowly.

  Some teams don’t have the bandwidth to adopt a new approach.

  The organization may not have resources to coordinate a rollout.

  You want to decentralize decision-making on this topic.

  Teams have agency to adopt the appropriate practices for themselves.

  It’s okay to approach change gradually.

  If your circumstances and your organization’s values align with the second list, then this approach may be more effective for you than making mandates. If you have the time, you can slowly flock toward great practice, without needing organizational authority. (You’ll still need some natural authority, the respect of your peers.)

  Although I’ve seen this approach work remarkably well, I’ve also seen it go nowhere. It’s a particular tool for certain circumstances, and it does fail. It may be an inexpensive failure—folks simply don’t adopt—as you haven’t spent much authority on it, but nonetheless you still haven’t accomplished your goal. It’s particularly important not to try to use this as a strategy to circumvent organizational authority. Operating in direct conflict with authority usually doesn’t end very well.

  3.12 Scaling consistency: designing centralized decision-making groups

  In small organizations, it’s easy for individuals to be aware of what others are doing and to remember how they’ve previously approached similar problems. This hive mind and memory create decision-making whose consistency correlates strongly with quality. As organizations grow, there is a subtle slide into inconsistency, which is often one of the most challenging aspects of evolving from a small team into a much larger one.

  There are many different approaches to try to manage inconsistency creep. Some of the solutions I’ve seen are formalized sprints, training, shadowing, documentation, code linters,34 process automation (particularly deploys), and incident reviews. However, when the problem becomes truly acute, folks eventually reach for the same tool: adding a centralized, accountable group.

  The two most common flavors of this I’ve seen are “product reviews” to standardize product decisions and the “architecture group” to encourage consistent technical design. There are hundreds of varieties, and they crop up wherever decisions are made.

  Some of these groups take on an authoritative bent, becoming rigid gatekeepers, and others become more advisory, with a focus on educating folks toward consistency. Depending on your culture and how you value consistency, there are an infinite number of approaches, and designing an effective decision-making group depends on a handful of core decisions.

  3.12.1 Positive and negative freedoms

  Before jumping into design, a few words on the framing that I use for reasoning about when to create a new centralized authority.

  These groups typically consolidate significant authority from the broader community into the hands of a few. Many folks will feel a significant loss of freedom when you create these groups, as their zone of decision-making will be newly limited. Less obviously, many folks find the creation of centralized groups to be extremely empowering. The difference? One group is largely populated by individuals comfortable with self-authorization, and the latter typically have a higher threshold for self-authorization.

  This is just one example of the dynamics that play out across many dimensions when you’re considering introducing a new authority, and the most useful framework I’ve found for thinking through this involves positive and negative freedoms. A positive freedom is the freedom to do
something, for example the freedom to pick a programming language you prefer. A negative freedom is the freedom from things happening to you, for example the freedom not to be obligated to support additional programming languages, even if others would greatly prefer them.

  How are you shifting freedoms, and who are you shifting them from? Particularly in cases where ownership is extremely diffuse, I believe that cautiously authoritative groups do increase net positive freedom for individuals without greatly reducing negative freedom. That also happens to be my goal when designing a new group!

  3.12.2 Group design

  Now that you’ve decided to create a decision-making group, it’s time to get into the choices!

  Influence. How do you expect this group to influence results? Will they be an authoritative group that makes binding decisions? Will you rely on the natural authority of the members you select? Will they primarily work through advocacy? The answers to these questions along with the particular folks in the group, will be the primary factor in how your group impacts the positive and negative freedoms of those they work with.

  Interface. How will other teams interact with this team? Will they submit tickets, send emails, attend a weekly review session? Will you be reviewing work before it launches, or previewing designs before they’re staffed? Depending on the kind of influence they’re exerting and how your company works, you’ll want to play around with different approaches.

  Size. How large should the group be? If it’s six or fewer individuals, it’s possible for them to gel into a true team, one whose members know each other well, work together closely, and shift a significant portion of their individual identities into the team. If the group has more than ten members, you’ll find it hard to even have a good discussion, and it’ll need to be structured into sub-groups to function well (rotation that spreads members over time, working groups of some members to focus on particular topics, and so on). The larger the group, the more responsibility becomes diffuse, and the more you’ll need to have specified roles within the group, for example a member responsible for coordinating members’ focus.

  Time commitment. How much time will members spend working in this group? Will this be their top priority, or will they still primarily be working on other projects? Higher time commitment correlates with more actions and decisions. Consequently, my sense is that you want time commitment to be higher for areas where folks are directly impacted by the consequences of their decisions, and to be lower for scenarios with weaker feedback loops.

  Identity. Should members view their role in the group as their primary identity? Should they continue to identify primarily as members of their existing team? You’ll need a small team and high time commitment to support individuals shifting their identity.

  Selection process. How will you select members? I’ve found the best method to be a structured selection process,35 in which you identify the requirements to be a member and the skills that you believe will be valuable, and then allow folks to apply. Membership in these groups often becomes an important signal of organizational status, which makes having a consistent process for selecting membership especially important.

  Length of term. How long will members serve? Are these permanent assignments, or are they fixed terms for, say, six months? If they are fixed terms, are folks eligible for subsequent elections? Is there a term limit? I’ve tried most combinations, and my sense is that the best default is fixed terms, while allowing current individuals to remain eligible, and without enacting term limits.

  Representation. How representative will this group be? Will you explicitly select folks based on their teams, tenure, or seniority, or will you allow clusters? Attention to this can help you avoid architecture reviews that are missing front-end and product engineers, as well as product reviews without infrastructure perspective.

  Predictably, each of these decisions will impact the effectiveness of the others, which can make designing the group you want quite tricky. Some formats will require particular kinds of individuals to staff them, and you have to design groups that will work with the people available to participate and the culture they are participating within.

  3.12.3 Failure modes

  Before you release that email you’re writing to spin up a new centralized decision-making group, it’s worth talking about the four ways these groups consistently fail. They tend to be domineering, bottlenecked, status-oriented, or inert.

  Domineering groups significantly reduce individuals’ negative and positive freedoms, and become churn factories for members. This is most common when those making decisions are abstracted from the consequences of the decisions, e.g., architecture groups in which the members write little code.

  Bottlenecked groups tend to be very helpful, but are trying to do more then they’re actually able to do. These groups get worn down, and either burn out their members or create a structured backlog to avoid burning themselves out, which ends up seriously slowing down folks who need their authority.

  Status-oriented groups place more emphasis on being a member of the group than on the group’s nominal purpose. The value of the group revolves around recognition rather than contribution, leading to folks who try to join the group for status, and the diffusion of whatever original mission the group set out to resolve.

  Inert groups just don’t do much of anything. Typically, these are groups whose members have not gelled or are too busy. On the plus side, this is by far the most benign of the four failure modes—but you’re also missing out on a great deal of opportunity!

  Having experienced each of these a few times, I try to ensure that there is a manager embedded into every centralized group, and that the manager is responsible for iterating on the format to dodge these pitfalls.

  3.13 Presenting to senior leadership

  Yahoo! BOSS36 was partially powered by an internal Yahoo! search technology named Vespa. We’d run into a bunch of challenges, and I’d decided to convince my team we should migrate to SOLR.37 My manager asked me to put together a presentation for our next team meeting. The meeting came, I started to present, and within two slides things fell apart.

  “No, no, this isn’t the way to put this together. This is like an academic presentation. You have to start from the conclusion first,” my manager lamented as he abruptly terminated my presentation. Pausing to brush my deck’s fragments from his boots, he offered some final wisdom: “And don’t use curved lines in your diagrams. Those never make sense.”

  It took me a few years to glean a lesson from that experience, but it comes to mind frequently now when I work with folks who are just starting to present to executives. Giving a presentation to senior leadership is a bit of a dark art: it takes a while to master, and most people who do it well can’t quite articulate how they do it. Worse yet, many people who are excellent rely on advantages that resist replication: charisma, quick wit, deep subject matter expertise, or years of experience.

  That said, few people watching me bomb my Yahoo! presentation would have bet I’d ever figure this one out, so you should know that it is a learnable skill. Along the way, I’ve picked up some tips that I hope will help you prepare for your next presentation:

  Communication is company-specific. Every company has different communication styles and patterns. Successful presenters probably can’t tell you what they do to succeed, but if you watch them and take notes, you’ll identify some consistent patterns. Each time you watch individuals present to leadership, study their approach.

  Start with the conclusion. Particularly in written communication, folks skim until they get bored and then stop reading. Accommodate this behavior by starting with what’s important, instead of building toward it gradually.

  Frame why the topic matters. Typically, you’ll be presenting on an area that you’re intimately familiar with, and it’s probably very obvious to you why the work matters. This will be much less obvious to folks who don’t think about the area as often. Start by explaining why your work matters to the compa
ny.

  Everyone loves a narrative. Another aspect of framing the topic is providing a narrative of where things are, how you got here, and where you’re going now. This should be a sentence or two along the lines of, “Last year, we had trouble closing several important customers due to concerns about our scalability. We identified our databases as our constraints to scaling, and since then our focus has been moving to a new sharding model that enables horizontal scaling. That’s going well, and we expect to finish in Q3.”

  Prepare for detours. Many forums will allow you to lead your presentation according to plan, but that is an unreliable prediction when presenting to senior leadership. Instead, you need to be prepared to lead the entire presentation yourself, while being equally ready for the discussion to derail toward a thread of unexpected questions.

  Answer directly. Senior leaders tend to be indirectly responsible for wide areas, and frequently pierce into areas to debug problems. Their experience “debug piercing” tunes their radar for evasive answers, and you don’t want to be a blip on that screen. Instead of hiding problems, use them as an opportunity to explain your plans to address them.

  Deep in the data. You should be deep enough in your data that you can use it to answer unexpected questions. This means spending time exploring the data, and the most common way to do that is to run a thorough goal-setting exercise.38

  Derive actions from principles. One of your aims is to provide a mental model of how you view the topic, allowing folks to get familiar with how you make decisions. Showing you are “in the data” is part of this. The other aspect is defining the guiding principles you’re using to approach decisions.