An Elegant Puzzle- Systems of Engineering Management

by Will Larson

  The elements that I’ve found effective for solution validation are:

  Write a customer letter. Write the launch announcement that you would send after finishing the solution. Are you able to write something exciting, useful, and real? It’s much more useful to test it against your actual users than to rely on your intuition.

  Identify prior art. How do peers across the industry approach this problem? The fact that others have solved a problem in a certain way doesn’t mean that it’s a great way, but it does at least mean it’s possible. A mild caveat: it’s better to rely on people you have some connection to instead of on conference talks and such, since there is a surprisingly large amount of misinformation out there.

  Find reference users. Can you find users who are willing to be the first users for the solution? If you can’t, you should be skeptical whether what you’re building is worthwhile.

  Prefer experimentation over analysis. It’s far more reliable to get good at cheap validation than it is to get great at consistently picking the right solution. Even if you’re brilliant, you are almost always missing essential information when you begin designing. Analysis can often uncover missing information, but it depends on knowing where to look, whereas experimentation allows you to find problems that you didn’t anticipate.

  Find the path more quickly traveled. The most expensive way to validate a solution is to build it in its entirety. The upside of that approach is that you’ve lost no time if you picked a good solution. The downside is that you’ve sacrificed a huge amount of time if it’s not. Try to find the cheapest way to validate.

  Justify switching costs. What will the costs of switching be for users who move to your solution? Even if folks want to use it, high switching costs may mean that they simply won’t be able to. Test with your potential users if they’d be willing to pay the full cost of migrating to your solution instead of their existing planned work.

  As an aside, I’ve found that most aspects of running a successful technology migration12 overlap with good solution validation! This is a very general skill that will repay many times over the time you invest in learning it.

  Putting these three elements in place today—exploration, selection, and validation—won’t make you an exceptional product manager overnight, but they will provide a solid starting place to develop those skills and perspective for the next time you find yourself donning the product manager hat.

  3.3 Visions and strategies

  As an organization grows beyond 50 people or so, you’ll feel a building pressure to add a third layer of management, and eventually you will. This ought to be a benign event: What’s the difference between supporting some managers and supporting their managers? It shouldn’t be too different, but for me it was when my previous mechanisms of alignment stopped working very well.

  Where I was once partnering with teams on their project roadmaps, now I found myself increasingly surprised by the projects that they were working on. Where I was once debating different approaches with the teams, now that conversation was happening in rooms I didn’t have time to join.

  My first instinct was to dive in and understand each instance, but that, unsurprisingly, wasn’t a very scalable solution. My second instinct was to design a series of “operating reviews” in which we periodically reviewed metrics and major projects. While those were useful in their own right, they proved more effective for learning and fine-tuning than for broad alignment. Being out of alignment for a quarter is just so much uncaptured potential.

  What I needed was a way to coordinate my approach across teams, both in terms of very specific challenges and in terms of our long-term direction. After experimenting with a handful of different approaches, agreeing on strategy and vision has been the most effective approach that I’ve found to alignment at scale.

  3.3.1 Strategies and visions

  Strategies are grounded documents which explain the trade-offs and actions that will be taken to address a specific challenge. Visions are aspirational documents that enable individuals who don’t work closely together to make decisions that fit together cleanly.

  Picking the right format for your needs is important, but the most important thing is probably to give both a try and get a feel for them! These are peculiar genres of literature that take some practice to master. It’s taken me years to get comfortable writing visions, and it was only when I started to support teams with seemingly incompatible ideologies that these documents value became clear. The same goes for writing strategies: it took a long time and skeptical study of several strategy books before what I wrote started to come together into a useful artifact.

  With that said, it’s time to dig into how to write strategies first, and then visions.

  Figure 3.4

  Table of differences between strategy and vision.

  3.3.2 Strategy

  A strategy recommends specific actions that address a given challenge’s constraints. A structure that I’ve found extremely effective13 is described in Good Strategy/Bad Strategy by Richard Rumelt,14 and has three sections: diagnosis, policies, and actions.

  The diagnosis is a theory describing the challenge at hand. It calls out the factors and constraints that define the challenge, and at its core is a very thorough problem statement. An example of a simple diagnosis might be “I am too busy to think about long-term goals. I attend 35 hours of meetings each week. I am under pressure to immediately increase team performance. I believe that if I stop doing my current meetings, short-term team performance will decrease. I am concerned that if my short-term team performance decreases, I may lose face as an effective leader, which will undermine my career opportunities. I believe that if I don’t think about long-term goals, our performance will never improve, which will also undermine my career opportunities.” Before you’ve even finished reading a great diagnosis, you’ll often have identified several good candidate approaches. That’s the power of a well-defined problem statement, and why it’s an important foundational element for your strategy.

  The second step is to identify policies that you will apply to address the challenge. These describe the general approach that you’ll take, and are often trade-offs between two competing goals. Continuing the above example, you might choose to allow short-term performance to dip in order to invest into long-term performance, combined with a policy of proactive expectation-setting with your stakeholders. Conversely, you might choose to take a hill-climbing approach to long-term performance, with iterative short-term improvement leading to improved long-term performance. Both are valid guiding policies, and both embrace the reality that you have limited resources to invest. When you read bad guiding policies, you think, “so what?” because its found a way to justify entrenching the status quo. When you read good guiding policies, you think, “Ah, that’s really going to annoy Anna, Bill, and Claire,” because the approach takes a clear stance on competing goals.

  When you apply your guiding policies to your diagnosis, you get your actions. Folks are often comfortable with hard decisions in the abstract, but struggle to translate policies into the specific steps to implement them. This is typically the easiest part to write, but publishing it and following through with it can be a significant test of your commitment. In the example above, your specific actions might be to stop attending weekly team meetings in order to free up time and move to a monthly metrics review, combined with blocked-out focus hours that you unapologetically shelter. When you read good, coherent actions, you think, “This is going to be uncomfortable, but I think it can work.” When you read bad ones, you think, “Ah, we got afraid of the consequences, and we aren’t really changing anything.”

  Because strategies are specific to a given problem, it’s okay—and even encouraged—to write quite a few of them. Over the past year, I’ve worked with people on strategies for how we partner with other teams, how we manage end-to-end API latency, and how we manage infrastructure costs.15 I’ve also peered over others’ shoulders as they worked on quite a f
ew more ideas. The act of writing a strategy leads folks through a systematic analysis, so, even if we don’t share them, writing these documents helps us work through quite a few challenges, both overwhelming and mundane.

  People sometimes describe strategy as artful or sophisticated, but I’ve found that the hardest part of writing a good strategy is pretty mundane. You must be honest about the constraints that are making the challenge difficult, which almost always include people and organizational aspects that are uncomfortable to acknowledge. No extent of artistry can solve a problem that you’re unwilling to admit.

  3.3.3 Vision

  If strategies describe the harsh trade-offs necessary to overcome a particular challenge, then visions describe a future in which those trade-offs are no longer mutually exclusive. An effective vision helps folks think beyond the constraints of their local maxima, and lightly aligns progress without requiring tight centralized coordination.

  You should be writing from a place far enough out that the error bars of uncertainty are indisputably broad, where you can focus on the concepts and not the particulars. Visions should be detailed, but the details are used to illustrate the dream vividly, not to prescriptively constrain its possibilities.

  A good vision is composed of:

  Vision statement: A one- or two-sentence aspirational statement to summarize the rest of the document. This is your core speaking point, which you will repeat at each meeting, planning period, and strategy review. It shouldn’t try to capture every detail of your vision, but it does need to memorably evoke your vision effectively.

  Value proposition: How will you be valuable to your users and to your company? What kinds of success will you enable them to achieve? There is a sequencing question of whether you should start with capabilities (the next bullet) and reason into value proposition or do the opposite, but I’ve found that starting from your users leads to visions that are both more ambitious and more grounded.

  Capabilities: What capabilities will the product, platform, or team need in order to deliver on your value proposition? Will it need to support multiple independent business lines? Will it need to deliver against the disparate needs of several distinct customer cohorts?

  Solved constraints: What are the constraints that you’re limited by today, but that in the future you’ll no longer be constrained by? For example, if you’re currently “spending into” developer velocity, perhaps in the future you’ll be able to achieve significant developer velocity while also maintaining low costs.

  Future constraints: What are the constraints that you expect to encounter in this wonderful future? Hopefully, these new constraints will be things that are easy to adjust, like funding or hiring.

  Reference materials: Link all the existing plans, metrics, updates, references, and documents into an appendix for those who want to understand more of the thinking that went into the vision. This allows you to shed complexity from your vision document without sacrificing context.

  Narrative: Once you’ve written the previous sections, the last step of writing a compelling vision is to synthesize all those details into a short—maybe one-page—narrative that serves as an easy-to-digest summary. In your final document, this is probably the second section, following the statement.

  Put all these pieces together, and you’ve crafted a document that is a guiding hand to align decisions yet still creates room for teams to make their own choices and trade-offs along the way. You’ll know a vision is succeeding when people reference the document to make their own decisions, and you’ll know it’s struggling when decisions keep happening that don’t fit into its direction.

  Compared to strategies, there is more room to play with the vision format. You’ll be reading far fewer visions than strategies, so consistency is less important. Play with the content a bit to find what works best for you.

  A few additional tips that I’ve found especially useful:

  Test the document! This is a core leadership tool, and your first version will almost certainly be bad. Write a draft, sit down with a few different folks to get their perspectives, then iterate. Keep doing this until you’ve synthesized feedback. If there is feedback you disagree with, embrace the vision as an opportunity to address conflict by explicitly acknowledging disagreements within the vision text.

  Refresh periodically. Take some time every year to refresh the vision, and prefer usefulness over consistency. If your old vision doesn’t resonate, it’s okay to start over: it’s a sign that you’ve learned a lot over the past year.

  Use present tense. This makes the writing impactful and concise, and conveys a sense of confidence about the future.

  Write simply. Often, visions are saturated with buzzwords, which turns readers off.

  You’ll likely want one vision for every complete distinct area, but no more. If areas overlap, you get the alignment value from having one unified vision; having two clearly articulated visions in one place is worse than having zero.

  Like other leadership tools, a vision or strategy document is a solution to a specific set of problems, and it’s not always useful. If your team is aligned and doing good work, time spent writing these probably won’t be too valuable.

  However, if your team is struggling to align with stakeholders, or if you’re struggling to lead a cohesive organization, these documents are exceptionally useful, fairly quick to write as you gain practice, and low risk (at worst, they get ignored).

  3.4 Metrics and baselines

  There is a moment in every company’s growth when top-level planning shifts from discussing specific projects to talking about goals. This happens recursively across each scope of leadership, as areas of accountability become too broad or complex for their leaders to consistently understand every project’s details.

  This can be a very empowering moment because goals decouple the “what” from the “how,” but it can also be a confusing transition for everyone involved: writing clear goals takes a bit of practice.

  • Defining goals

  Bad goals are indistinguishable from numbers. “Our p50 build time will be below two seconds,” or “We’ll finish eight large projects.” You’ll know a goal is just a number when you read it and aren’t sure if it’s ambitious or whether it matters.

  Good goals are a composition of four specific kinds of numbers:

  A target states where you want to reach.

  A baseline identifies where you are today.

  A trend describes the current velocity.

  A time frame sets bounds for the change.

  Put these all together, and a well-structured goal takes the form of: “In Q3, we will reduce time to render our frontpage from 600ms (p95) to 300ms (p95). In Q2, render time increased from 500ms to 600ms.”

  The two tests of an effective goal are whether someone who doesn’t know much about an area can get a feel for a goal’s degree of difficulty, and whether afterward they can evaluate if it was successfully achieved. If you define all four aspects, typically your goal will fulfill both criteria.

  • Investments and baselines

  There are two particularly interesting kinds of goals: investments and baselines. Investments describe a future state that you want to reach, and baselines describe aspects of the present that you want to preserve.

  Imagine that you wanted to speed up your data pipeline. Your goal might be, “Core batch jobs should finish within three hours (p95) by the end of Q3. They currently take six hours (p95), and over the course of Q2 they got two hours slower.” This is a well-structured goal, but it’s also incomplete because you could likely reach that goal tomorrow by doubling the size of your cluster, which is probably not a desirable outcome.

  The best way to avoid such unintended outcomes is to pair your investment goals with baseline metrics, sometimes referred to as countervailing metrics. For the data pipeline example, a few of the baseline metrics might be:

  Efficiency of running core batch jobs should not exceed current price of $0.05 per GB.


  Core batch jobs should not increase alert load on teams operating or using the pipeline, which are currently alerting twice per week.

  Baseline metrics are useful for narrowing the solution space that you explore in order to accomplish your investment goals. They are also useful for identifying when you should pause pursuing your goals and instead invest in platform quality. For example, if you were making excellent progress toward launching a new feature but site stability has regressed below your baselines, this framework provides a structure to trigger rebalancing your priorities.

  Although your baselines will often be about preserving a current property, you can also decide to accept some degradation before you want to trigger reprioritization. Perhaps you’re okay with costs increasing by 10 percent as long as your investment goals are accomplished. This kind of upfront clarity around trade-offs can be quite powerful.

  • Plans and contracts

  The most common way to use goals is during a planning process. By agreeing on the mix of investment and baseline goals for each team, you’re able to set clear expectations for a team while still giving them full ownership of how they’ll satisfy the constraints. I’ve found that you should specify as few investment goals as possible, maybe three, and that those should be the focus of planning discussions.

  You’ll probably want to identify more baseline goals than investment goals, but it’s easiest to separate them out to avoid bogging down the conversation. Ideally, baselines are carried over across planning periods, such that they frame the investment goals but don’t require too much active discussion during any given planning cycle.