by Marty Cagan
Business analytics (active users, conversion rate, lifetime value, retention)
Financial analytics (ASP, billings, time to close)
Performance (load time, uptime)
Operational costs (storage, hosting)
Go‐to‐market costs (acquisition costs, cost of sales, programs)
Sentiment (NPS, customer satisfaction, surveys)
Hopefully, you can see the power of analytics for product teams. However, as powerful as the role of data is for us, the most important thing to keep in mind about analytics is that the data will shine a light on what is happening, but it won't explain why. We need our qualitative techniques to explain the quantitative results.
________________
Note that we often refer to analytics as key performance indicators (KPIs).
Flying Blind
Remarkably, I still encounter too many product teams that either aren't instrumenting their product to collect analytics, or they do it at such a minor level that they don't know if and how their product is being used.
My own teams—and every team I can think of that I've ever worked with—have been doing this for so long now that it's hard to imagine not having this information. It's hard for me to even remember what it was like to have no real idea how the product was used, or what features were really helping the customer, versus which ones we thought had to be there just to help close a sale.
This is easiest to do with cloud‐based products and services, and most of us use web analytics tools, but sometimes we use homegrown tools for this too.
Good product teams have been doing this for years. And, not just with cloud‐based sites, but also with installed mobile or desktop applications—on‐premise software, hardware, and devices that call home periodically and send the usage data back to the teams. A few companies are very conservative and ask permission before sending the data, but mostly this just happens silently.
We should all be anonymizing and aggregating the data so there's nothing personally identifiable in there. Occasionally, however, we see in the news that another company is in trouble for sending raw data in the rush to market. It seems the press thinks we're tracking these data for nefarious purposes, but at least with the companies I know and work with, they're simply trying to make the products better—more valuable and more usable. This has long been one of our most important tools for doing so.
The way the overall process works is that we first ask ourselves what we need to know about how our products are used, then we instrument the product to collect this information (the particular techniques depend on the tool you're using and what you want to collect). Finally, we generate various forms of online reports to view and interpret these data.
I still encounter too many product teams that either aren't instrumenting their product to collect analytics, or they do it at such a minor level that they don't know if and how their product is being used.
For everything new we add, we ensure we have the necessary instrumentation in place to know immediately if it is working as we expect, and if there are significant unintended consequences. Frankly, without that instrumentation, I wouldn't bother to roll out the feature. How would you know if it was working?
For most product managers, the first thing they do in the morning is to check the analytics to see what happened during the preceding night. They're usually running some form of test almost all the time, so they're very interested in what's happened.
There are of course some extreme environments where everything lives behind very strict firewalls, but even then, the products can generate periodic usage reports to be reviewed and approved before being forwarded (via electronic or printed reports, if necessary) back to the teams.
I'm very big on radically simplifying products by removing features that don't carry their own weight. But, without knowing what is being used, and how it's being used, it's a very painful process to do this when you don't know what's really going on. We don't have the data to back up our theories or decisions, so management (rightfully) balks.
My personal view is that you should start from the position that you simply must have this data, and then work backward from there to figure out the best way to get it.
CHAPTER 55
Testing Feasibility
When we talk about validating feasibility, the engineers are really trying to answer several related questions:
Do we know how to build this?
Do we have the skills on the team to build this?
Do we have enough time to build this?
Do we need any architectural changes to build this?
Do we have on hand all the components we need to build this?
Do we understand the dependencies involved in building this?
Will the performance be acceptable?
Will it scale to the levels we need?
Do we have the infrastructure necessary to test and run this?
Can we afford the cost to provision this?
I don't want to scare you. With most product ideas that your engineers review in discovery, they will quickly consider these points and simply say “No problem.” That's because most of our work is not all that new, and engineers have usually built similar things many times before.
However, there are definitely ideas where this is not the case, and some or many of these questions can be very difficult for the engineers to answer.
One very common example right now is that many teams are evaluating machine‐learning technology, considering build/buy decisions, and assessing whether the technology is suitable for the job at hand—and, more generally, trying to understand its potential.
Here's some very practical and important advice for you to consider. Holding a weekly planning meeting where you throw a bunch of ideas at the engineers—and demand they give you some sort of estimate either in time, story points, or any other unit of effort—is almost certain to go badly. If you put an engineer on the spot, without time to investigate and consider, you are very likely to get a conservative answer, partly designed to make you go away.
If, however, the engineers have been following along as the team has tried out these ideas with customers (using prototypes) and seen what the issues are and how people feel about these ideas, the engineers have probably already been considering the issues for some time. If it's something you think is worthwhile, then you need to give the engineers some time to investigate and consider it.
The question isn't, “Can you do this?” Rather, you are asking them to look into it and answer the question, “What's the best way to do this and how long would it take?”
The engineers will sometimes come back and say they need to create a feasibility prototype to answer one or more of these questions.
Many of our best product ideas are based on approaches to solving the problem that are only now possible, which means new technology and time to investigate and learn that technology.
If that's the case, first consider whether the idea is potentially worth investing the necessary time in discovery. If so, then encourage the engineers to proceed.
One last point on assessing feasibility: I meet many product managers who hate any product idea that the engineers say they need additional time to investigate. To these product managers, this means it is already too risky and time consuming if that happens.
I tell these product managers that I personally love these items for a few reasons. First, many of our best product ideas are based on approaches to solving the problem that are only now possible, which means new technology and time to investigate and learn that technology. Second, I find that when engineers are given even a day or two to investigate, they often come back not only with good answers to the feasibility question but also with better ways to solve the problem. Third, these sorts of items are often very motivating to the team, as it gives them an opportunity to learn and to shine.
Discovery for Hardware Products
So many technology‐powered products today have a hardwa
re element within them. From phones to watches to robotics to cars to medical instruments to thermostats, smart devices are all around us.
With hardware, the consequences of a mistake in terms of time and money are much more severe.
So how does adding hardware to the equation affect everything we've discussed thus far?
There are some obvious differences, such as different engineering skill sets, the need for industrial design, and of course, manufacturing still takes substantially longer than software, although it continues to improve.
For the most part, however, everything we have discussed thus far still applies, although there are some additional challenges as well. Moreover, when hardware is a part of the equation, the discovery techniques we've discussed are even more important, especially the role of prototyping.
The reason is because, with hardware, the consequences of a mistake in terms of time and money are much more severe. With software, we can usually issue corrections relatively inexpensively. With hardware, no such luck.
Specifically, there are more technical feasibility risks with hardware, and there are additional business viability risks. For example, with hardware we need a much more sophisticated analysis of parts, manufacturing costs, and forecasting. That said, the necessary prototyping of the hardware device has been helped dramatically with the advent of 3D printing technology.
The bottom line is that hardware products require tackling the value, usability, feasibility, and viability risks aggressively and raising your bar on the level of confidence you have before you commit to manufacturing.
CHAPTER 56
Testing Business Viability
There's no question that it's hard enough just trying to come up with a product that your customers love and your engineers can build and deliver. Many products never get to this point.
However, the truth is this is not enough. The solution must also work for your business. And I will warn you now that this is often much more difficult than it sounds.
Many product managers confess to me that this is the least favorite part of their job. While I understand that, I also explain to them that this is often what separates the good product managers from the great ones, and that more than anything else, this is what is really meant by being the CEO of the product.
Building a business is always hard. You must have a business model that's viable. The costs to produce, market and sell your product must be sufficiently less than the revenue your product generates. You must operate within the laws of the countries you sell in. You must hold up your end of business agreements and partnerships. Your product must fit within the brand promise of your company's other offerings.
This is what is really meant by being the CEO of the product.
You need to help protect your company's revenue, reputation, employees, and customers you've worked so hard to earn.
In this chapter, I name the main stakeholders in a tech‐powered product company, discuss their typical concerns and constraints, and explain how the product manager would test business viability with each area.
While this is a very common list, and most or all of these areas probably apply to your products, it is also very common that any company will have one or more special stakeholders that are unique to the business. Just because it's not listed below does not mean it's not absolutely critical for you to deal with.
The last thing you want to have happen is that your team moves forward and takes the time to commercialize the solution and deliver a shippable product, only to find out that you can't ship because you are violating one of these constraints. Make no mistake about it, when that happens, it's on the product manager. It is your job to ensure that you understand each of the relevant constraints, and take positive action to work within them.
Marketing
We've already discussed product marketing, which we view more as a member of the product team than as a stakeholder. But, more generally, marketing cares about enabling sales, they care about the brand and reputation of the company, and they care about market competitiveness and differentiation. Marketing needs the resulting products to be relevant and compelling, and work with the go‐to market channels. So, anything that you're considering that puts those at risk will be a major concern.
If what you are proposing to build could impact the sales channel, the major marketing programs, or is potentially outside of the brand promise (the range of things your customers expect from your company), then you'll want to discuss this with marketing and show them prototypes of what you are proposing before you consider building anything. Work with them to find ways to address their concerns.
Sales
If your company has a direct sales organization or an advertising sales organization, then this has a very big impact on the product organization. Successful products typically need to be designed around the strengths and limitations of the sales channel.
For example, a direct sales channel is very expensive, and this requires a high‐value product and price point. Or, you may have built up a sales channel with a certain set of skills, and if your new product requires a very different set of skills and knowledge, your sales force may completely reject the product.
If what you are proposing would represent a departure from what the sales channel has proven their ability to sell, sit down with the sales leadership and show them what you are proposing before you build anything, and see if together you can figure out a way to effectively sell this.
Customer Success
Some tech companies have what's referred to as a high‐touch model of helping their customers, and some have a low‐touch model. You need to understand what your company's customer success strategy is, and you need to ensure that your products are aligned with that strategy.
Again, if you are proposing something that would represent a change, you'll want to sit down with leadership and discuss the options.
As a side note, if you have a high‐touch service model, these people are exceptionally helpful for product insights and prototype testing.
Finance
Finance often represents several different constraints and considerations, not the least of which is whether you can afford to build, sell, and operate your new product. But, business analytics and reporting are often in finance, and investor relations and other concerns may bring their own set of constraints.
If there are cost issues involved, sitting down with someone in finance and modeling the costs will be critical to demonstrating to leadership that you have worked out a viable approach.
Legal
For many tech‐powered companies, especially those that are working hard to disrupt markets, legal can play a very significant role. Privacy concerns, compliance concerns, intellectual property, and competitive issues are all common constraints related to legal. You can save yourself a whole lot of time and grief by sitting down early with someone from your legal team and discussing with them what you are proposing and whether they anticipate any issues or areas you should be aware of.
Business Development
Most businesses have some number of close business relationships with partners of various types, usually with a contract behind each that has a defined set of commitments and constraints. Sometimes these agreements can cripple your company's ability to compete. Sometimes they are a huge win. In either case, you need to understand the impact of these relationships on your products and what you are proposing to do.
Security
We would normally think of security not as a stakeholder, but more as an integral part of the engineering organization and hence a part of the product team. However, the issues involving security are so important for so many technology‐powered companies that I think it's useful to call the area out. If you are proposing anything at all remotely related to security, you will probably want to bring your tech lead and sit down with the security leadership early to discuss the ideas and how you will address their concerns.
CEO/COO/GM
Of
course, with every company there is some CEO or general manager that is responsible for the business unit. They are very likely aware of all these constraints, and normally they are worried about them. And if the product manager is not also aware of the issues, or does not have a plan for dealing with them, the exec is not going to trust the product manager or product team.
It doesn't take long for a CEO to figure out whether a product manager has done her homework and understands the different aspects of the business.
Testing business viability means making sure that the product solution that your team is proposing will work within the constraints of each of these areas. For those stakeholders that are impacted, it's important that they have a chance to review the proposal and ensure their concerns have been addressed.
User Test versus Product Demo versus Walkthrough
Throughout this book, I have talked about “showing the prototype.” In truth, there are three very different techniques for showing the prototype, and you have to be careful to use the right technique for the right situation.
A user test is when we test our product ideas on real users and customers. It is a qualitative usability and value‐testing technique, and we let the user drive. The purpose is to test the usability and value of the prototype or product.
A product demo is when you sell your product to prospective users and customers, or evangelize your product across your company. This is a sales or persuasive tool. Product marketing usually creates a carefully scripted product demo, but the product manager will occasionally be asked to give the product demo—especially with high‐value customers or execs. In this case, the product manager does the driving. The purpose is to show off the value of the prototype or product.