Why “Outcomes Over Outputs” is Flawed Advice
4 October 2020 ‧ 8 min read
“Outcomes over Outputs” is a mantra that has swept the product management and product development community over the last couple of years, even leading to a major book outlining the concept. At the core of the paradigm is the insight that just pushing teams to deliver ever more output (= shipping more product features) is not a good way to build great products—you need to focus on outcomes for the customer and in turn for the business instead.
In this article, I will outline a few ways in which this advice is at best incomplete and at worst flawed, and describe ways in which to improve the concept. Don't get me wrong: I'm not advocating the return of a singular focus on output. In reality, we need to add more perspectives, not shift from one to the next.
Rebalancing outcomes and outputs
The mantra “outcomes over outputs” stems from the realization that a myopic focus on output or “velocity” of shipping product improvements is both unhealthy and ineffective. Output focus promotes a mindset and culture of a “Feature Factory”, in which product development teams resemble an assembly line that as quickly as possible delivers produces features that were conjured up by someone else.
The Feature Factory model is unhealthy because it hurts morale and motivation. Constant pressure to deliver faster along with little say over what is coming down the pike makes team members feel like they are cogs in a machine that's being run at its limit.
An even bigger challenge is that the model is ineffective. Unfortunately, most of our (feature) ideas will either not deliver value or even destroy value, and we don't know up front which ones will be the winning ideas. So the Feature Factory ends up churning out a bunch of features that don't even deliver value, while burning out the team in the process.
For these reasons, the mantra Outcomes over Outputs was born. This manta is based on the realization that producing features is not the end goal—achieving outcomes is. These outcomes are value that is realized for the user and customer and/or value that is extracted for the company in the form of revenue.
Outcomes over outputs means that the main way of measuring success and of managing teams is through outcomes. Instead of allocating features to delivery teams and measuring success in terms of how fast they can deliver them, desired outcomes are allocated to teams and success is measured by the extent to which the outcomes have been achieved.
The advice to focus on outcomes over outputs has its merits, of course. Focusing myopically on outcomes only has real drawbacks, though. Output remains important. In general, the only way to realize outcomes is by producing output. (There are exceptions—occasionally you can improve outcomes by removing something, simplifying the product and removing value destroying parts, but that requires output to have been created first.) Outcomes over outputs tells us that we should make sure the output we create is the right output, but we can't stop producing output.
Even more challenging: we don't know up front what the right output will be. Most of our ideas fail, and we don't know in advance which ones will fail and which ones won't. If you understand “outcomes over outputs” to mean that one perfectly executed idea beats many poorly executed ones, then you may be right in theory, but in practice you are probably in for a rude awakening. If you only pick one idea and aim to execute that perfectly, the odds are that the one idea you picked will fail to realize the hypothesized value.
As with any other creative endeavor, the best way to produce great work in product development (= great outcomes) is to produce more work (= high output). This has to do both with practice and variability. The practice point is obvious: the more you build, the better you get at building. The variability point is related to the “most ideas fail” point above: you will need to try many different things in many different ways with many different iterations to produce one version that performs great.
Of course, output in this case doesn't refer only to shipped product changes. Prototypes, user research, and AB tests can all be considered output that helps both practice and identify ideas that will actually have an outcome. Running more of these small validations will help find the great ideas and weed out the poor performing ones. Less output of this kind will mean the team will struggle to do so. So this means: more output (of the right kind) = better chances of achieving the desired outcomes:
So, while realizing outcomes certainly should be the desired end result and the ultimate success criterion, producing high output is often the best way of getting there.
In a way, this makes the advice “outcomes over outputs” similar to the advice “work smarter, not harder”. Yes, it sometimes needs saying. Yes, it sounds obvious, but can be incredibly hard to realize in practice. And the highest performance is achieved by doing both: work smart AND hard, or focus on achieving outcomes AND high output.
Inputs over outcomes
So, we've understood that output can't be neglected, but outcomes are still the ultimate success criterion. All is well, then, right? Not so fast. Managing by outcome has its inherent challenges as well. Not all desired outcomes are easily measurable, and even with the ones that are, you risk short-termism in which outcome metrics are artificially inflated through unsustainable efforts.
The canonical example for this issue is what often happens in social products: Long term user retention is the desired outcome. (It's generally valuable for the business if users are retained, and also understood to be a proxy for the user getting value from the product. After all, if they weren't getting value, why would they keep returning?) Long term retention is of course impossible to measure in the short term—to measure what proportion of users will be retained after a year, you have to wait a year. So instead, a leading indicator is measured that is known to be correlated with retention. Often, that leading indicator will be something related to engagement: it is often statistically provable that someone who is more deeply engaged with the product today is more likely to be retained. However, short term engagement is easily “gamed” without real value to the user: sending lots of notifications, for example, will likely drive users back to the product more often, but the long term value is questionable.
Generalizing this problem a bit, the longer the time horizon in which the outcomes are expected, the more difficult it is to measure the outcomes in a way that provides timely and actionable feedback that you can manage the process with—and often, the more innovative a new product is, the longer it will take for the outcomes to materialize. In this context, it is enlightening to read how famed venture capitalist Marc Andreesen thinks about venture capital investment decisions, the ultimate long term innovation play:
We're basically all about inputs. It's basically process versus outcome. […] Venture capital is too elongated an activity. We don't really know whether something is going to work or not work in the first five years of its life after we passed. […] The craft of investing as a process of separating process and outcome. We are almost entirely a world of trying to optimize the process. The outcomes come when they come 5, 6, 8 or 10 years out. […] What we’ve zeroed in on is - what's the optimal way to run the process? What’s the optimal way to run the firm? What’s the optimal way to help the entrepreneur?
This adds another aspect to the outcomes vs outputs discussion, namely input, the information and processes that are used to generate the output (which in turn realizes the outcomes):
Of course most product development efforts don't have such a long time frame. However, the more longer term something is, the more relevant it becomes to consider quality of inputs a relevant factor to manage. Often, this means that the more innovative the product or feature is, the more quality of inputs will outweigh managing by outcomes, since outcomes will be hard to measure and take a long time to manifest.
An additional reason to focus on quality of inputs for innovative products is that measuring outcomes on a first attempt might give you a false negative: your innovative idea might be directionally correct, but the implementation might be flawed, leading to unrealized outcomes. Of course, such a result should not be considered a success, but making sure that the right inputs were collected (such as qualitative research substantiating evidence that the innovative direction should have customer value) will give the team the confidence to iterate and adjust until the outcomes are realized.
In terms of product development, input means all of the data and research that flows into the design and development, as well as the process and craftspersonship in the execution of a product idea. Managing the quality of these inputs is often more of a qualitative exercise than a quantitative one and requires leadership that both demands excellence in inputs and develops the team to exhibit such excellence.
In summary: the more longer term and the more innovative a product development effort is, the more you want to put emphasis on the qualities of input and execution as opposed to measuring the outcomes and making a decision to ship or kill a feature based on measured results and metrics movements.
Completing the cycle
So we've now gained an understanding of why in order to effectively manage the development of new products and product features, we need to consider and measure inputs, outputs, and outcomes, and balance them appropriately according to the quality of the work. This way, we've set up a management system for an innovation pipeline.
However, innovation and product development is not a pipeline. It's not a linear process with inputs that flow through the process and lead to results that are disconnected from the input. A properly managed innovation and product development process requires a feedback or learning loop, connecting the results back to the inputs for the next cycle:
In addition to inputs, outputs, and outcomes, it's important to actively manage this feedback loop as well. A well-executed feedback loop is crucial in order to both build on successes of the past and to not repeat the same mistakes again and again. It sounds obvious, but developing a successful product requires identifying what works and what doesn't and then doubling down on the things that work and not trying the things again that don't. This feedback loop is typically easy when the product is developed by a stable and small team—the team members will remember all the past context and act accordingly. However, the larger and longer-lived a product development organization becomes, the more important it is that this feedback loop gets formalized and managed so that teams and individuals can learn both from other parts of the organization as well as from those that were a part of the organization before them and have since left.
Examples of activities and artifacts of this feedback loop include retrospectives (within product teams or across broader parts of the organization), documentation of research and validation results, as well as the dissemination of this knowledge through the organization, for example through presentations in all-hands meetings or self-serve knowledge bases.
It is crucial to actively manage the feedback loop and consider it part of the overall product development process. Documenting and sharing lessons learned needs to be understood as a success criterion of product development efforts. Otherwise, it's easy for teams to take shortcuts and skip documenting and sharing results, since the benefits of the more formalized feedback loop are often external to the team: who most benefits from it are people in other parts of the organization or even in the future. It can therefore feel faster to omit the formalized feedback loop in the short term, but it slows the organization down in the long term since context and lessons learned get lost and forgotten. If you are attempting to build a business that lasts, it is therefore never too early to start actively managing the feedback loop.
A side-benefit to actively managing for learning and understanding learning to be part of success is that all product improvement efforts can produce learning, even the ones (especially the ones!) that fail to produce the desired outcomes. In the long term, of course, product development needs to achieve outcomes or it's ultimately a failure, but in the short term, making sure that learning is accumulated quickly is really important to set the product up for success. You should make it feel like a success if a short-term failure (of a product improvement effort) leads to substantial learning. Making learning feel like success will both be beneficial for team morale and for the long term prospects of producing a great product through learning and iteration.
In summary, innovation and product development needs to be managed by inputs, outputs, outcomes, and learning (feedback). Optimizing for one of them at expense of the others will either lead to shipping a bad product right away or to an unsustainable model in the long term.
Great product managers and product leaders know to never lose sight of all four, and prioritize tradeoffs between the factors depending on the circumstances, for example whether the current work is more optimization focused (requiring rapid iteration, frequent output, and success determination by outcomes) or strategic innovation (requiring high-quality inputs, taking into account learning accrued in the past).
I hope you found this article useful. If you did, feel free to follow me on Twitter where I share thoughts and articles on product management and leadership.