The Problem with Prioritization Frameworks
1 September 2019 ‧ 5 min read
Product managers often complain that prioritization is one of the most challenging aspects of the job: there is always so much to do and so little time. It is therefore not a surprise that there are a ton of frameworks out there that are meant to help make difficult prioritization decisions, for example:
- ICE/RICE scoring (reach, impact, confidence, effort)
- Value vs complexity 2X2 matrix
- Stakeholder scoring
These frameworks are used to make prioritization decisions by classifying opportunities (or features) according to some measures, and then making prioritization decisions based on the resulting list or matrix.
In my opinion, these frameworks have two problems: firstly, the outcomes are somewhat arbitrary, and secondly, they don't solve the actual reason why prioritization is hard. In other words: they solve the wrong problem, and they solve it badly.
Prioritization outcomes are somewhat arbitrary
The reason the results of these frameworks are somewhat arbitrary is that some of the dimensions are un-knowable before the feature has been built. Of course, you know that the potential user benefit of a small tweak is smaller than the potential user benefit of a massive change—but the massive change could actually have negative value to users. And Marty Cagan’s first inconvenient truth about product applies: “at least half of our ideas are not going to work”. So confidence in the projected user benefits should be low. Of course, you can increase that confidence through research and prototyping, but usually you have too many ideas to invest in research and prototyping for each of them, so you need to prioritize with a high degree of uncertainty.
Some of the frameworks try to compensate that fact by introducing confidence as a factor. However, it's still hard to really put a number to this uncertainty. If half our (prioritized) ideas fail, then confidence in all ideas should be below 50% on average. But how low? And what's the difference between 20% and 30% confidence? It's hard to answer this objectively. Tweaking that assumption slightly will dramatically change the overall score of an idea.
What the frameworks can usually show quite easily are the “quick wins” (high value, low effort) and the “duds” (low value, high effort). However, these two types of ideas are usually easy to spot even without a framework (and unfortunately you run out of quick wins pretty fast). It's the messy middle that is murky and where tweaking a few assumptions can completely change the order of ideas and thus the resulting prioritization.
This all is not to say that the information and inputs collected for these frameworks are not useful. Having an idea of the maximum reach of an improvement idea or the rough level of confidence is valuable information for the prioritization process. However, the mechanistic scoring and picking the highest-ranking items does not make sense—it's pseudo-scientific at best, and subject to arbitrary decisions by whoever is putting together the model at worst.
A much better way of prioritization is collecting all of these inputs and then having a discussion as a cross-functional, empowered product team on what the top priorities should be. Since the uncertainty is so high for product improvement efforts, you are somewhat rolling the dice anyway. You are much more likely to deliver successful improvements to the product if the entire team buys into the idea, which a joint team decision guarantees but a framework doesn't. If team members are bought in, they bring their A-game. They contribute their ideas to the improvement effort and deliver solutions of the highest craftspersonship, because they want the effort to succeed. If they are just delivering on what fell out of a framework, that's much less likely. A well-aligned product team is worth much more than a prioritization that's slightly more optimal in theory according to some framework.
Prioritization frameworks solve the wrong problem
The bigger issue with prioritization frameworks, however, is that they aren't solving the real problem with prioritization.
Prioritization is the act of saying no. People say prioritization is hard not because it's hard to identify the most promising ideas, but because it's difficult to say no to the less promising (but still good!) ideas.
A product team steeped in their product, given a reasonably well-defined goal (such as “increase activation rates for user segment X”) and time of collect inputs and generate ideas will often have no big problem selecting the ideas that are most likely to help reach the goal.
However, overconfidence and high ambition levels, as well as the reluctance to tell stakeholders that their favorite ideas won't make it, often leads to teams taking on too many initiatives. This means half-assing too many things, not taking enough time to properly research and understand the problem, and delivering a rushed solution taking too many shortcuts. You are much less likely to deliver a great product that way.
In addition, taking on too many things also disregards Marty Cagan’s second inconvenient truth: “even with the ideas that do prove to be valuable, usable and feasible, it typically takes several iterations to get the implementation of this idea to the point where it actually delivers the expected business value”. So even if the team has enough time to deliver their prioritized ideas with the required diligence and craftspersonship, they may not have planned enough time for necessary iterations.
The solution to this biggest problem about prioritization lies in strong product leadership: product leaders should set clear, well-scoped goals for product teams, empower them to make their own prioritization decisions, and then push them to focus on fewer ideas then they normally would. This is somewhat unintuitive so I will say it again: product leaders should push their teams to deliver less, not more. Only that way will they be able to deliver improvements that really move the needle on creating user and business value.
The one benefit of prioritzation frameworks
There is one real benefit to prioritization frameworks though: they can be used to communicate why ideas aren't being pursued. If (especially senior) stakeholders contribute ideas or request product changes, a prioritization framework might be used to explain to the stakeholders why their ideas have been deprioritized. In this case, the prioritization framework is a communication tool though, not a decision tool. It is also an indicator that product teams aren't fully empowered. An empowered product team should of course be able to reason through why they have prioritized the ideas they have, but they shouldn't have to debate every idea they didn't prioritize.
Side note: Some other prioritization frameworks
A small side note: there are a few prioritization frameworks that work differently to the ones I've mentioned above, in that they are based on working directly with customers to elicit their input (as a proxy for customer value). Some notable examples are the following:
For prioritization questions at the level of individual feature-level ideas, I find these frameworks to have two issues: the first one is that they might be too heavy to apply for every micro-prioritization decision. The second issue is the “iPhone” problem: it can be hard for customers to appreciate an innovative solution until they've experienced it in action.
Both of these issues can be overcome, though, especially if this kind of input and prioritization is done not at the individual feature level but rather at the level of opportunity areas to prioritize what themes and objectives the product organization should work on. I am therefore much more positive on these frameworks than on the ones discussed above. However, they are still sensitive to how exactly the questions are phrased that customers are asked, and even in areas that score high according to these frameworks it's not guaranteed that you can discover and deliver solutions that realize that potential. The frameworks should therefore also be applied with caution and supplemented with team discussion and agreement.
If you liked this article, feel free to follow me on Twitter for more thoughts and interesting reading about product management.