Take-home assignments for product management candidates
13 March 2022 ‧ 7 min read
I've recently gone through the process of designing a product management interview process for RevenueCat. This is not the first interview process I've set up from scratch, but one of the questions that comes up every single time is whether a written take-home assignment should be a part of the process.
RevenueCat uses written assignments in most interview processes. However, they are not without criticism. Therefore, I wanted to be deliberate about the whether and how to ask for this kind of homework. In this article, I will walk through some of the pros, cons, and ways to mitigate the cons. At the end, I will also share what I ended up doing (spoiler alert: I did ask for a homework assignment, but it was a little bit different than the usual "design a solution" homework).
The drawbacks of take-home assignments
Let's start with all the reasons that speak against giving candidates written assignments.
Firstly and perhaps most obviously, take-home assignments require time, and often the response will be better the more time the candidate was able to invest. This biases against anyone who has less free time at their disposal – candidates with children, for example. Therefore, take-home assignments could be considered non-inclusive.
These assignments can also feel like free work. I am personally of the opinion that they never actually are free work, because the responses are unlikely to contain truly novel ideas and even if they do, the hard work is not the idea but fleshing it out and implementing it. However, the candidate might feel differently and therefore you might put off some good candidates by giving an assignment.
Assignments can also feel like an uneven burden. In interviews, the candidate and the interviewer spend the same amount of time on the interview. In assignments, the candidate spends much more time writing than the recipient will spend reviewing.
There is also the risk of cheating on an assignment by soliciting help, which might reduce the validity of the signal you get from an assignment.
There are also some specific points that make the responses to a written assignment not comparable with how a product manager would work in real life. Firstly, the candidate will have much less context than any real life product manager would have. Perhaps most importantly, they will not have access to customers to discover actual problems and solutions to those problems, which is arguably more important than having good-sounding ideas in isolation.
In addition, the more domain knowledge is required for (or helpful in) responding to the assignment, the more you might inadvertently filter out candidates who are great product managers (or have high potential for it) in favor of domain experts with poor product management skills.
Written assignments also completely disregard the fact that product management is a team sport. Asking a product manager to design a solution in the absence of a product designer is something that wouldn't happen in any but the smallest teams. You want to hire a product manager who can effectively collaborate with a designer to discover and design a solution, not someone who believes they have to know all the answers themselves.
The benefits of take-home assignments
Let's now look at what speaks for giving out take-home assignments.
Firstly, written assignment give better signal about how candidates think and deliberate. Interviews only give you signal on how well candidates can think on their feet. Most actual product management work, however, is done with the ability to research, reflect and form an opinion, not having to come up with solutions on the spot. In most circumstances, you would rather hire someone who comes up with great solutions with a bit of time than someone who comes up with just good ones on the spot.
Written assignments also test structuring problems and written communication. These skills are always important for product managers, but particularly so in today's remote and asynchronous work environment.
The assignment also acts as a filtering function for the candidate's interest. While this is to be taken with a grain of salt (because of the fact that different candidates may have different levels of free time available), you can say that candidates who are more interested in the domain will be naturally more diligent and thoughtful about their responses.
An interesting corollary to the previous point is that for candidates who are more interested in the domain and the problem you are solving (the “missionaries”), the assignment can actually generate additional excitement about the opportunity. These candidates will likely find working on the problem so intrinsically motivating that it gives them energy rather than draining it.
Lastly, take-home assignments can be a great way for candidates who are not currently in a product management position to show off their potential and their product sense. The best way to ask interview questions is often behavioral (“tell me about a time you…”), which can be tricky for candidates that aren't currently in a role where they can showcase a lot of product sense.
Alternatives and mitigation
Some of these drawbacks can me avoided or mitigated depending on how the take-home assignment is designed. Here are some options to consider.
If you just want to test written communication, you could just ask candidates for a writing sample, which allows them to reuse a work product they've already produced and therefore reduces the time investment that's required. Of course, this approach limits some of the benefits of the take-home assignment.
In a similar vein, you could ask candidates to give a presentation about one of their previous projects instead. This is often done in interview processes for designers (portfolio review). The biggest challenges with this approach are that product management can differ widely between different companies, and it also biases against candidates who haven't been in product management before.
Several of the drawbacks can be addressed by assigning a problem that isn't related to the company's own product, but rather a third party product. This makes it clear that the assignment isn't “free work”, and it also levels the playing field in terms of domain knowledge. However, it makes it harder to filter for candidates with enthusiasm for the space.
To reduce bias against candidates with less free time, it generally makes sense to time-box and/or length-box the assignment (i.e., “spend no more than 2 hours on this” or “submit no more than 2 pages”). Of course, you can't enforce a time-box, and a length-box is an imperfect substitute (in general, short documents are harder to write than long ones, so the more time you invest, the better a short document will get).
To make the assignment more alike real product management work, you can provide contextual information as part of the assignment that candidates have to analyze, for example, customer quotes or data points. On the flip side, the more information you include, the more time it will require from candidates to process that information before even starting to write.
More generally, it is often a good idea to limit the amount of domain knowledge required in the assignment (unless you absolutely need to hire a domain expert). Even if the assignment is about your own product, there are surely some product questions that are easier to answer without deep domain knowledge than others.
To avoid asking for activities that a product manager would never do without collaborating with their team members, consider not asking them to design a solution. Instead, you could ask for researching an opportunity, drafting a strategy, or writing up a one-pager for a mission or initiative.
Lastly, let's talk about some ideas to avoid the assignment feeling like “free work” or an uneven burden where the candidate invests a lot of time without receiving anything in return. Placing the assignment step late in the process (i.e., after panel interviews) is one way to achieve this. It does, however, reduce the leverage it provides for the process.
You can also make sure to provide detailed feedback or ensure to always have one face-to-face discussion after the assignment, even in the case that the assignment wasn't strong.
Finally, as an even more extreme measure, you could monetarily compensate people for their time spent on the assignment.
What we ended up deciding at RevenueCat
As with any tradeoff decision, there isn't a perfect solution for product manager take-home assignments. You can optimize for one aspect or another and give different weight to the various pros and cons. Here's what we ended up deciding for our recent product management role at RevenueCat.
Firstly, we did ask candidates to complete an assignment. Being a fully remote, globally distributed company, the ability to communicate well in writing was too critical a skill to not test for. We did not go for one of the alternative ways to test written communication (writing sample, portfolio review), because it makes different candidates harder to compare and it also biases against candidates who may not have something as impressive to show for reasons unrelated to their own performance.
We also chose to make the assignment about our own product, mostly for the reason of filtering for as well as spurring additional excitement for our domain. We did, however, push the assignment back in the process past the panel interview, so that it would feel more like a conclusion of the process instead of free work. Interestingly, the filtering function worked quite well in the sense that one candidate withdrew his application after seeing the assignment, stating that he didn't have as much interest in the space as he initially thought. I consider this a feature, not a bug – better to have found that out by means of the assignment than after joining the company!
In terms of the actual content of the assignment, we moved away from the typical "design a solution" prompt, because that's not something a product manager should ever do in isolation. Instead, we asked the candidates to come up with a 1-2 page proposal for how to break down and start operationalizing one of our strategic focus areas. The candidates had been given some context for the focus area in the interviews, and the assignment itself provided additional context as well. In addition to that overall prompt, we also included some guiding questions that we expected the candidates to cover (for example, “What information would you collect and how?” or “What risks do you see and how would you mitigate them?”).
The advantage of this kind of assignment, from my perspective, is that this is precisely the kind of work that I would expect product managers to do independently when tackling a new area. In real life, they would of course have more time and access to more information, but they would still have to start somewhere. This kind of assignment gives the candidates the ability to show how they would break down such an ambiguous and big problem into pieces that are small enough to handle.
Asking this kind of assignment question was a bit of an experiment, but one that turned out well so far. We will probably keep experimenting with this approach and further refine it.
Before wrapping up, I want to share one last aspect about written assignments that I have found absolutely crucial. Before looking at the first completed assignment, develop a list of questions to grade the assignment by. I don't just mean the general assessment rubric (e.g.,”clarity of communication”), but concrete and specific questions (e.g., “does the candidate identify trade-offs and do they make reasonable trade-off decisions?”). This allows a more objective assessment and comparison between different candidates' responses. It also ensures that compelling writing doesn't paper over the fact that a candidate hasn't covered all crucial aspects of the assignment.
I hope you found this article useful. If you did, feel free to follow me on Twitter where I share thoughts and articles on product management and leadership.