This quick guide will teach you how to deal with the metrics question in a PM job interview.
This type of question assesses how you apply analytical skills to gauge the performance of a product from various angles.
We will try to be as thorough as possible but if you feel like you need more information visit myproductmentor.com for a comprehensive guide on PM interviews and being a stronger PM in general, as well as details on our mentoring programs which are by far the best way to get into product management.
Let’s move on to how to answer the actual question!
1. Product Description
You will need to start by describing the product or feature you are addressing, making sure you show the interviewer that you understand it and the problem it solves. In the process, you should make a list of all your assumptions which will help narrow down the question as well as ask any questions that help you clarify the interviewer’s question.
This stage is all about getting on the same page as the interviewer for what will be measured.
For example, the interviewer might ask you what would be a good way to measure the success of Facebook Likes. Your answer could be that likes are a way for users to show their interest in a post and bookmark content that they like. On the other hand, it helps Facebook gain a general understanding of what users are interested in which should yield more revenue.
To narrow down this question, you can let the interviewer know that you will concentrate on the likes that users give posts and comments rather than to Facebook pages or ads.
2. Product Goal Definition
When defining the goal of the feature or product you should bear in mind that what is being measured is the success in reaching a certain goal, so it’s paramount that you define this goal.
It may be that the interviewer already provides it – such as in a question like ‘How can you gauge the success of Facebook likes when it comes to making users more engaged?’.
You need to remember that a product or a feature might have several goals. In our Facebook example, likes could both make users more engaged but also drive up revenue for Facebook.
3. User Journey Mapping
In this step, you will have to think about what behaviors by the users influence a feature’s or product’s success (this product will be discussed in step four)
At this point, you can either go ahead and walk the interviewer through the user journey or ask them for some time to write it down as a reference point for step four.
4. Metrics Listing
Here you will need to discuss the behaviors that have an impact on your goal and map them to metrics that aid you in measuring the success of the product or feature when focusing on the user journey.
It can be helpful to remember what phases the user goes through:
- User Awareness
- User Acquisition
- User Activation
- User Engagement
- User Retention
- Monetization of the Product
- Referral by User
Remember to pick out only those phases that are relevant to the particular feature or goal that you are discussing.
At this point you might be asking yourself ‘How many metrics do I need to list?’. As a rule of thumb, you should have 1-3 metrics per phase of the user journey or, if you narrow your focus to one key phrase, a minimum of five.
Let’s examine each phase.
4.1 User Awareness
This is step number one in the journey since a lack of awareness of a feature leads to the user not using the said feature. Therefore, you should think about awareness metrics which can indicate the level of awareness of either a product or a feature.
This is the first step of a user journey. If users are not aware of a feature that won’t use it. So consider awareness metrics that measure the level of awareness of a feature or product.
If this phase is important to your product you should delve deeper and examine the performance of different channels for bringing awareness.
In the Facebook Like example, we can assume that the majority of users on Facebook know what the like is and how to use it so there is no need to go into details.
4.2 User Acquisition
User acquisition tracks how interested the users are in a product or feature; metrics that could be of use here are how many sign-ups you have, how many downloads, and how many clicks to start.
User acquisition indicates how successful the product or feature is in getting users’ attention. It can also point to the size of the opportunity.
4.3 User Activation
The inability to convert the user you acquire into activated users indicates that users may be getting discouraged at some point from activating the product or the feature. Activation metrics will aid you to understand the onboarding flow.
In our Facebook Like example, you can choose the percentage of users who liked a post at least once as the activation metrics
4.4 User Engagement
This phase is important and you will already be familiar with what it means.
We have many engagement metrics to choose from when it comes to Facebook likes – for example, the number of likes/users in a certain period, which can indicate their level of popularity.
The average number of likes/type is another option, with ‘type’ being defined as either media types (video vs audio vs image vs text) or as areas of interest (technology, food, politics, etc.)
Some other engagement metrics could be centered on determining whether you are reaching the goal defined at the start. You can ask yourself whether users whose number of likes increases spend more time on Facebook or whether a user who receives more likes then posts more often. Alternatively, you can try to find a correlation between a user’s number of likes and their number of comments and whether an increase in the former also leads to an increase in the latter.
4.5 User Retention
Again the importance of this step varies depending on context.
For example, in the case of service-based products user retention significantly impacts the company, but in the case of Facebook likes, its impact on revenue is not as significant.
You can bear in mind the following retention metrics: the way user reviews are distributed, the retention rates of the product, and the number or percentage of users with repeat purchases.
One metric you can suggest in the case of Facebook likes is the percentage of users in the past month that have liked a certain post, comparing it to the month before to determine if and how many users you are retaining.
4.6 Monetization of the Product
E-commerce sites and service-based products care more about revenue metrics, but in the case of features like Facebook Like the impact of revenue is not as crucial – if engagement on the core product is increased you can consider yourself successful.
But if you’d still like to discuss some monetization metrics, you could examine the extra revenue that users who like more posts are generating.
If you get a question where revenue is important, the most common examples of metrics are percentage of page users, average revenue/user, frequency of purchases, and revenue from ads.
4.7 Referral by Users
When users are excited about a product they will generally tell their friends about it so referral indicates the success of the said product as well as how useful it is to the user.
Some examples of referral metrics are the percentage of converted referrals or the percentage of users referring the product.
5. Metrics Evaluation
After you have the list of metrics, you should find some meaningful criteria which you can use to evaluate them. This could be the impact on the confidence of users, how accurate the metric is, how difficult it is to collect it or how relevant it is to the mission of the company.
You should have a table, where the metrics are the rows and the columns are the criteria you use for evaluation.
In our previous example, it is harder to determine how much of an impact the Like has on secondary users and we have already determined that revenue metrics are less important.
6. Metrics Prioritization
Here you can concentrate on the number of likes/post, the number of likes/user, or how the number of likes/user impacts how much time they spend on Facebook. You can use these metrics to ascertain the success of Facebook likes.
We tried to provide you with a thorough way of answering questions about product metrics. Now you need to cement these through practice, either with peers or even better with experienced PMs, which you can find on our website.
