The first question I would ask is: what does “success” mean? Is it aimed at increasing user satisfaction, or usage/engagement/increased ad revenue? My assumption is that the answer is “yes” – at the core of a successful feature is the personal and social value for the users which ideally also generates profitability for Facebook and its advertisers. (A win/win/win).

A lot of telemetry can be measured, but in order to gain the most valuable insights, let’s start with the value for the users. Here are some reasons why they might want to make use of the save function:

-1- Ability to return to a certain post at a later time to consume or share/react/comment
-2- Ability to browse more content faster, while marking (=saving) a subset of interesting posts for more focused consumption/interaction
-3- Ability to collect posts related to certain themes/topics
-4- Ability to collect posts from certain friends or groups (what happens when a friend deletes a saved post from their wall??)

The assumption is that the save function is successful if it enables the above listed abilities. In the absence of actual user feedback, the combination of the following measurements are good proxies to assume that the save function is successful:

a. #saves per user per Facebook visit
b. #of returns to saved content per user
c. #interactions with saved content per user (or per saved piece of content)
d. #saves per amount of content scrolled through per user and FB visit

High numbers in a. plus any of b/c/d are a good indicator that a user is likely satisfied with the feature.

However, we also need to look at how many Facebook users (with regular activity on FB) have never used the save function. A high percentage would indicate a possible discoverability (awareness) issue.
If such an issue had been identified and addressed, then a decrease in the population which has never used the function is another measure of success.

If, furthermore, good analysis of what individual users tend to save helps surface more similar content in their newsfeed, we might be able to measure increased engagement with the newsfeed on content which had been better personalized based on previous save actions. Any such engagement can also be considered an (at least indirect) success metric for the save feature. Relevant measurements would be:
– #reactions/shares/comments/active consumption (e.g. play video) for content suggested based on a user’s saved list (even if it does not lead to added saves, which would be counted in a. above)

Other success metrics could be derived from non-instrumented data sources:
– actual user-feedback (e.g. one-question pulse queries on individual FB features)
– counterfactual testing (what if for a population of FB users the save function would be removed; how many complaints would we get?)
– Mining of any mention of the FB save feature in FB posts, comments, or FB-related user forums

The cost of deriving the last 3 suggested metrics is not likely justifying the result. Especially the first two suggestions have the potential to irritate the impacted users, while the ROI for the third is likely low.

I would therefore focus primarily on the readily available, yet relevant, logged usage telemetry described under a.-d. above, secondarily on the “indirect” success metrics, and furthermore measure if we can confirm any correlation between engagement with the save feature and increased # of visible and/or clicked ads. The latter measurement would be a metric related to the successful generation of financial profitability for the advertisers.

Alright so before proceeding with the answer to the questions i have a couple of clarifying questions that i need to ask.

Step 1: Ask Clarifying Questions

Are we considering any particular objective in mind while defining the success of the save feature or is it something that i can assume. For example is the company objective to increase user adoption or user engagement or revenue.

Are we talking about the entire product or only a part of the product or is that something that you’d like me to define

Step 2: Talk and explain your understanding of the product.

The mission of facebook is to connect people across the world and bring them closer to each other.

Save feature is used to save content in the form of videos, articles, posts etc which can be used for later reading and consumption.

If a person saves any piece of content then it gets saved in the saved folder, person can create mulitple folders and name the folder depending on the type of content they are saving. Used for organising content also this feature is helpful when someone doesn’t have time to go through a piece of content and want to do it in the future.

Save feature has been out for a while and gotten a very good adoption rate, however not everyone uses this feature. Therefore we can say that the feature is in its growth phase.

Broad level target audience for facebook save button are Consumers (Brands, Individuals who consume a lot of content and if they like any sort of content and would like to go back to in the future they save and bookmark it using the save feature)

If you save a particular piece of content other’s wont be able to see but it helps facebook in making better decision regarding the kind of content you like and thus shows more personalised and related content to the user giving them a personalised and improved user experience.

Step 3: Identify and define  about the goal of the product

Facebook save is part of the facebook ecosystem thus since its a very huge feature it can have multiple goals ranging from Driving user engagement as more saves would mean that people are liking and enagging more with the content, which will help facebook show better personalised content which will driver user experience. Driving engagement amongst users will lead to revenue growth for facebook

Thus these are the 3 goals that we have in mind for this product, however for this particular question sake I’ll be going with Increasing/User engagement as the main goal of this feature as this is aligned to the mission and vision of facebook of bringing more people closer across the world.

Step 4: Go through the user journey (user actions) that will drive user engagement of facebook

1) User goes on facebook newsfeed

2) User sees a lot of content, but doesn’t have much time to perform any action

3)User saves the content that he/she likes for future reference

4) Once user has time user comes back to the saved folder and engages with the content that he/she had saved earlier.

5) User can engage with this content in different ways, like, comment, tag someone or share that content with people in their network.

Step 5: List down the metrics that are aligned towards the end goal. (Since our goal is to drive user engagement some of the metrics that we will choose is)

1) % of total posts saved -This will tell us about the adoption rate of the save feature as how many people are using the save feature actively.

2) Average number of likes and comments per saved post on a daily, weekly and monthly basis- This is a very good metrics as it talks about the user engagement on a broader level

3) Average Number of users that use the save feature on a daily, weekly and monthly basis- This metric means is a good metric as well since if the adoption rate will increase chances of engagement with the feature and product will increase as well

4) Average number of posts saved per 1000 users- Same as 1 this will tell us about the adoption rate of the save feature

5) % of users who have saved and engaged (like, commented or shared) with atleast 3 posts in a week’s time- This is a very good metric as well as it talks about and showcases user engagement in a limited timeframe

Step 6: Now we will evaluate and prioritize the metrics based on certain success criterias such as

Relevance to the end goal and company mission, Impact to End User and effort required to collect and measure these metrics.

Based on the above description of the metrics that were listed down in step 5, the 2 metrics that i think would be the best fit to measure success of Facebook save to drive user engagement are as follows

1) Average number of likes and comments per saved post on a daily, weekly and monthly basis

and 2) % of users who have saved and engaged with atleast 3 post in a span of a week.

Step 7: Summarize your Answer

This will just be the summary highlight the goal of the feature, metrics we listed down and the metrics we adopted.

Thank you

Define & Clarify
I’d first want to clarify and define what the FB Save Feature is. The Save Feature allows users to save favorite posts from their news feed for later consumption. Users can save a post by clicking Save from the drop-down menu in the upper righthand corner of a post. Once a post has been saved, you can view it later by accessing the “Saved” page in the lefthand sidebar. The saved posts are organized by categories.

At it’s core, the Save Feature solves for two main user problems:

• Users don’t have time to read favorite posts in one session.
• Users can’t find favorite posts during a repeat user session.

Set Goal
The success of this features depends on what my goals are as a PM. There are two types of goals I’d would evaluate:

Value Added to User 

• Help users read their favorite posts in multiple sessions.
• Help users easily find their favorite posts during a repeat user session.

Value Added to Company

• Rev: Increase monetization opportunities
• Strategy: Diversify into the bookmark market.

I’d prioritize focusing on the user experience goals since it’s not wise to monetize features that don’t add value to users.

List User Actions
Before diving into metrics I’d first want to identify the user actions for the Save Feature because I can’t define the metrics without knowing what I should be measuring:

• User scrolls through FB newsfeed and clicks on ‘Save’ for the posts he wants to view later.
• User returns for repeat FB session at a later time.
• User accesses the Saved page from the sidebar
• User scrolls through the Saved content under the All tab
• User clicks on the category tabs to find the saved content
• User clicks through to read the Saved content
• User likes the Saved post
• User comments on the Saved post
• User Shares the post
• Poster navigates back to the Saved page to view more bookmarked content

Identify Metrics
Now I’d translate the user actions to quantifiable metrics:

• Avg # Saved posts per user
• Avg # page views of the Saved page per user
• % of users who viewed a Saved post after saving the post.
• Avg # likes of Saved posts per user
• Avg # comments of Saved posts per user
• Avg # Shares of Saved posts per user

Evaluate
To identify the most important metrics for success, I’d map the metrics back to my goals as a PM of this feature.

1. Primary Goal: Help users read their favorite posts in multiple sessions

• MetricA: Avg # Saved posts per user session
• MetricB: Avg # views per Saved posts per user within a 90 day period.

2. Secondary Goal Help users easily find their favorite posts during a repeat user session.

• MetricC: Avg # page views of the Saved page per user

The most important metric I’d focus on is MetricB — Avg # of views of Saved posts per user within a 90 day period. MetricA and MetricC are means to get to MetricB. I.e. Users would save posts and list the Saved page in order to read their favorite content at a later time.

I will answer this Facebook metrics question by taking these steps:

 

• I will clarify what the feature is.
• Then I will explain what our business goal is with this feature.
• Next I brain storm the metrics.
• I prioritise the metics
• At last I summarise.

Clarify: Save Feature allows users to to Save Links, Pages, Posts, Locations, Movies, etc to view later. FB also reminds users about what items they have saved.

This feature affects users and marketer. Marketers do not want to be forgotten, so if they post something that attracts the attention of the user, they want the user to be able to find it again later, if they don’t have immediate time to spend on it. For example, if there is a nice shoe advertised on FB and user likes it, but cannot check it now, or there is a discussion about a TV series that the user potentially finds interesting to watch later, the user can save it to check it out later.

 

User Goal: The benefit for user is that they do not need to copy paste or make screenshot of thing that they want to checkout later. They can have all these items in a categorised way (e.g. Movies, Pages) and can check them later.

 

So I as FB expect that Save Feature

• Marketer Goal: increases revenue for marketers by increasing clicks and impressions
• BIZ Goal: increases user engagement
• BIZ Goal: Increase FB revenue by increasing CTR, and CPC and CPM. Because user might make a click that he would not have done otherwise if he could not save the post. So the goal is to increase CTR and consequently the revenue.

 

Metrics to measure the success:

1. Discoverability: %of users the have at least once Saved an item. –> User knows that he can Save items
2. Discoverability: %of users from (1) that have at least once opened Saved Page –> User knows where to find Saved items and knows how to work with it
3. Discoverability: %of users from (2) that at least once engage (metric 11, 12, 13) with at least one item in Saved Page
4. The avg amount of time it took user from Saving an item to opening it again
5. %of users that engage with an item in Saved page too late, so that the item is expired already (difficult to implement, how would the algorithm know if the page/link/… does not have the value it had at the time it was Saved?)
   1. Examples: it has been an offer for a discount/coupon that was only 1 week, or it has been a event would have been streamed live in an FB page
6. #of times a user clicks to Save an item vs. the #of items the user visits (% in 1 month)
7. #of times the user opens the Save page vs. the #of times the user visits FB (% in 1 month)
8. #of categories in Save page the user goes through to see what items are saved in each category
9. the amount of time user spends just looking at items in Save page
10. distribution of #items Saved in each category, e.g. Links, Movies, TV, Locations. —> This can help us to clean our list of categories and remove the noise (i.e. categories that are barely saved) or add new categories based on what is Saved in what category, but our algorithm tells us the category does not match.
11. engagement: %of Saved items that the user opens from Saved page
12. engagement with the items that are opened from Saved page, i.e.
    1. #likes
    2. #shares
    3. #clicking links
    4. #playing videos
    5. #comments
13. engagement: amount of time spent on a page, after opening it from Saved page.
14. %of Saved items that the user deletes without engaging with or opening them
15. %of Saved items that the user deletes after engaging with or opening them
16. retention: reaction to reminders from FB about items in Saved page,
    1. #of times user clicks on reminder to see Saved items.
    2. #of times he “engages” with items after getting the reminder.
17. retention: change in the # of items that user Saves every month .
18. Are we cannibalizing?  —> Is there a decrease or increase in likes, shares, comments, #of posts after pre-launching this feature for test group?
19. User Segmentation: What is the user demographics for those who engage with this feature and those who do not
    1. Age
    2. Gender
    3. Mobile/desktop
    4. Location
20. User Segmentation: What type of users use this feature most often? Power users? Passives? Medium users?
    1. If we have a problem in Discoverability, can we encourage power users to write about this feature in their posts?
    2. Shall we begin the testing phase of the feature with power users to fine tune it and do A/B testing?
21. What features items have that are most Saved?
    1. Item type (page, link, movie, location)
    2. What is in the item?
       1. amount of text
       2. video
       3. image
    3. They have a lot of comments, shares, likes
    4. They are shared/liked/commented by a friend
       1. How close is the friend?
       2. what has been the type of the friend’s reaction to this post? Like? Share? Comment?
       3. How many friends have reacted to it before I save it? –> Hypothese: It makes a difference for user if 2 people he knows have liked sth vs. 100 strangers.
    5. They have a high similarity to what user already interacts with: Hypothesis: We can use this information to suggest to users to Save an item, but needs to be implemented carefully to not to cannabalize active engagement of users with the hope of an engagement at a later time point.
    6. They have a high similarity to what user sees in his News feed everyday, but does not react with –> Hypothese: User has been potentially interested in these items, but did not have the time to interact with them? We can also suggest users to Save items that he rarely interact with and see what users reactions is? Does he Save them? Does he go back to see them?
22. If we adjust News Feed of the user based on the items he has Saved, i.e. recommend new items in NF or give higher priority to items in NF that are highly similar to the Saved items, do we see an increase of engagement (likes, shares, comments, #of posts, clicking links, playing videos, time spent)?
23. Monetization: %Increase in CTR (#of clicks measured for click ads, the increase should be very low, but still since CTR is always very low, we can only increase it in very small steps) –> increase in revenue for Businesses
24. Monetization: %Increase in Impressions for Impression Ads
25. Monetization: %of revenue for FB increase just based on clicks and impressions made through the funnel that includes Saved items

Prioritise:

 

Based on BIZ goals and user goals, I choose the following metrics:

 

1,2,3, 11, 12, 13, 21 (1-4), 22, 23, 24, 25

 

Summarise:

Save Feature is helping users to Save items to interact with them later. It expects to increase the revenue for marketers by helping them not to be forgotten by users. It also helps FB to increase engagement and finally revenue.

We brain stormed metrics to measure discoverability, engagement, retention and monetization effects for feature.