Structure

I’m going to make sure that I understand the journey of adding friends, ask for clarification questions, and explore the potential factors for the drop; I will ask for additional data and then provide one or a few hypotheses.

 

The Product

Facebook’s mission is the make the world closer and make tools for building communities.

A friend request is an attempt to contact a person you know or would like to know in FB’s social network.

I would say that there are 3 categories of friend requests,

(1) Suggestions – Facebook’s friend request suggestions mechanism.
(2) Direct – You see someone, in a group, in a post, in a comment, and you ask for a friend request.

(3) Search Bar – You know someone’s name and search for it.

Clarification Questions (answers)

(1) Are we talking about the number of requests or number of friendships establishedaccepted? (Requests. Relatively the number of approvals also decreased)

(2) Did the drop occur suddenly, a day or a few days, or over a period? (A drop in the average number of friend requests per user in the last quarter)

(3) Is it a problem? Did we experience such drops in the past? What is the variance of changes? (We didn’t experience such things in the past, 10% is high)

(4) Are we experiencing a significant drop or increase in other features’ activities, generally in WAU or new registers? (No)

Factors (answers)

General Factors

(1) Is it worldwide or only a specific region? (worldwide)

(2) Is it for the app or the web? (all platforms)

(3) Can we segment it by devices or operating systems? (No)

(4) Can we segment it with some other demographic dimension? (No)

(5) Is there a point in time that we could say that change started? (Yes, Weeks 14-16) – This is an important answer since it allows us to focus on a smaller period.

 

External Factors

(1) Did one of our competitors launch a feature or changed a policy? (Yes, but nothing seems to be significant)

(2) Were there regulation changes? (Not something significant)

(3) Was there some significant event (holiday, sports events, elections,..)? (Pandemic)

(4) PR – Was there something in social media or news? (Hard to tell)

 

Internal Factors

(1) Is there a difference in drop between user journeys?

(1.1) Suggestions – Facebook’s friend suggestions panels widgets (No change in conversions for relatively new users nor users with a lot of friends)

(1.2) Direct – A user sees someone and tries to connect (No change)

(1.3) Search Bar – A user is searching for someone by typing her name (A significant drop – Bingo)
Hypothesis 1: since it is a high drop, I would assume that more than one factor caused it. Such a thing can occur if a new change or a feature isn’t causing a significant change, but a combination of multiple changes can cause a snowball effect.

(2) Where in the search process we experienced a drop relative to previous periods?

(2.1) # of users click on the search (wo a change)

(2.2) Average searches per user (Decreased)

(2.3) Average # of characters typed (Decreased)

(2.4) Average # of suggestions presented during typing (wo a change)

(2.5) % Conversion of clicks suggestions (Decreased)

(2.6) % of users click to start search – get to the results page (Increased)

(2.7) Average # of results (wo a change)

(2.8) % Conversion of click on results (Decreased)

Hypothesis 2: Since there was no decrease in the number of clicks on the search, I assume that nothing changed in the discoverability of the Search bar.

Hypothesis 3: Something has changed in the search experience; we might have started presenting suggestions earlier or changing the suggestions algorithm. Since there are fewer characters, the quality of suggestions decreases, and the number of clicks on those suggestions decreases. In addition, maybe because the suggestions are not good enough, it causes users (pain and) to click faster on the search button (get to results page quicker to explore it deeper there). Since we are receiving fewer characters, the quality of the results decreases, causing a decline in conversion rates on the results page.

 

(3) Did we launched a feature or running an AB test on the search journey? (Yes):

(3.1) AB Test start presenting suggestions after typing 1 charter, 2 characters, 3 characters

(3.2) Feature launch – Added a button at the bottom of the suggestions of “Search” (After AB testing a few color options)

(3.3) New Suggestion Algorithm roll out – it was tested on 5% of the users and showed improvement and now was rolled out to 50% of users.

 

(4) Change in Performance? – time to load suggestions (wo a change) | time to load search results (wo a change)

 

Summary

We made some changes in the search user experience, causing a drop in the number of new friend requests.

Although there is a drop, it doesn’t mean that it is a bad thing. For example, a decrease in friend requests can cause an increase in requests approvals, the quality of the relationships, and the content on the feed (time spent on the site, engagement, reactions, and more).

 

Since there are many moving particles, I suggest improving the collaboration between the different teams working on that area and making sure that we agree on the trade-offs before deciding to revert something.

Short Outline to the thinking model for this questions. What the exact answer looks like depends a lot on the answers to the clarifying questions.

 

Clarifying Qs:

• Where specifically and for how long?
  • Country/region
  • Platform
  • Trend over week/month/quarter
  • What other metrics are sharply down or up or showing unexpected trends?

Structure

• Contrast with other metrics > acquisitions, daily active users, login sessions, feature
  • Clearly if other metrics are changing then this could just be a symptom of another issue.
• Measurement Accuracy/Issues
  • Data stores/ Pipelines Upgrades causing regression or data availability delays
  • Failed Data Jobs (in particular regions)
• Recent Code Changes
  • Prompts for adding friends was changed
  • Density of feed/add friends/advertisement was changed
  • UI or any general UX changes – resetting the learning curve for users
  • New feature released
    • DAU, Session frequency and length are not impacted, but significant interest in a new feature Vs news feeds which is where most promienent prompts to add friends exists
• Platform Changes
  • IOS/Android changes
    • Notifications blocks > impacting usage and consequently the realted core actions from users.
• Backend Infrastructure Health
  • Network Traffic Outages
  • Server Health/Outages
• Large Social/ Political Event (impacting the usage in general)
  • Major Sporting Events
  • Environmental Events
  • Blocks/Bans/Backlashes
• Competitors

 

The way I would like to approach this question is by first understanding the situation and trying to infer some valuable insights from them. Later, I would like to bucket things at a high level as Internal and External.Depending on the situation, I would like to jump into one bucket and dive deep into the factors that play a role there. Similarly, would cover the next highlevel bucket as well.

So to start with, I would want to know:

For how long are we facing this issue? The requests are down for a month by now

Is this a gradual change or a Sudden Change? Gradual Change

Do we find any demographic relation in the drop like across devices, location, or Types of users (existing Vs New)? No, It ahas been same across various demographics ike types of devices and Types of users. However, the drop is common to US.

Do we see any change in related metrics like DAU/MAU, Average session length etc..? No. All these metrics are pretty much same with a little bit of noise.

 

So with the above understanding, I would want to dive deep into the External Reasons first as I believe that this could have been a issue as the changes have been more gradual and moreover the issue pertains to a particular location.

External Reasons:

The PR factor can be avoided as this is a more Gradual Drop rather than some Serious Quick Drop. Moreover, Since every other related metrics haven’t changed significantly, It doesn’t make sense to focus on Government Regualtion. Hope these assumptions are valid. Yeah, they are valid.

So the left out options are some new activity by a new Competition or some change in user behaviour. In this do we know if there has been any new launches by the competition like Snapchat that could have prevented us from generating the required new Friend Requests? We don’t have any reliable information on the same.

Though, we don’t have proper info, we know that the related metrics are doing great wrt to FB. So we can be pretty sure that Competition has no play in this.So I would like to ignore this aspect.

Since pretty much all external factors are not a reason, I would next jump to the Internal Bucket.

Are we sure that the data that we are considering has no error associated with it? Yes, you can assume that and infact we haven’t chnaged anything in terms of metric definition, the frequency at which it is measured etc..

Oh. that’s great. So, next I would like to see if there are any changes wrt the Tech aspects.

Did we make any UI change to the Friend Request option? No, we haven’t changed anything to that.

Was there any upgrade to the app, like any new feature, A/B test that’s happening? Yeah, we tried incorporating a new change. It is basically a feature that Enables user’s to message without adding the other person as a friend.

This feature might have a direct relation to what we are facing as a user now no longer needs to send a friend request to talk/connect with someone. What you mentioned does make sense. How would you confirm your hypothesis?

So, to confirm my hypoothesis, I would look into the following metrics for the past month:

DAU of Messenger App.

# of Messages sent: # of Messages Received

# of users interacting without becoming friends

If there is increase in the above 3 metrics espescially the 1st and 3rd, it confirms our hypothesis.

Okay. So as a PM, what would be your next step at a high-level?

First of all, I would understand why did we launch this feature. Understand how it connects with our Goal of Bringing the World together. This feature certainly poses a threat of incresed spamming by people as now one needs no permission to interact with one another. So, would like to see the cons of this similar to the above and then would decide if we can have this feature or should remove it.