Structure

I will make sure that I understand the group product and the process of creating a group and ask for clarification questions; I will explore the potential factors for the drop; I will ask for additional data and then provide one or a few hypotheses.

The Product

Facebook’s mission is to make the world closer and make tools for building communities. Groups is one of the main features for communities to interact; therefore, drop-in group creations may be an emergency. In groups, group members can interact with each other (in some groups, the interactions are moderated by admins).

The users who create the groups are called admins; Any user can create a group.

Clarification Questions (responses)

1. Are we talking about the number of groups created or the number of users who create a group? (number of groups created)

2. Is the decrease in the number of clicks on “Create” or in successful group creations? (Clicks on Create; There is no change in % of completion)

3. During which period we experienced a drop? (In the last two weeks)

4. Did the drop occur suddenly (a day or a few days) or overtime? (a steady decrease of 20% on average for every day in the last two weeks)

5. Do we know when it started? (Between SundayMonday last week)

6. Are we experiencing a significant drop or increase in other features or generally WAU? (No)

7. Did we experience such a drop in the past? What were the reasons? (Vary)

Factors

Internal Factors (responses)

1. Did we launch something? (No)

2. What AB tests are currently running, were there any changes? (1- Changing the location of the marketplace in the left navigation. 2 – Making posts wider. 3 – Language on the terms of use page.)

3. Were there changes in privacy or policy (Yes an AB test)

4. Were there changes in the way we track creation events (internal external)? (No)

5. Are we experiencing a relative decrease in engagement inside of groups? E.g., average postscommentsreactions (No)

6. Did we change the button’s location, color, or anything around it, e.g., the background color, other items around it, something that might be more eye-catching? (Location)

External Factors (responses)

1. Did one of our competitors launched a feature or changed a policy? (No)

2. Were there any regulation changes? (No)

3. Is there some significant event (holiday, sports events, elections,..)? (A big storm on the east coast)

4. Was there something in social media or news? (The storm)

General Factors (responses)

• Is it worldwide or only a specific region? (US)

• Is it for the app or the web? (Desktop)

• Can we segment it with some other demographic dimension? (No)

• Can we segment it by devices or operating systems? (No)

Hypothesis

1. The AB test around the location of the Marketplace button could affect the clicks on the groups’ button – need to check the test groups size and correlation with the drop.

2. The storm may affect the creation of some kind of groups, for example: around a specific event, e.g., sports, live show, community,.. – need to check:

   1. if there is a drop in the creation of a particular type of groups

   2. Did we experience such a drop in previous storms?

I want to first validate my understanding of Fb groups.

Groups are essentially a topic that could be private or public. Both businesses and individuals can create groups with a goal of discussing common interests. Does that sound good? IVR: Yes

Clarify:

• Was this 20% decrease a sudden one or gradual? IVR: Gradual
  • Gradual means over a month, quarter or year. IVR: assume quarterly
  • I am assuming this is the first time we have come to know about the decrease? IVR: Yes
• Is it fair to assume that it is across all platforms (web/app) IVR: Yes
• Any information abt specific geography or user group impacted IVR: No
• Any info abt other metrics that have gone down. IVR: None

• Dashboard Metric: I would see if there has been any change the way we measure the metric has changed. Work with data partner to double check the source of truth for dashboard report
• Absolute no: I would see if there is a decrease a absolute nos. as well. It could be that due to other factors, there was high no. of group creations last quarter which is driving the metric down
• A/B testing: Any form of A/B testing by other teams that is impacting this. Could be that it was not considered as a counter metric when trying to increase no. of ads annoying people
• UI changes/placement: same reason as A/B testing
• Other metrics: check if any change in new signups and MAU/QAU. Could be that new users also decreased or existing users churned squeezing the top funnel
• Latency/performance: Check if there is drop in intent as well. If not, means users are trying to create but due to latency are dropping
• New limits introduced: Any change in capping the no. of groups a user can create
• Marketing campaign: any change in promotions or policies that creator wont be incentivised
  • policies such as certain topics/words wont be allowed and thus the request wont be approved

• Seasonality: would compare it to last year as well. could be festivals such as lent when many people don’t use internet. Unlikely because it would cause a drop in many metrics
• Regulation: Government cracking down on group admins/creators pushing people away from creating new ones.
• Competition: new features/promotions on reddit or internal (Instagram) for creating groups that are attracting users. New entrant in the space that launch last quarter
• OEM: App store / play store / browser updates making the UI incompatible – could be found by looking patterns at each OS/browser level (countrywise)
• Third party: language translation API from third party not working. Unlikely because it would have resulted in many complaints and we could have found the issue earlier
• Privacy/BAD PR: unlikely as it would have caused sudden drop
• Global event: some calamity resulting in people spending less time on Fb. Unlikely as it would have caused drop in all other metrics.

The first thing I would do is to get a better clarification of this Facebook execution question, starting with:

1. Gone by 20%, since when? This is important to differentiate an anamoly or gradual change in user behavior.

2. Has it gone down at the start of the week  or is it compared to a longer time period (for e.g. year on year basis)

3. Are there any other relevant metrics which are down with it such as group engagment/ activity amongst users?

Each answer to part 2 will help direct the conversation to a new branch

— If metrics have been down since past one week, i.e. shorter time period:

We can safely assume this is an anamoly in pattern. For this purpose, we will look in to the following:

1. Can we divide the users in to various segements/groups – for e.g. can we identify the groups where it is low – is it uniform across all user group or not. An example of user groups can be:

a. Dividing users by app/website

b. Users across various geographic regions

c. iOS vs Android users

2. Once a user group is identified, we look in to the following for that particular group:

a. Was there a version roll out which didn’t work out? Or similarly a roll out which removes bots, leading to overall reduction in group creation [lesser fake groups]

b. Has there been any sort of outage? such as app outage or error out when a user tried to create a group

c. A new feature which requires users to enter a captcha [helps with fake users] when creating a group

d. Are there any other metric such as friends request / user engagment metric effected too?

3. In case no group is identified:

a. Was there a global update which caused the outage?

b. Was there a version roll out which didn’t work out? Or similarly a roll out which removes bots, leading to overall reduction in group creation [lesser fake groups]

c. Has there been any sort of outage? such as app outage or error out when a user tried to create a group

d. A new feature which requires users to enter a captcha [helps with fake users] when creating a group

e. Is there a mega event happening which is effecting overall user engagement to be effected too? Such as election going on where users are more interesting in news sharing or facebook watch rather than creating new groups?

— In case we are comparing this to year on year, i.e. a long time period:

a. Was there a new feature roll out in the past which removes fake users, thus reducing the creation of fake content?

b. What was the absolute number of groups last year vs this year >> with an increase in total number, the %age creation would definitely go down [ as the denominator is greater]

c. Was there a new feature that requires extra authentication to create a group?

d. Were any other acitivies associated with groups being decreased/ increased too?