As the product manager responsible for Facebook’s newsfeed, your task is to increase the number of posts users see per day. To test this, the company conducted an A/B test with a randomized 1% sample of the population. Which metrics would be affected by this change, and what type of data should you anticipate as a result?
Meta- Dan Coelho
CQ:
-
What are we testing -> more posts from people and fewer ads
-
Platform specific – web or app -> agnostic
-
Any particular geography we are focussing on -> agnostic
-
Any specific metric we are targeting to inc the no of posts -> lets say 5% in 1 month
Metrics for the A/B test:
-
Avg session length for the control group vs test group
-
Avg no of posts seen per session for the control group vs test group
-
Measure reaction:
-
Avg no of likes made in the test duration for the control group vs test group
-
Avg no of comments made in the test duration for the control group vs test group
-
Avg no of friends made in the test duration for the control group vs test group
-
Guard Rail Metrics:
-
Impact on retention: % churn rate for the control group vs test group
-
Revenue per 1000 impressions for the control group vs test group