Say you’re the PM of Meta (Facebook) and you have launched a new feature X with 20% engagement. How will you know if the overall engagement of Meta (Facebook) has increased/decreased or remained same?
Meta- Matthew Shun
First clarify what 20% “engagement” refers to for the new feature. It could simply be Click Through on a banner, a like/comment or share. Let’s assume engagement means any of these combined. So the new feature has 20% engagement. Note that this may not be an “increase” of 20%, since we don’t have a comparison for the new feature (no feature means no engagement with the feature).
Given this, the cleanest way, IMO, to answer the question of whether overall FB engagement has increased starts with identifying the right metric(s) that represent overall FB Engagement (let’s say average number of likes+shares+comments per user per day over 7 days). This can be any number of metrics that the leadership/business team care for, and can range from engagement, to the DAU/MAU rate itself, etc (or even monetization since you don’t really want to hurt that).
Once you have the overall metric(s), you setup a randomized A/B test, with users in group A (“Control”) don’t have the feature enabled for them, while users in B (“Treatment”) have the said feature enabled. After the pre-defined period, you compare the difference in the target metric for B vs A using statistical methods (two-sample variance tests). If mean(B) is greater than mean(A) with an acceptable p-value (usually 0.05), you know the feature is likely increasing the overall engagement.
If you cannot run an A/B test, there may be approaches around cohort analysis to compare metrics between customers who were exposed to the feature versus those who weren’t, but this is generally not as scientific due to potential self-selection and survivorship biases.