Let’s say that Meta (Facebook) would like to change the user interface of the composer feature (the posting box) to be more like Instagram. Instead of a box, Meta (Facebook) would add a “+” button at the bottom of the page. How would you test if this is a good idea?
Meta- Rohit Kumar
I would say – this question is suppose to be answered around “how would you test if + button is a good feature or not “. As a clarification question- you could have asked what does “good” implies here. He may have answered with the metric like “Increased number of total posts”. Once you have this metric, you can start working out how will you test the feature
Here is a quick gist of what could have been done
Problem – “how would you test if + button is a good feature or not ”
Amit – What does “good” implies here
Interviewer – “Total increase in the number of posts”
Amit – Do we have any constraints like on resources
Interviewer – No you do not have any constraints on the resources
Approach
“how would you test if + button is a good feature or not ”
1. Figuring out the testbed segments
1.1 Users from one most active user country like US, OR India
Hypothesis – Due to the dominant of users of these countries – Feature success much depends on them
1.2 Users from mix of most active users countries like US AND India and China
Hypothesis – Due to different cultural and social behaviours of the countries, the feature success vary significantly
Assume, we are going ahead with our 1.2 hypothesis
2. Figuring out the customers in the segment
2.1 Users from mix of most active users countries like US AND India and China
2.1.1 – Select customers which have a definitive pattern of posting the posts over last few months
Hypothesis – Due to the definitive post pattern, it would be easy to see if these users have same of the declined post after rolling out the “+” feature button
2.1.2 – Select the users who have no pattern in posting the posts over last few months
Hypothesis – Due to non post pattern, it would be easy to see if these users are posting as it ( the + button feature ) has made it convenient to post the posts
Assume, we are going ahead with our 2.1.1 hypothesis
3. Figuring out the number of users in the segment
Do the same thing – here we have to make sure you have chosen enough users which makes the test results statistical significant.
Here statistical significance number matters a lot as Facebook as a huge audience base of more than 1B. So you need to figure out on how many users would you like to test this feature rollout.