You launched a new signup flow to encourage new users to add more profile information. A/B test results indicate that % of people that add addtl. profile information increased by 8%. However, 7 day retention decreased by 2%. What do you do?

Let’s start from the WHY behind the change- likely that you were implementing this change to improve retention ; let’s proceed with the assumption that this is a social app where profiles play a critical role.

An increase of 8% for sign up flows is a significant increase in number of people completing profile information – I would ask how was this implemented because from experience I know that every time we add a step it introduces a 3-4% drop off.

My assumption here is that this is an added pop up screen during sign up which is causing a drop off in total number of users completing sign up successfully since they drop off on the profile info screen.

*Interviewers nods yes*

Also I would recommend changing the way we measure success of the A/B test , let me tell you why – consider the following scenarios

In case A for every 100 people signing up 50 people ended up signing up of which 20 completed profile information

In case B ( winning variant ) for 100 people signing up 48 people ended up signing up of which 22 complete their profile

So the blocking screen is causing an overall drop off which reduces D7 while increase number of people completing their profile

I would re-configure the experiment hypothesis to ” users with better profile information have higher retention than users who don’t – how can i increase in the number of users filling their profile information on D0 ( primary metric) while increasing/not affecting the successful sign up rate ( secondary metric that doubles up as a kill metric) ”

if my goal is to increase profile information of new sign up I would focus on passive methods  ( push notif, in app pop ups, incentives)  POST users successfully signing up so as to negate this drop off

if my larger business goal is to increase retention I would reduce the steps of sign to increase successful signup and focus on passive methods ( push notif, in app pop ups, incentives) POST users successfully signing up so as to increase total user successfully signing up and help improve D7

Asking clarifying questions is critical to properly answering this Google problem solving question:

1. What was the new sign up the flow and how is it different than the old sign up flow. My assumption is the new sign up flow asks for more user information, thus the increase in the % of profile information.
2. What was our goal? Was it to increase profile information? What was the acceptable counter metric decrease? Is the result within range?

1. Increase in profile information – the more data we have, the better our recommendation and personalization system and the network becomes more valuable to the users. Cons, depending on the types of information we ask and require of the user, the user may have a different level of comfort and privacy concerns
2. Decrease in 7 day retention – if a user does not come back in 7 days, this is not great for the platform, however, I want to also consider whether 14 day or monthly retention has decreased, perhaps the new user no longer needs to come back on 7th day and add a profile pic or other information.

I would start by asking some clarifying questions to help you better grasp the 1. main business KPIs.2. the purpose of the A/B test, and 3. the company’s current stage, i.e., is money (from the additional user info) at risk of fewer long-term users, or is user retention more valuable?

1. What is the App’s primary business model? Is this, broadly speaking, an ad-funded company or an app with a subscription model?

2. Is an LTV model designed for the app that considers 1. the importance of more user data 2. A Seven-Day Hold

3. Could you please clarify the original purpose and hypotheses of the A/B test? Was it to increase the amount of information per user, and if so, how was that measured in terms of the impact on business?

If a company receives advertising funding, more user data may translate into higher CPCs or CPMs; this needs to be measured and contrasted with a decline in the number of interactive users.

In addition, could the 2% 7-day retention loss be divided to determine whether the users leaving are from valuable segments? That would have a more significant impact in the early stages of a product than any immediate gains from higher ad revenue.

Conversely, in the event that this is an established product with a prior product market fit—that is, the people being held onto are the valued ones—then providing more information to them would be advantageous.