Facebook’s 3rd party logins is place where other website use FB authentication layer to provide their Login support. The FB login credentials would be used as tokens for login into their respective websites. Metric of successful logins for 3rd part are down. First of all, I would like to understand the following queries

C: Which metricseactly ; Is metrics of login into the 3rd party or login into the Facebook?

I:  Success metric on Facebook login is working fine. There is no notable change into it.

C: Is it only for specific geography?

I: No, Its global observation.

C: Is it for any particular handheld device or on operating system?

I: As such, we haven’t seen into the data for same but let’s assume it’s same. There is no segregation.

C: Is there any decrease in metric of users trying to login or they are same?

I: Number of users trying to login into the

C: Is there any one particular 3rd party facing the issue?

I: There could be. Let’s fetch the info. Meanwhile go ahead with troubleshooing.

C: I would like to transverse user journey. User opens website. User click button Sign in using facebook. Now there would be 2 scenarios

Scenario A: If user is already logged using Facebook. He will automatically login into the website. In case of any Authentication APIs of FB are intact, this scenarios is less likely to get some errors. I would like to check the following metrics

# Of signins on Saturday and Sunday on Facebook

# Of signins on 3rd party using Facebook.

# Of errors on Facebook Signin API.

Scenario B: If user is not logged in FB previously, they will be asked to enter FB credentials. In case the screen appears but users are not able to click submit button or API collaboration of 3rd party to Facebook Login Api is failing. Here we need to check the error of API and 3rd party.

Lastly we also need to check, there is no leak in privacy concern of users, no cyberattack or hot fix rolled out busted in production.

 Summarizing

 I would check error, success metrics. I would also be browsing the external factors along with checking the internal factors of metrics and errors.

Clarifying questions:

1.Is this happening for new user login via 3rd party or even for returning users? – New users

2.Is this a sudden drop or gradual? Also when you say down, down by what % to understand the magnitude of drop. – sudden 10%

3. Are we seeing this on a specific browser? Specific platform ? – NO

4. Are we seeing this on a specific type of client and HW combination? such as some specific Samgun phones etc? _ NO

5. Is this happening across all 3rd party apps or any specific 3rd party app? – ALL

6. Did we release a new SDK for login recently that is being used by these 3rd party app? If so, did we change anything specifically in the login process?

7. Did we release a new login notification that lets people know what data FB provided as part of login that is causing people to not allow login via FB

If the above is all NO, then I will look closely to see what happens when a 3rd party app sends the Oauth dialog and the exchange of exchange token happens. where in the process is that getting broken ?

I will also look at customer support tickets, feedback on forums to isolate the problem.

Product Understanding:

Facebook login is a system through which users can use their Facebook accounts to log into 3rd party apps and websites instead of creating a new account on that service. Users will receive a notification that they used Facebook login to log into a 3rd party app or website. Is that correct?

Clarify:

If it is a first time user or a returning user who logged out of his previous session, they will be shown the login page to login using Facebook login

So how do you measure a successful login metric? Since a “successful login” is when a user logs into a 3rd party apps using FB login successfully, I am assuming that you are tracking # of logins on a daily, weekly and monthly basis? Yes

Okay, so the number of logins into 3rd party apps using FB login is down. I have few more questions to understand more about the context so that I could formulate my hypothesis and go deeper into the problem

Some more clarifying questions to form a theory:

1. Since this is observed on a Monday morning, should I assume that it is a sudden decline and not a gradual one? YES
2. Was the decline specific to a region or all over the world? It’s not specific to a region
3. Was the decline specific to a particular platform like ios or browser or android? If it is specific to a platform, I want to calculate the number of logins done using each platform and compare it with the total number of logins. If there is no change, then it is possible that some users are just logging through a different platform. Let’s say it is total logins and not platform-specific

Okay so we observed a sudden decline in the number of logins into 3rd party apps using FB login and it is not platform-specific and it is not region-specific

I want to make some hypothesis regarding this and I will mention how I will test each hypothesis to get to the bottom of the problem

 Hypothesis

1. Users are not preferring to login using Facebook because of privacy concerns. Test: The user behavior would be gradual and not sudden.
2. Users are trying to login but they are not able to login. They are getting authentication errors. Test: There might be some infrastructure changes made yesterday. I would check this with the infrastructure team
3.  There was news of a cyberattack that compromised several users data Test: I would check with the security team if there were any reports of cyberattacks news
4. Were there any feature releases with respect to Facebook logins yesterday that made users wary of 3rd party logins using Facebook? Test: I would check back with the product team and if there are any feature changes that lead to this decline, I will look back to the goals of the a/b test that lead to the launch of the feature. There can be a possibility that in this case even though the number of logins through 3rd party apps is reducing, the number of Facebook deactivations that happened due to the feature launch didn’t significantly change or might have even significantly reduced. Since users are not completely churning from Facebook, the value that is being created in a long-term perspective with respect to this feature launch is more than the value lost in launching this feature. If the Facebook deactivations are also significantly increasing, then I would discuss with the stakeholders to roll back the feature

Summary:

I came up with a metric that measures successful logins and formulated my theory to test. I would check for possible infrastructure changes and any reports of cyberattacks. I would also look back to the goals of the feature release to understand why the feature release happened even though there is a negative impact on the success metric.