1) Did this key metric drop for a particular market (US/UK)?
2) Did this key metric drop on a specific platform (PC, mobile (android, iOS..))?
3) If this metric is usage related, did the flow from source/referrers (SEO, SEM, partners, FB..) to our streaming service change? If so, which one?
4) Is this fall in key metric seasonal? Has it happened before? Over what time frame has this key metric fallen?
5) If this metric is usage related, did the streaming video usage go down for ALL services, or was it just ours?
6) Was this drop in the metric related to problems of internet connectivity? DNS failure? Data center down? if related to a specific market, did the ISPs go down?
7) Was there a natural calamity that has caused this metric to go down?
8) Is the metric going down a telemetry related issue?
9) If this metric is usage related, is the metric going down a result of a recent app redesign ? (user error)
10) Is the metric going down a result of the app misbehaving? (app error)
11) Have the users gone to our competition?

• As a PM for a streaming video service, I will have different kind of metrics

• North star metric being

  • Total number of subscribers

• If I divide the metrics into different parts, they will be along with major metrics

  • Awareness

    • Number of new visits per day (divided by channel)

  • Acquisition

    • # of signups per day

  • Activation

    • We can measure quick activation

      • % of users who have watched at least one episode by the number of accounts which were created in the last 24 hours

  • Engagement

    • DAU, WAU, MAU

      • Active user is

        • Watched at least one show per day

    • Time spent by the activated users on the product on average

    • Number of episodes/movies watched per activated user

  • Retention

    • Stickiness; DAU/MAU

    • Cohort analysis

      • Daily retention by cohort when they first logged in

  • Referral

    • # of users who share the

  • Revenue

    • Average customer lifetime value

    • Churn

      • # of subscribers not renewing or canceling subscription by total number of subscribers

  • Performance metrics

    • Page load timing

    • # of buffering required per video

• Based on the metric which has dropped, I will further solve the problem

  • Let’s say DAU dropped by 80%

    • What can be the different reasons

      • I will not check the metrics which are optimizing ones, will only check the ones that can impact the most

      • First, I will speak to users to find out the reasons and find which part of it the reason might be

        • Check the social media comments

      • Content

        • A new show which is launched and is available on competitors is not available on netflix

      • Channel

        • Content takes a lot of time to load (buffering)

        • Site/app crash

      • Network issues

        • Faced throughout the user group

      • Other issues

        • Public relations issue?

          • What are people saying on Social Media

          • Are people banning our product for some reason

  • Based on the reason, I will then, solve the problem working with the engineering team

Framework/steps: describe the product, clarify the question, define the problem statement, draw the user journey and ask a series of questions to zoom in on which part of the funnel results in the problem, and provide a quick solution (optional).

 

Describe the feature/product:

The company’s primary business is its subscription-based streaming service which offers online streaming of a library of films and television programs, including those produced in-house. We offer video streaming in the US and internationally. Is this correct?

Interviewer: Yes.

 

Clarifying questions.

Is this a new product (post launch)?

Interviewer: No, we’ve had this service for years.

 

What’s the absolute number of the drop (sometimes the absolute number is small but percentage seems large)?

Interviewer: Assume that the absolute number is also large.

 

Does this drop affect US, international or both markets?

Interviewer: All markets.

 

Are internal or external stakeholders affected by the drop?

Interviewer: External.

 

So, the drop is affecting our end users, ie customers?

I: Yes.

 

Device, mobile or desktop?

Interviewer: Mobile.

 

Do we offer it on all types of phones (eg, iphone, android)?

I: Yes.

 

Did the drop was device specific (e.g. iphone only)?

I: No.

 

Let me restate the problem and continue with more questions.

We’ve been providing streaming video service to customers in the US and internationally for years. The product is available on desktop and on mobile (all devices). Today we discovered an 80% drop in key metric on mobile platforms (all devices) that affects external customers in all markets (US and internationally). I’m in charge of investigating the problem and finding the right course of actions. Is this correct?

I: Yes.

 

As this is affecting our external users, i.e. customers, I’d like to draw the funnel/user journey and try to identify where the drop occurred. Does this sound good?

I: Yes.

 

As a user of the video service, I download the app on mobile to watch the content in the car/etc, I create the account/pay subscription fee, log in, search through database of videos, filter the output, find the video I want to watch, watch it, log out, repeat. Is this the right sequence? Did I miss anything significant?

I: Yes, it’s correct and no, you didn’t miss anything.

 

As you mentioned the problem only affected mobile users, is % of app downloads the same?

I: Yes.

 

What about % of paid mobile subscribers?

 I: The drop is in paid mobile subscribers.

 

Is it a monthly/quarterly or daily subscribers?

I: Monthly.

 

With 80% in paid mobile monthly subscription, I’d need to investigate further.

 

Internal:

Did we change how we calculate this metric?

I: No.

 

Has the drop happened in the past (seasonality)?

I: No, that’s the first time the drop is that dramatic.

Was there a technical incident, bug, server issue that affected the number of downloads this month?

I: No.

 

Did we release a new version that’s lagging technically or on UX?

I: No

 

Did we release a new free feature that’s cannibalizing our paid service?

I: No

 

External:

Did the % of mobile devices sold drop this year/quarter/month?

I: No

 

Are there any micro/macro economic factors that affecting the decline in % of paid mobile subscribers?

I: No.

 

Is there a competitor that’s taking our market share?

I: Bingo.

 

I assume the competitor is focused on both the US and international market and specifically on mobile. Is the competitor providing free service or a major discount?

I: Free video content on mobile apps.

 

I’ll restate the problem:

We’ve been providing a paid streaming video service to customers in the US and internationally for years. The product is available on desktop and on mobile (all devices). This month we’ve discovered an 80% drop in monthly paid subscription on mobile platforms (all devices) in all markets (US and internationally). The reason for the drop is a new competitor who entered a video mobile market and offers a free video content.

This is the type of Netflix problem solving interview question that has more than one answer. This makes it critical to establish the right framework from the start.

80% is a huge drop in any key metric.

I would try to first narrow down exactly what metric this is so I would ask the interviewer if they could tell me if the metric is new user retention, churn, monetization etc.

Second, I would try to understand if this was a sudden or gradual drop. for 80% definitely sounds like a sudden drop or else somebody would’ve said something already.

If it’s a sudden drop, I would try to pinpoint around what time this drop occured and figure out if there were any internal/external factors that could have caused it.

internal factors include: new feature was released, server went down, a new bug became prevalent. For the last two, you can segment it by region, browser/device type, and OS type. The issue could also be that the metrics we are grabbing is incorrect.

external factors include: a new competitor has joined into the market, bad PR, maybe a firmware was pushed outside of your control. It could also be due to seasonality or a major temporary event. If it’s a major temporary event, you should see KPIs begin to return to their normal state shortly.

Third, I would try to see if any other relational KPI drops. It’s easier to know what KPI it is before, but we can go along the user journey and see if any KPI before it dropped.

IE: A user signs up for the service -> enters in a credit card for payment (optional) -> clicks on a video to watch -> Watches the video -> chooses another video to watch

This is important in narrowing down exactly when the problem first starts. For example, if a key KPI is number of videos watched, perhaps the sign in is where most people are failing.

If the issue is a feature, I would try to clarify what the goal of the feature is. It could be possible that we started doing targeted ads and conversion dropped but the first time purchase after clickthrough increased. It would be important to understand if the goal of the feature change was met even with this big of a KPI drop.

If I can ascertain the exact issue, I would work with Sys Ops, Engineering, and other people on my team to try to address it. If the issue is a bug, we would have to issue a hotfix. If the issue is a server, than sys ops can look into it. If the issue was due to a feature release, we should probably look into either fixing it or reversing it quickly.

If the issue is external, this would be harder to solve immediately and would often require going through the normal cycle of product development to address them.

So in summary, I would first make sure we can ascertain if the drop was temporary or permanent, gradual or sudden and if the KPI drop may have occured elsewhere int he user funnel. I would look at internal and external factors to try to see if I can pinpoint the issue. Third, if the issue can be fixed immediately, I would contact my team to put out a hotfix or roll back a change that we may have made. If not, we should understand the issue thoroughly before acting and let people in the company know of our findings.