Feature Description – 

Netflix recommends users movies and shows based on their interests, past views, user profile, popularity  etc on the home page.

Goal 

Online streaming industry has become extremely competitive in the past few years, Netflix wants to provide users with relevant content in order for them to spend more time on the platform and renew their subscription. I would say providing relevant content, user engagement/retention is the primary objective of this feature.

User Journey 

User opens netflix, choses her profile, Netflix knows who has logged in and based on that Netfllix recommends shows or movies to the user. User either decides to watch a recommended movie or searches for a particular show or movie and decides to watch it.

Metrics 

Based on the objective, below are the metrics I would like to measure –

Relevant Content

• Avg # of shows watched after recommended

• Avg. watch time for recommended shows.

• Ratio of Avg. # shows watched after recommendation to Avg. # shows watched without recommendation

• Avg. # of recommended shows rated up

• Avg. # of recommended shows rated down

• Avg time spent searching for a show

Engagement and Retention Metrics

1. Avg. Session Length

2. Avg. Session Frequency per day

3. DAU and MAU

4. Total # of hours viewed

Prioritization

From the above metrics I would primarily like to measure –

• Avg time spent searching for a show – This metric shows how well is my recommendation feature working

• DAU and MAU – This metric gives me my engagement metrics for the overall platform

Feature Description: Netflix recommendation helps user to get personal suggestions based on his/her interest areas and let the user know what’s trending and what others are seeing most .

Goal: To        – improve the user experience by reducing the time spent on search, also reduces the bounce rate that is probability of leaving the app

Business Goal: Increase the average time spent watching the content

Customer Journey:

Scenario 1 (when user is sure of his choices)-  User opens the App either continue watching or searches for content, now the search could be done in following ways

• In the search bar (if he knows exactly what he/she is looking for )

• Or uses standard navigation bar and uses various filters according to his interest areas – to get the search results and then chooses among them

Scenario 2- When user is unsure 

• Then either he/she will try randomly and there could be a chance that user leaves if doesnt get desired content to watch

• Or would go by what’s recommended

Now in case of scenario 2 and at times in case of scenario1 too, recommendation plays an important role.

Following metrics could be used to measure the success of recommendation engines

• How many people know ?

                 # % of subscribers who clicked at least once  on the movie/series shown in recommendation

• Usability ?

              # What % of total movies/ series watched are based on recommendation per subscriber?

• Engagement?

            # % increase  in Average user session of those who are using it

           #% increase in no of movies/series completed?

• User experience?

# Average reduction in time spent on search per user?

# Reduction in bounce rate that is the number of users who leaves the app altogether after searching for few minutes or seconds

Netflix is an OTT platform which lets you stream TV shows and movies. You can explore new shows, movies, documentaries etc on Netflix and binge watch them. Netflix has its recommendation engine to recommend shows and movies to its users based on what kind of movies and shows they have watched, added to their list and liked. It takes into a variety of factors for eg. genres watched at different times of day, movies from specific directors , artists etc.

The main goal of this recommendation engine is to suggest new shows and movies to the users which they like so that they spent more time on Netflix binge watching shows and movies 

The user journey for Netflix user is that the users opens Netflix —> Sees recommendations for shows and movies on the home screen — > clicks through to watch trailer and directly watches the show or adds to his list —> plays the content , watches it —>  likes the show —> returns back to the platform to watch more episodes and more of the similar content —

Metrics to look at

CTR of home screen

CTR of Trending movies

CTR of different section on the home screen to help improve the recommendations

Number of New items added to my list from the recommendations per week

Number of new trailers watched from the recommendations per week

Time spent watching these trailers per week

%Conversion to actual watching of shows / movies after watching trailers per week

Average number of minutes for which recommended shows / movies were watched per week

We can look at the funnel conversion – were movies actually explored , added to the list down to actually watching them./

Minutes of content watched from the recommendations per week

New shows / movies watched from the recommendations per week

It should impact the overall goal of Netflix to make you binge watch on the back of great recommendations.

So we would look at :

Average time spent by user watching Netflix per day 

Average Time spent on Netflix per session

To see if the recommendations were which would eventually bring those users back to the platform and also accurately figure out their interests : –

% of shows/movies liked from the recommended list

For retention

Average Time between Netflix sessions 

DAU/MAU

Average time spent by user watching Netflix per day

To prioritize a few metrics which impact the goal the most, I would look at

• CTR of home screen 
• DAU/MAU
• funnel conversion from recommendations to actually watching the shows 
• New movies/shows watched by user per week 

That’s a very interesting question to take. However, before jumping in I would like to share my understanding of Netflix and clarify doubts if any.

Netflix is a OTT platform that helps users watch their favourite TV shows and Movies from different languages. One can explore different genres of movies, TV shows, and documentaries through Search and the Reccommendation Engine. The Reccommendation engine feature helps users explore different things that they will like and in turn engages the user to the platform by making them watch things. It takes into many factors like Previously watched genres, Casts in prev watched including directors, Time watched, Language explored etc before suggesting a list.

The major competitors to Netflix is Amazon Prime, and other local players like Zee, MX Player, etc.

Clarifications/Assumptions:

I would like to measure the success of Recommendation engine of Netflix in the context of Indian Market. This is because, it is one of the growing market for Netflix and also a place where I’m experienced. Hope it is fine (Assuming Yes)

How are we defining the success? I would like you to suggest what definition makes more sense to us.

With the above grasp, I would like to first Define what is Success for us before moving to the metrics as defining success will help us understand what actions of user will be important and only then can we come up with reasonable metrics.

So, When I think about Netflix, it is one of the leader in the OTT space with more resources to watch in every genre and every language. Due to the above nature, Every other person knows Netflix nowadays. Thus, we can ignore Awarness right away. Due to high awareness, there is a lot of Word of mouth happening and thus in turn, acquiring users are also not much of an issue at the current stage of Netflix.

While, Due to higher presence of Competition, it is highly important for us to engage our users effectively at a large rate so as to save our users. So we need to focus on Engagement. By Engagement, I mean the engagement related to watching/finding movies and not other actions like creating multiple profiles, updating a profile, etc.. Before finalising it to be our definition of Success, we are at a stage where cash flow is of higher importance as only then, we can increase our Banner to further new productions and gets things under our umbrella.

So due to the above reason, I strongly Believe that Monetization and Engagement should be our Goal.

With the above definition of Success, I would like to next list out typical steps in a user’s Journey while using Netflix.

User Actions Related to Recommendation System:

• User decides to Binge, and opens Netflix
• Clicks on the given recommendation/ Scrolls in the given recommendation
• Hovers on a movie tile
• Engages with the Recommended tile(like/unlike/adding to watchlist/playing)
• Watches a movie/trailer
• Closes

1. # of searches per user per week (The lesser it is the more our recommendation system is doing good that users are  directly coming here)
2. Like : Unlike Ratio in our Recommendation
3. # of movies added to watchlist per week from recommended ones
4. # of hovering before clicking on a movie in recommendation list
5. # of trailers watched from recommended movies per week
6. Time spent by 99 percentile of users in Recommendation before clicking a movie
7. # of clicks on the Next Button (Side Arrow) before a selection
8. % of movies watched after watching trailer from recommended list
9. Ratio of movie watched through Recommendation: Ratio of movie watched through Search
10. % of movies/shows watched with Recommendation score less than 70 (The number can be decided through some further discussion)
11. Conversion of users to Premium after 30 days trial
12. # of New Converted users per month
13. # of New trial users 

I would like to prioritize top 3 metrics  from the above list using the following criterias

1. Relevance to Goal
2. Impact to End User
3. Easiness of Measuring