Definition of the app

I understand this product as a mobile app, which connects people who want to order food to restaurants, and delivery person. Via app user chooses the restaurant and food, sets delivery location, and makes a payment. Do I understand it right?

 

Clarifying questions

• What do you mean by “late deliveries”? Do you have any specific time range when it’s happening?

• What exactly do you mean by churn (uninstallation, stop using service)? Do you have any specific definition of churn?

• Is this happening in any specific location (City, country)?

• Do we see customer churn across all devices (mobile, web, iOS, and Android)

• Do we see this across all versions?

 

From the questions above I understand that:

A cohort of late deliveries (9-12 pm) leads to customer churn (they do not make orders during next month). Our goal is to find out the data that proves this hypothesis for the delivery app (mobile, across all locations, and across all versions)

 

Finding the cause of the problem

Overall there are internal and external factors, but first, let’s focus on internal

 

There are 3 types of users involved in this process of ordering food:

1. Those who order food

2. Those who cook

3. Those who deliver the food

 

Let’s think about each of them:

Those who order food

• Do they have enough of a selection of the restaurants that work late?

• How much time should they wait until delivery? Is there any difference between regular time and late time?

• Do we see any satisfaction decrease for late-night deliveries?

• Are there any price experiments for the users for late-night deliveries?

• Do we see churn for all types of restaurants?

• Requests to support

• User cancellation

• Delivery app notifications click-through rate.

• Can we extract data about Sleep mode on customers’ phones?

Those who cook

• Do we have any data that shows us a significant decrease /increase in orders on their side?

• How the average time of handling orders change during late-night?

• How the average time of cooking change during late-night?

• Do we know anything about special rates for the restaurants during late-night hours?

Those who deliver the food

• Do we have any data about the availability of delivery person? ( i.e # of orders vs the number of an available delivery person in the area)

• Do we see any friction in time to pick up the order from the restaurant?

• Do we see any friction in time from picking up until the delivery?

• Do we know anything about the transportation type of the deliveryperson at late night?

• Cancellation by the delivery person (unable to find or reach the customer)

• Amount of tips

• Do we know anything about special rates for the delivery person during late-night hours?

• Are there any experiments or known bugs on the person delivery side?

 

External factors we need to consider:

• Do we know anything about competitors’ activity during late-night hours?

• Do we know anything about transportation regulations in the area, city, or country?

• Are there any natural diseases in the region?

• Do we know anything about public catering regulations in the country/city of investigation?

• Do we know anything about criminal risks during late-night hours in the country/city of investigation?

• Is there any information about cellular networks at late night time, that may affect communication with customers/delivery person?

 

Identify problem:

 

From the answers above I could understand and identify the problem of churn and focus more specifically on the data, that we should look.

 

Action points:

Depending on the root cause I can come up with action items that can fix the problem of customer churn.

 

‘Late Deliveries lead to Customer Churn’, what data we should look at to prove this hypothesis for a food delivery app

Hypothesis 

Late deliveries leading to customer churn

Clarity 

We need to prove this via existing data, we are looking to prove this leads to churn rather than look at causes/fix them

How do you define customer churn – assuming it mean users ordered successfully in M0 and didn’t order in the M1

Why do we want to look at this data? can be manifold, maybe we are deciding if we need to invest more in supply chain/is this an important enough problem to solve

What do we define as late deliveries? does it mean we took more time than expected or the time we showed was too high? assuming it means the former ie we took more time than promised to the customer

how do you measure delivery time? assuming it means time from the moment user paid to the moment the delivery person reaches the door step

How much data do we have? I’m assuming this is a mature app and has enough data

Data analysis

This is a data deep dive to understand if delayed deliveries lead to customer churn to do so we must compare data between delayed deliveries and on time deliveries

We must clean up the data to ensure we only pick relevant data set:

1) right user set To ensure we are looking at the right user set, let’s look at retained customers, ie users who’ve placed an order last month who are not new users ( ie their tenure on the delivery app is more than 1 month)

2) right time period: Take last 6 months data

3) Assuming the delayed order frequencies will lead to higher churn , simlarly higher delay will lead to more churn

4) We can compare Month on month average retention against frequency  and delay so the graph would look something like

Customers who didn’t have any delayed order vs customers who had at least 1 delayed order vs customers who had 2 delayed and so on.

Similarly we can look at retention after looking at the average delay vs MoM retention.

I would at this data ideally at a city/clusters within city level to ensure we don’t mask any problems at an area level with averages. It is likely that higher delays is acceptable to users in one cluster of the city owing to its infrastructure/access vs another.

Combining the 2 we should be able to see if delays impact retention, if yes, at what points both in terms of frequency/delay amount is the problem and decide actioanble in terms how to minimise.

Before diving into the finer details of this metrics question, let’s clarify what goes into churn.  Churn is the # customers lost / total # customers.  In the case of a food delivery service, I’d think # customers lost means customers who have been inactive for some time (have not made an order in the last 60 days, for example), as most food delivery services tend not to have subscription plans.

To determine that late deliveries lead to customer churn, we’d need proof of causation and not just correlation.  Thus, it’s good practice to first consider all the reasons customer may churn.

Potential reasons for customer churn fall into two potential categories:

–       Internal factors

o   Seasonal Factor – e.g., customers tend to churn in August, when they are traveling on vacation

o   Access to Product – e.g., major outage on web or mobile app prevent customers from placing orders

o   Product Changes –  e.g., modified UI that changed the size of the order button, resulting in lower # orders placed

o   Product Quality – e.g., late deliveries, inaccurate deliveries, poor customer service

–       External Factors

o   User habits – e.g., Online food delivery is down across the industry

o   Referrers – e.g., Google / Apple made a change that makes our service harder to find

o   Competition – e.g., Competitor released new delivery service that captured our users

o   Society – e.g., People are ordering less food due to societal factor like COVID-19

To isolate the effect of late deliveries on customer churn, we will want to find a data set that is sensitized for as many of these other factors as possible.

To test the hypothesis, I would separate customers into two groups: customers who churned, and customers who were retained.

–       Among the churned customers, what % of delivery orders overall met the delivery window?  I would look at these metrics over daily / weekly / monthly periods and compare against customers that did not churn over the same periods.

–       Among the churned customers, what % of the last 5 / 3 / 2 / 1 orders met the delivery window? I would compare against customers that did not churn over the same periods.

–       I would also look at % of accurate orders (overall and the last 5 / 3 / 2 / 1), as well as avg order rating scores (overall and over the last 5 / 3 / 2 / 1).

Results from this analysis will give me a better idea on whether this hypothesis holds true.