What A/B tests will you run to increase the booking rate among Airbnb guests?
Airbnb- Gerard Kolan
1) Describe the Product
Just to make sure. Airbnb is the product that enables people, travelers or guests, to rent apartments and houses of other people, owners or hosts. Guests can search on the website by using filters and an interactive map. After finding the apartment, they can book directly on the website by looking at available dates.
2) Clarification
- Are we focusing on a specific segment of users? (new, existing, demographics, etc)
- You can assume.
- Whey you say booking rate, you mean that guests should rent more apartments during any time range, right?
- Yes, we are talking about renting more frequently.
3) Goal
Perfect, so I understand that this objective of increasing booking rate among guests is tied to the goal of growing, thus having more revenue.
I would like to focus on existing users since they already know how the experience looks like, and we already have their trust in the platform.
A brief talk about the customer journey. People tend to travel during work holidays or long holidays, like spring and winter breaks. They tend to plan ahead of time, and then start searching for places on the internet. To decide the place, they look primarily for activities and fun places to visit. After deciding it, they start looking for hotels or Airbnb. They care about location, mobility, food, parks, cleanliness, etc. After choosing the right place, they book and pay for it.
4) Hypothesis
- The prices shown on the map and listings after searching should be total instead of per night, and more highlighted.
- Hypothesis: Giving the total cost to users can help them better to decide which apartment to rent. Therefore, they would rent faster and not search anywhere else.
- Impact: Medium –> uncertain about this result
- Cost: Low –> not very difficult to change this
- Show professional pictures of famous touristic spots and fun places of cities that users are searching to go to.
- Hypothesis: Showing photos of what they can experience in that city may induce them to book more.
- Impact: High –> essential information to decide where to go
- Cost: High –> hard operational work
- Marketing Campaign recommending new places/experiences to users based on their past experiences.
- Hypothesis: Showing possible places that users would love to go may motivate them to book more.
- Impact: High –> Airbnb has good marketing, so it might be effective
- Cost: Low –> already have the data
Based on Impact and Cost, I would go with C), A), and B).
Now focusing only on C to design the experiment.
5) Experiment
- Segmentation
- By user_id –> avoid noise and track the same user on the web and app.
- Make sure to have homogenous groups, segregated by avg number of bookings, avg $ spent, members since X period, and others.
- Control Group
- People who are not receiving the Marketing Campaigns
- Test Groups
- People who are receiving the Marketing Campaigns
- Operation
- Campaigns will be personalized by each user based on their past data
- Campaigns will be sent by email
- Campaigns will be sent 1-3 times per week, once per day, scheduled to the time when users open the most Airbnb emails on average
- Parameters of Test
- Test power: pocket rule –> 80%
- Alpha: 5% (false positive)
- Beta: 20% (false negative)
- Simulation
- Simulate the test to make sure everything is working out
6) Metrics
- booking rate per user_id of both groups
- open and click-through rate per user_id of both groups
- avg price rental per user_id of both groups
- satisfaction rate per user_id of both groups
- We do not want to disturb their experience by sending emails
- number of “enable emails” notifications per user_id of both groups
–> All metrics will be measured weekly/monthly, to avoid the effects of seasonality.
7) Trade-offs
- Couples and groups of friends who travel together may be split into different groups, thus they might end up sharing the content.
- Users might have been disabled to receive email notifications.