Given the scale at which Amazon operates 10% drop is a very large number when we consider the absolute numbers. My initial line of questions would be to understand if the drop is:

• Sudden or gradual
• Specific to a particular region of the world or global
• Mobile or web traffic

Investigating the internal factors:

• Did we do any major releases recently or push new featuresIf yes then find the plan of action to roll back the feature or fix the issue/bugs from the new feature which is causing this drop 
• Study the internal monitoring tools:
  • data & analytics tools to see if there are any instrumentation errors
  • API monitoring systems to see if some APIs are not working
• Are there any products in the catalog that are sold out
  Was Amazon running an offer for the new iPhone and now the product is sold out which is why traffic reduced 

Investigating external factors:

• Any new competitors in the market selling products or catalog of Amazon at a better pricing hence traffic is getting diverted
• The country that Amazon is operating in is facing some kind of crisis — like political unrest, natural calamities, recession etc
• Has Amazon completed a sale season or a festive season and now entered a time of the year or month where users in general are not in the mood to buy

Alright few clarifying questions to begin with,

Q. Are we seeing this drop suddenly or is it a gradual decline?

A. It’s a sudden drop from yesterday, if it was gradual we would have known that.

Q. Is this happening for all the geography or a particular?

A. Yes, it’s happening across geography

Q. Is this due to seasonality or this is abnormal behavior?

A. This is not due to seasonality, it is an anomaly

Q. Is there an instrumentation issue, check with QA around this.

A. There is no instrumentation issue

Q. Is this happening for some browsers or devices or OS?

A. No there is no pattern related to the browser, device, or OS

Alright then moving along I’ll get to the following factors here

1. Looking at the funnel to understand which KPIs are taking a hit

Users go to Amazon.com > w% > Users search for items > x% > Users adding items to cart > y% > User checking out  > z% > User completes payment

Which of the KPIs has seen an impact here?

• All the KPIs are taking a hit here
• This implies that users are not going to Amazon.com

• Is there a server outage?
• Did we roll out any experimentation?
• Was there a new feature rollout?
• Is the mobile app seeing an increase in number of users?
  • If so did we launch any new feature for mobile?
• Did some of the users get logged out by the system?
• Did we delete certain user profiles?

• How many users land on the website, is there a change in any of those numbers?
  • With direct address
  • Referred by Search (Google, Bing, Yahoo, etc.,)
  • Referred by social media and influencers
• Was there any bad PR?
• Was there a massive internet outage?
• Is there some external competitor?

Clarifying Questions:

I am well aware about the amazon website . It is an ecommerce platform which offers a vast selection of products, free shipping and excellent customer service.

Am I right? Yes

Q: Which metric is actually getting dropped? Is it for user engagement, user acquisition? User engagement

Q: What is the exact metrics?Average daily Orders

Q: Does the BI tools your are using is providing you the correct result.

A: Yes

Q: Is there any change in metric calculation?

A:No

Q: Is it sudden or gradual change? Sudden. Observed since last week

Q: Does this change is due to some particular segment like location, gender? No

Q: Does this is change is device specific?

A: Android Users are getting affected

External Factors

Q: Does there is drop in the Android mobile devices sale?

A: No

Q: Does new android devices has launched recently that people shifted there?

A: No

Q: Do we have govt regulations which has affected amazon strategy

A: No

Q: Does any competitor came and providing you the good offers and people started moving there?

A: No

Internal Factors

Q: Did we recently had major release?

A: Yes . Last week

Q: What kind of changes were in this release?

A: Call to action button (Checkout) position was change?

Q: Did you tested in all the mobile devices. Is it visible fully?

A: In Android , it is visible at the bottom of the screen but the use needs to use scroll bar .

Summarize,

We discussed that there is a drop in the Average daily orders for the Amazon. We started our discussion with knowing more about the metrics and product. We found out that it is a sudden drop for Average Daily orders and for the Android users. Then we disusses more about the external factors and found out that there is no problem over there. Lastly we discussed about the internal factors and found out we have major release last. week and changed the location of call to action button which was not visible in Android.

Next Steps

We will have to roll back our changes immediately so that users should not get the uninterrupted experience. Meanwhile , we need to call the team for the hotfix which should be released asap.