As the first step, I communicated this to the partner to make sure that they understand the root cause which was not because of us. Secondly, since the users were getting impacted, I took the decision to stop sending any charging requests for that partner. In meantime, we quickly put a hack that, the charging request won’t be send for the length of charging period. These 2 things solved the immidiate issues. Then I went back the to the drawing-board and re-wrote the requirements which the engineering worked on to solve this issue permanently.

This is what I learned from this :-

1. Always before the launch, figure out the scale you expect and make sure all internal & external systems are able to handle it. From that point on, I incorporated what I called as PSR as part of requirements.
2. At the end as PM, you are responsible for consumers so always plan for failure/worst cases scenarios from the external systems as well. So from now on, I always gave requirements and use-cases for failure in 3rd party systems.
3. I got our contractual terms modified legally so as we don’t get legally liable for issues in the partner systems. It was made more explicit.
4. Last but not the least, we made a process to make sure that we scale slowly and don’t increase traffic massively. Also put in relevant alerts for such scenarios.

I love this question because when it goes well, it goes really well. And when it goes poorly it’s pretty amazing as well.* The good news is that having your answer go poorly is really more a sign that you are unprepared for the interview, and all of you are preparing so this should not happen to you!

General framework for failure questions:

1. Give context. What is the product / business you were working on?
2. Talk about what failed. If you are an engineer, talk about how much money that outage cost the company, it’s OK if it was pretty catastrophic as long as you talk about how complex the system is. If you are a PM, pick a more curated failure, not the time that you bet all your engineers on a failing product since that may call into question your judgement. Ideally, PMs should pick an execution failure, to avoid giving the interviewer any reason to believe that you may lack good judgement or strategic thinking skills.
3. Explain what you did when you learned you were failing. Ideally this was calm and structured and has a good ending.
4. Tell me what you learned. 
5. Share what you’d do differently next time.

An example from a PM lens:

1. [context] I was the PM for user onboarding in a clinical research platform. The platform contained different clinical research studies that users could sign up to participate in. When onboarding to a study, users could read about the study in their native language and decide if they wanted to consent to participate in the research. From a user perspective, correct and available translations are critical to deciding if you want to participate. Correct and available translations were critical to the business from a regulatory perspective, since participants must willingly provide their consent in a language they are fluent in.
2. [what failed] Engineering pushed a change that knocked out all non-English content translations for over 24 hours, meaning that users who were not fluent in English were only able to read study information in English on the platform, for 24 hours. This outage invalidated the consents for users who signed up for a clinical study during that 24 hour period, since we could not ensure that their consent was truly informed. The failure happened for three reasons:
   1. new feature code broke the existing translations service
   2. there was no testing in place to stop engineering from pushing that bad code
   3. there was no process in place to ensure test planning and test coverage
3. [what I did during the failure] I worked with the engineer who discovered the outage and gathered the cross-functional team who needed to understand the outage and prepare additional countermeasures (ie reaching out to the clinical research sites to let them know that there had been an outage of translations). I stayed in close contact with the operational team throughout the outage and helped to create some content that we could send to recover those users once we were sure the solution was in place and robust. After we found the root cause, I helped the testing team write out the system-level UI tests to catch this issue in future testing, if there was a regression. I also worked with the engineering manager to improve the engineering team processes around test planning. I also added a missing step to our launch planning process, to ensure that all new features have adequate integration and system test coverage. In the end, we recovered quickly and were able to get over 80% of the lost users re-consented in their native languages, within one week.
4. [learning] I learned three key things from this experience.
   1. as a PM, it’s my responsibility to ensure that my features are being adequately tested by engineering to protect and ensure the user experience.
   2. I can help ensure that my engineering teams have process and hygiene around test planning and test coverage.
   3. put users first, always. We had many users thank us for reaching out to get them re-consented in their native language, and it was an opportunity to build our relationship with them since for many of them it was their first time interactiing with our platform.
5. [what I’d do differently] In general, I now take a more process-driven approach to ensuring adequate testing is in place. Since I am the one responsible for signing off on feature launches, I have implemented better checklists and processes that ensure that we don’t miss key testing, and I’m happy to say that we have not had a test-coverage related outage in the platform since that one.