Context – I joined a large public-facing service as a PM midway through its development.

Situation – Due to the SOPs of the company the team already had a metrics framework. That included your standard DAUs, Retention and Acquisition Metrics

Concern –

• As SOP metrics were publicly accepted, the team did not internalise what success actually means to the product they are developing, as the actual value was not being encapsulated in the framework
• This was evident in the show and tells as the team was going through a lot of metrics to actually show value to the stakeholders
• Due to the sensitivity of the data, some of the metrics were not available, this added to the complexity

Action

• I worked with UR and UX people to work out the value chain in the product/service.
• Mapped the user’s external and internal
  • Mapped what’s in it for both sets of users ( value for the users)
  • Implemented North Star Framework on it
  • Changed the framework to measure value delivered, for example, rather than DAUs, started measuring submission and submission growth for external users.
  • Changed the North Star for internal users to accuracy based on NPS survey rather than just submission.
• Due, to the unavailability of some of the metrics I created a proxy metrics framework that closely simulated the actual metrics. For example, instead of actual data checking for accuracy, creating an anonymous NPS survey aimed at accuracy for analysts who had clearance to look at the data.

Results

• Took a lot of time to bring stakeholders on board, but when they saw metrics being a close representation of the value being generated, the framework was approved
• The framework allowed the team to do subsequent successful iterations that improved North Star by 5% over an already high base in just 2 iterations.
• The easy availability and trust in the metrics brought a lot of ownership from stakeholders and internal users.

Learnings

• Metrics should reflect the value your product is creating for your users
• There are always alternative way of gathering metrics, you just have to be willing to experiment
• At large scale a good metrics framework allows you to iterate successfully

Context – I joined a large public-facing service as a PM midway through its development.

Situation – Due to the SOPs of the company the team already had a metrics framework. That included your standard DAUs, Retention and Acquisition Metrics

Concern –

• As SOP metrics were publicly accepted, the team did not internalise what success actually means to the product they are developing, as the actual value was not being encapsulated in the framework
• This was evident in the show and tells as the team was going through a lot of metrics to actually show value to the stakeholders
• Due to the sensitivity of the data, some of the metrics were not available, this added to the complexity

Action

• I worked with UR and UX people to work out the value chain in the product/service.
• Mapped the user’s external and internal
  • Mapped what’s in it for both sets of users ( value for the users)
  • Implemented North Star Framework on it
  • Changed the framework to measure value delivered, for example, rather than DAUs, started measuring submission and submission growth for external users.
  • Changed the North Star for internal users to accuracy based on NPS survey rather than just submission.
• Due, to the unavailability of some of the metrics I created a proxy metrics framework that closely simulated the actual metrics. For example, instead of actual data checking for accuracy, creating an anonymous NPS survey aimed at accuracy for analysts who had clearance to look at the data.

Results

• Took a lot of time to bring stakeholders on board, but when they saw metrics being a close representation of the value being generated, the framework was approved
• The framework allowed the team to do subsequent successful iterations that improved North Star by 5% over an already high base in just 2 iterations.
• The easy availability and trust in the metrics brought a lot of ownership from stakeholders and internal users.

Learnings

• Metrics should reflect the value your product is creating for your users
• There are always alternative way of gathering metrics, you just have to be willing to experiment
• At large scale a good metrics framework allows you to iterate successfully