Making ideal decisions as a product leader in a web-facing company is incredibly challenging. Beyond navigating the ambiguity of customer satisfaction and achieving business goals, leaders must also ensure their products and services remain relevant, desirable, and profitable. Data and experimentation are crucial for testing product hypotheses and informing decisions. Online controlled experiments, such as A/B testing, can provide highly reliable data to support decisions. However, these experiments can be time-consuming and costly, particularly when assessing impacts on key business metrics like retention or long-term value.

Offline experimentation allows for rapid iteration and testing but often lacks the same level of confidence and clarity regarding business metrics impact. To address this, we introduce a novel, lightweight, and flexible approach called scenario analysis. This method aims to support product leaders' decisions by using user data and estimates of business metrics. While it cannot fully replace online experiments, it offers valuable insights into trade-offs involved in growth or consumption shifts, estimates trends in long-term outcomes like retention, and can generate hypotheses about relationships between metrics at scale.

We implemented scenario analysis in a tool named ForTune. We conducted experiments with this tool using a publicly available dataset and reported the results of experiments carried out by Spotify, a large audio streaming service, using ForTune in production. In both cases, the tool reasonably predicted the outcomes of controlled experiments, provided that features were carefully chosen. We demonstrate how this method was used to make strategic decisions regarding the impact of prioritizing one type of content over another at Spotify.