Using Survival Models to Estimate Long-Term Engagement in Online Experiments

Abstract

Online controlled experiments, in which different variants of a product are compared based on an Overall Evaluation Criterion (OEC), have emerged as a gold standard for decision making in online services. It is vital that the OEC is aligned with the overall goal of stakeholders for effective decision making. However, this is a challenge when the overall goal is not immediately observable. For instance, we might want to understand the effect of deploying a feature on long-term retention, where the outcome (retention) is not observable at the end of an A/B test. In this work, we examine long-term user engagement outcomes as a time-to-event problem and demonstrate the use of survival models for estimating long-term effects. We then discuss the practical challenges in using time-to-event metrics for decision making in online experiments. We propose a simple churn-based time-to-inactivity metric and describe a framework for developing & validating modeled metrics using survival models for predicting long-term retention. Then, we present a case study and provide practical guidelines on developing and evaluating a time-to-churn metric on a large scale real-world dataset of online experiments. Finally, we compare the proposed approach to existing alternatives in terms of sensitivity and directionality.

Related

April 2022 | The Web Conference (WWW)

Sequential Recommendation via Stochastic Self-Attention

Ziwei Fan, Zhiwei Liu, Alice Wang, Zahra Nazari, Lei Zheng, Hao Peng, Philip S. Yu

April 2022 | The Web Conference (WWW)

Choice of Implicit Signal Matters: Accounting for User Aspirations in Podcast Recommendations

Zahra Nazari, Praveen Chandar, Ghazal Fazelnia, Catie Edrwards, Ben Carterette, Mounia Lalmas

April 2022 | The Web Conference (WWW)

Mostra: A Flexible Balancing Framework to Trade-off User, Artist and Platform Objectives for Music Sequencing

Emanuele Bugliarello, Rishabh Mehrotra, James Kirk, Mounia Lalmas