Model Selection for Production System via Automated Online Experiments

Abstract

A challenge that machine learning practitioners in the industry face is the task of selecting the best model to deploy in production. As a model is often an intermediate component of a production system, online controlled experiments such as A/B tests yield the most reliable estimation of the effectiveness of the whole system, but can only compare two or a few models due to budget constraints. We propose an automated online experimentation mechanism that can efficiently perform model selection from a large pool of models with a small number of online experiments. We derive the probability distribution of the metric of interest that contains the model uncertainty from our Bayesian surrogate model trained using historical logs. Our method efficiently identifies the best model by sequentially selecting and deploying a list of models from the candidate set that balance exploration-exploitation. Using simulations based on real data, we demonstrate the effectiveness of our method on two different tasks.

Related

April 2022 | The Web Conference (WWW)

Using Survival Models to Estimate Long-Term Engagement in Online Experiments

Praveen Chandar, Brian St. Thomas, Lucas Maystre, Vijay Pappu, Roberto Sanchis-Ojeda, Tiffany Wu, Ben Carterette, Mounia Lalmas, Tony Jebara

February 2022 | WSDM

Variational User Modeling with Slow and Fast Features

Ghazal Fazelnia, Eric Simon, Ian Anderson, Ben Carterette, Mounia Lalmas

November 2021 | ISMIR - International Society for Music Information Retrieval Conference

Multi-Task Learning of Graph-based Inductive Representations of Music Content

Antonia Saravanou, Federico Tomasi, Rishabh Mehrotra and Mounia Lalmas