Towards Fairness in Practice: A Practitioner-Oriented Rubric for Evaluating Fair ML Toolkits

Abstract

In order to support fairness-forward thinking by machine learning (ML) practitioners, fairness researchers have created toolkits that aim to transform state-of-the-art research contributions into easily-accessible APIs. Despite these efforts, recent research indicates a disconnect between the needs of practitioners and the tools offered by fairness research. By engaging 20 ML practitioners in a simulated scenario in which they utilize fairness toolkits to make critical decisions, this work aims to utilize practitioner feedback to inform recommendations for the design and creation of fair ML toolkits. Through the use of survey and interview data, our results indicate that though fair ML toolkits are incredibly impactful on users’ decision-making, there is much to be desired in the design and demonstration of fairness results. To support the future development and evaluation of toolkits, this work offers a rubric that can be used to identify critical components of Fair ML toolkits.

 

Related

April 2022 | The Web Conference (WWW)

Mostra: A Flexible Balancing Framework to Trade-off User, Artist and Platform Objectives for Music Sequencing

Emanuele Bugliarello, Rishabh Mehrotra, James Kirk, Mounia Lalmas

February 2022 | WSDM

Variational User Modeling with Slow and Fast Features

Ghazal Fazelnia, Eric Simon, Ian Anderson, Ben Carterette, Mounia Lalmas

November 2021 | ISMIR - International Society for Music Information Retrieval Conference

Multi-Task Learning of Graph-based Inductive Representations of Music Content

Antonia Saravanou, Federico Tomasi, Rishabh Mehrotra and Mounia Lalmas