Challenges in Translating Research to Practice for Evaluating Fairness & Bias in Recommendation Systems


Calls to action to implement evaluation of fairness and bias into industry systems are increasing at a rapid rate. The research community has attempted to meet these demands by producing ethical principles and guidelines for AI, but few of these documents provide guidance on how to implement these principles in real world settings. Without readily available standardized and practice-tested approaches for evaluating fairness in recommendation systems, industry practitioners, who are often not experts, may easily run into challenges or implement metrics that are potentially poorly suited to their specific applications. When evaluating recommendations, practitioners are well aware they should evaluate their systems for unintended algorithmic harms, but the most important, and unanswered question, is how? In this talk, we will present practical challenges we encountered in addressing algorithmic responsibility in recommendation systems, which also present research opportunities for the RecSys community. This talk will focus on the steps that need to happen before bias mitigation can even begin.


September 2023 | RecSys

Accelerating Creator Audience Building through Centralized Exploration

Buket Baran, Guilherme Dinis Junior, Antonina Danylenko, Olayinka S. Folorunso, Gösta Forsum, Maksym Lefarov, Lucas Maystre, Yu Zhao

July 2023 | SIGIR

Hear Me Out: A Study on the Use of the Voice Modality for Crowdsourced Relevance Assessments

Nirmal Roy, Agathe Balayn, David Maxwell, Claudia Hauff.

July 2023 | SIGIR

When the Music Stops: Tip-of-the-Tongue Retrieval for Music

Samarth Bhargav, Anne Schuth, Claudia Hauff