Translation, Tracks & Data: an Algorithmic Bias Effort in Practice

Abstract

Potential negative outcomes of machine learning and algorithmic bias have gained deserved attention. However, there are still relatively few standard processes to assess and address algorithmic biases in industry practice. Practical tools that integrate into engineers’ workflows are needed. As a case study, we present two tooling efforts to create tools for teams in practice to address algorithmic bias. Both intend to increase understanding of data, models, and outcome measurement decisions. We describe the development of 1) a prototype checklist based on existing literature frameworks; and 2) dashboarding for quantitatively assessing outcomes at scale. We share both technical and organizational lessons learned on checklist perceptions, data challenges and interpretation pitfalls.

Related

June 2019 | ICWSM

Local Trends in Global Music Streaming

Samuel F. Way, Jean Garcia Gathright, Henriette Cramer

October 2018 | CIKM

Towards a Fair Marketplace: Counterfactual Evaluation of the trade-off between Relevance, Fairness & Satisfaction in Recommendation Systems

Rishabh Mehrotra, James McInerney, Hugues Bouchard, Mounia Lalmas, Fernando Diaz

March 2017 | AAAI Spring Symposium Workshop on Designing the UX of ML

Not so Autonomous, Very Human Decisions in Machine Learning. AAAI Spring Symposium on UX for ML: Designing the User Experience of Machine Learning Systems.

Henriette Cramer, Jennifer Thom