Consumption-based approaches in proactive detection for content moderation

Abstract

Implementing effective content moderation systems at scale is an unavoidable and complex challenge facing technology platforms. Developing systems that automate detection and removal of violative content is fraught with performance, safety and fairness considerations that make their implementation challenging. In particular, content-based systems require large amounts of data to train, cannot be easily transferred between contexts, and are susceptible to data drift. For these reasons, platforms employ a wide range of content classification models and rely heavily on human moderation, which can be prohibitively expensive to implement at scale. To address some of these challenges, we developed a framework that relies on consumption patterns to find high-quality leads for human reviewers to assess. This framework leverages consumption networks, and ranks candidate items for review using two techniques: Mean Percentile Ranking (MPR), which we have developed, and an adaptation of Label Propagation (LP). We demonstrate the effectiveness of this approach to find violative material in production settings using professional reviewers, and on a publicly available dataset from MovieLens. We compare our results with a popular collaborative filtering (CF) baseline, and we show that our approach outperforms CF in production settings. Then, we explore how performance can improve using Active Learning techniques. The key advantage of our approach is that it does not require any content-based data; it is able to find both low- and high-consumption items, and is easily scalable and cost effective to run.