A new arXiv paper argues that content moderation should be evaluated as an end-to-end product decision, not just as a classifier score. The authors compare where to intervene and what happens after a flag.
The study measures usefulness, defined as shown responses that are non-harmful and relevant, alongside harmful exposure, or shown harmful responses. It compares input-only blocking, response-only blocking, and input-plus-response blocking on a human-labeled product benchmark and ToxicChat.
At the tested settings, response-only filtering produced the highest filter-only usefulness, while input-plus-response filtering reduced harmful exposure. Replacing response blocking with response rewriting recovered much of the blocked traffic while matching the selected response-only configuration’s observed harmful exposure count, though the authors caution this is not a universal equivalence result.
The practical message is that moderation design involves trade-offs among safety, usefulness, latency, and error handling. Teams should test configurations against their own deployment constraints rather than copying a single rule.