New preference-mapping study shows why one-size-fits-all AI alignment breaks down

A new study maps what people actually want from AI systems by analyzing 1,500 open-ended responses from the PRISM dataset across 75 countries.

The authors find that preferences vary widely, with most values requested by fewer than a quarter of respondents and truthfulness standing out as the main exception.

That matters for AI alignment because standard RLHF pipelines often compress conflicting preferences into a single reward signal, hiding real disagreement among users.