TRACE Bench turns roleplay model evaluation into evidence-backed checklists

TRACE Bench proposes a more transparent way to evaluate roleplay models than assigning one broad score after a conversation.

The framework decomposes each role profile into a fixed checklist before testing begins. A user agent then holds a natural conversation with the target model while privately updating which checklist items have been satisfied or missed. Scores can be traced back to specific requirements and dialogue turns.

The authors compare this approach with released free-dialogue transcripts from the MiniMax Role-play Benchmark. They report that those transcripts covered 73.74% of key role-profile points, while TRACE Bench reached 99.91% coverage in fewer turns. Experiments also found stable rankings across repeated runs and user-agent replacements.

The practical benefit is auditability. For roleplay, customer-service, and character-agent systems, developers need to know not only which model ranks higher, but which behavioral requirements were actually tested and where failures appeared.