A watermark intended to identify AI-generated text can also change whether a model follows harmful instructions, according to experiments on six open-weight systems. The research tested a public implementation of Google’s SynthID-Text, which embeds a detectable signal by subtly changing how the model selects its next token.
Those small choices can propagate beyond wording. Under adversarial prompt injection, several tested models became more likely to answer harmful requests they would otherwise refuse. In an agent, sampled tokens may also determine which tool is called and which arguments it receives. The researcher calls this effect “sampling drift”: a provenance mechanism changes behavior because it modifies the same generation process that drives safety decisions and actions. Results also varied with the secret watermark key.
The study did not test Claude or Anthropic’s future implementation of SynthID. It used Hugging Face’s unmodified watermark processor so the feature could be switched on and off while other settings stayed fixed. That makes the finding a warning about at least some configurations, not proof that every deployment has the flaw. As platforms adopt text provenance to meet regulatory and authenticity goals, developers will need to repeat refusal, prompt-injection and tool-use evaluations with watermarking enabled rather than assuming an otherwise invisible output change is behaviorally neutral.