. The know-how that proves the place AI textual content got here from is weakening the know-how that checks whether or not it’s true. Watermarking, or embedding a hidden signature in generated textual content in order that its origin might be confirmed later, is used to extend transparency on AI purposes (Europe’s EU AI Act Article 50 [1] ; China’s AI labeling regulation by the Our on-line world Administration of China [2]). Moreover, hallucination detection tries to flag the elements of a solution that the mannequin made up. We use it to examine if mannequin outputs are fairly predictable. Regulators are pushing each edges. However there’s a hidden collision inside AI fashions between each transparency and legal responsibility. The collision is hidden within the mannequin’s internals however can have actual penalties in AI techniques’ security.
How watermarking works
An LLM doesn’t write sentences. It computes, at each step, a likelihood distribution over the following token. Generally that distribution is sharp. After “The capital of France is”, the token “Paris” carries virtually all of the likelihood mass, and there’s no actual option to make. Generally it’s flat. After “She left the get together as a result of”, a number of continuations are equally believable, and the mannequin flips a weighted coin. The technical title for the flatness of that distribution is entropy or semantic entropy to be extra particular [1]. Excessive entropy means the mannequin is selecting amongst many comparable choices. Low entropy means the following token is actually compelled. Maintain this distinction in thoughts, as a result of each applied sciences on this story dwell on the high-entropy positions.
A textual content watermark biases the coin flips utilizing a secret key. One of the best recognized scheme for watermarking comes from the work of Kirchenbauer and colleagues in 2023 [4]. This method splits the vocabulary at every step right into a pseudorandom “inexperienced” and “crimson” checklist derived from the important thing and the latest context, then provides a small bonus to the inexperienced tokens (Determine 1). The textual content nonetheless reads naturally, however it accommodates statistically extra inexperienced tokens than probability would produce. A detector holding the important thing reconstructs the lists, counts the inexperienced tokens, and computes how unbelievable that rely could be in unwatermarked textual content.
Google DeepMind’s SynthID-Textual content [5], deployed in all its Gemini household fashions, refines the concept with a mechanism known as match sampling (the mannequin drafts a number of candidate phrases and the key key referees a knockout between them; the champion will get written), utilized as a processor on the mannequin’s output chances. The scheme is open sourced within the Hugging Face Transformers library.
A 3rd household, the distortion-free schemes impressed by Aaronson’s Gumbel trick, arranges the bias in order that the output distribution of a single era is provably unchanged on common, whereas the textual content stays correlated with the important thing [6].
A watermark can solely push the place there may be room to push. At a low-entropy place, biasing the selection would imply writing the incorrect phrase, and high quality would visibly collapse. Take an bill abstract. Within the sentence “The entire due is 4,320 {dollars},” every part after “is” is constrained. There is just one appropriate continuation, and a watermark that nudged the mannequin towards every other quantity wouldn’t be hiding a signature, it will be corrupting the doc. Now take the sentence “Cost was obtained instantly” The mannequin may have written “shortly”, “swiftly”, or “directly”, and the reader may by no means say which phrase it will have picked by itself. That second slot is the place the watermark lives. A signature can solely conceal inside decisions that make no distinction, which is precisely why it’s invisible, and precisely why it can’t exist the place the selection issues. So each scheme concentrates its sign at high-entropy positions, the moments of doubt. This signature is statistical and average-case. Detection is dependable “given sufficient textual content”, which isn’t the identical as a per-output certificates.
At the moment Google marks all their Gemini household fashions. OpenAI constructed a scheme and selected to not deploy it. Anthropic has simply introduced they’ll add watermarks to their Claude household fashions. Open-weight fashions go away decoding within the person’s fingers, so their textual content is unmarked at any time when the person prefers.
How hallucination detectors work
A hallucination is a fluent assertion that occurs to be incorrect: an invented statistic, a fabricated quotation, a quantity that seems in no supply doc. Present detectors might be divided into two mechanistically completely different households.
- Grounding-based detectors learn the reply in opposition to a trusted supply and ask whether or not every declare is supported. Token-support fashions akin to LettuceDetect classify which spans of the reply lack backing within the context. Entailment fashions akin to MiniCheck ask whether or not the supply logically implies the declare. LLM judges do the identical with a prompted mannequin. What’s frequent in these approaches is the enter: the output textual content and the proof, nothing else.
- Uncertainty-based detectors exploit a behavioral signature as a substitute: a mannequin that is aware of the reply produces it constantly, whereas a mannequin that’s guessing drifts throughout resamples. Semantic entropy[1] formalizes this by sampling a number of solutions, clustering them by which means, and measuring the entropy over clusters. SelfCheckGPT and the logprob-calibration strategies are variations on the the identical thought. On this household of detectors additionally the enter is a standard characteristic. the mannequin’s uncertainty, learn both from repeated sampling or from the possibilities immediately.
The collision
The watermark operates at high-entropy positions as a result of these are the one positions the place the selection might be steered invisibly. Uncertainty-based detectors reads these positions as a result of hesitation is its “right here” ignal. The watermark steps in in the meanwhile of doubt, and resolves it with the key key (a personal quantity that fixes how each uncertain selection suggestions). The detector’s trick is determined by the resamples being recent, impartial tries. The watermark schema breaks that. Each retry makes use of the identical secret key, so the coin flips contained in the mannequin land the identical biased manner every time, and within the strictest watermark schemes a set key produces precisely the identical reply, phrase for phrase (Determine 2). If we ask a guessing mannequin the identical query 5 occasions we must always see 5 barely completely different guesses. That variation is the warning signal. With the watermark the 5 tries agree, not as a result of the mannequin stopped guessing, however as a result of the identical loaded coin determined all 5. The stableness we observe belongs to the sampler, to not the mannequin’s data.
As a consequence, consistency-based hallucination detectors overestimate settlement and underestimate uncertainty on watermarked textual content.
Not each confidence examine works by asking the query once more. A second household skips the retries and reads the mannequin’s inner likelihood numbers immediately, the working report of how certain it was about every phrase. However the watermark has already edited these numbers earlier than any examine will get to learn them, so this household inherits the distortion as properly. A 2025 research throughout seven instruction-tuned fashions [7] discovered that watermarking measurably modifications how fashions behave on downstream duties, and that the modifications stay after accounting for the loss in textual content high quality. So, for any examine that reads retries or likelihood numbers, the interference follows essentially from how the watermark works.
Grounding-based detectors sit on the opposite facet. They learn solely the output textual content in opposition to the proof (for instance in RAG techniques), and the watermark selects amongst semantically equal continuations. So, in precept, they aren’t affected by watermarks as a lot as function in a distinct area. If we have to run reliability checks on watermarked textual content, that is the household to run.
The watermark removers
The elimination instruments arrived on schedule, and practically all of them are paraphrasers: a second mannequin reads the watermarked textual content and says it once more in several phrases. We’re at the moment seing three approaches:
- Watermark stealing [8] confirmed that for below fifty {dollars} of API queries an attacker can study sufficient of a scheme’s hidden green-list rule to strip the mark from schemes beforehand thought of secure, with success charges above 80 p.c, and in addition to forge it.
- BIRA [9] (September 2025) rewrites textual content whereas steering the rewriter away from the phrases the watermark favored, and reviews over 99 p.c evasion with out realizing which scheme was used.
- WASH [10] (Could 2026) doesn’t even rewrite: it averages the outputs of three extraordinary fashions, the impartial watermark biases cancel out, and detection scores fall from far above the alarm threshold to beneath it.
The query is, do they work? The response is sure, and that is settled and it has a theoretical clarification. There’s a mathematical proof of this with a becoming title: “Watermarks within the sand: Impossibility of sturdy watermarking for generative fashions” [11]. The argument is easy (Determine 4). Suppose the attacker can do two issues: inform whether or not a small edit made the textual content worse, and maintain making small edits that don’t. Then they’ll wander step-by-step by means of hundreds of rewrites which can be all equally good, and someplace alongside that stroll the watermark’s sample will get left behind, as a result of the sample lived within the particular phrase decisions and the phrases are now not those the mannequin selected. No present watermark scheme can survive that, no matter key it makes use of. Google, in its personal description of SynthID-Textual content presents it as a marker for good-faith use, not as safety in opposition to somebody decided to take away it.
Squeezing a balloon
Bear in mind which detectors the watermark couldn’t harm: those that by no means ask how assured the mannequin felt. They take the output reply and maintain it up in opposition to your supply paperwork, checking phrase by phrase and phrase by phrase whether or not every declare is grounded within the offered context. The watermark modified which phrases acquired picked, however the claims nonetheless matched the sources, so these checks stored working. However the washing watermark-removal method breaks precisely that security. A washed textual content says the identical factor in several phrases. Totally different phrases now not line up with the supply paperwork, so the matching scores fall, and an accurate reply begins to look unsupported.
Furthermore, the washing software is itself an “AI rewriting the textual content”, and no person is checking its output. Rewriting is understood to vary issues on the edges: a quantity shifts, a “most likely” disappears, a reputation will get swapped. The cleanup step can create the very errors the security checks had been constructed to catch. And since washing occurs after the textual content is completed, it can’t convey again the pure variation the watermark eliminated. Conclusion: the confidence-based checks keep damaged.
The distortion solely strikes
The distortion produced by the watermark and its elimination course of by no means disappears. It strikes, like squeezing a balloon. The watermark presses on one facet at writing time and distorts the checks that learn doubt. The remover presses on the opposite facet at cleanup time and distorts the checks that learn proof. Textual content that has been by means of each steps has weakened each sorts of hallucination-detection approaches. So far as I do know, no person has measured that second impact immediately but. The experiment is sitting there ready for somebody to run it.
Forgery closes the loop from the opposite facet. The identical methods that strip a watermark off can stamp one onto textual content the mannequin by no means wrote. So the label “watermarked” can fail each methods: lacking from machine textual content that was laundered, current on false textual content that was faked.
💬 Feedback and strategies are wellcome.
✉️ You may contact me [email protected]
References
[1] Regulation (EU) 2024/1689 of the European Parliament and of the Council of 13 June 2024 laying down harmonised guidelines on synthetic intelligence and amending Rules (EC) No 300/2008, (EU) No 167/2013, (EU) No 168/2013, (EU) 2018/858, (EU) 2018/1139 and (EU) 2019/2144 and Directives 2014/90/EU, (EU) 2016/797 and (EU) 2020/1828 (Synthetic Intelligence Act) (Textual content with EEA relevance) [https://eur-lex.europa.eu/eli/reg/2024/1689/oj]
[2] Measures for Labeling of AI-Generated Artificial Content material Doc. State Data Workplace Tongzi [2025] №2. Cybersecurity Administration, Ministry of Trade and Data Expertise, Ministry of Public Safety, State Administration of Radio and Tv [https://www.cac.gov.cn/2025-03/14/c_1743654684782215.htm]
[3] Farquhar, S., Kossen, J., Kuhn, L. et al. Detecting hallucinations in massive language fashions utilizing semantic entropy. Nature 630, 625–630 (2024). https://doi.org/10.1038/s41586-024-07421-0
[4] Kirchenbauer, J., Geiping, J., Wen, Y., Katz, J., Miers, I., & Goldstein, T. (2023, July). A watermark for big language fashions. In Worldwide convention on machine studying (pp. 17061–17084). PMLR.
[5] Google DeepMind, SynthID documentation and the Hugging Face Transformers integration (SynthIDTextWatermarkLogitsProcessor, BayesianDetectorModel). https://deepmind.google/applied sciences/synthid/ and https://huggingface.co/docs/transformers
[6] Fu, J., Zhao, X., Yang, R., Zhang, Y., Chen, J., & Xiao, Y. (2024, August). Gumbelsoft: Diversified language mannequin watermarking by way of the gumbelmax-trick. In Proceedings of the 62nd Annual Assembly of the Affiliation for Computational Linguistics (Quantity 1: Lengthy Papers) (pp. 5791–5808).
[7] Verma, A., Phan, N., & Trivedi, S. (2025). Watermarking degrades alignment in language fashions: Evaluation and mitigation. arXiv preprint arXiv:2506.04462.
[8] Jovanović, N., Staab, R., & Vechev, M. (2024). Watermark stealing in massive language fashions. arXiv preprint arXiv:2402.19361.
[9] Hwang, J., Park, S., & Okay, J. (2025). LLM Watermark Evasion by way of Bias Inversion. arXiv preprint arXiv:2509.23019.
[10] Wu, Z., Gong, G., Zhu, Q., Chen, Y., & Zhao, R. (2026). Linear Ensembles Wash Away Watermarks: On the Fragility of Distributional Perturbations in LLMs. arXiv preprint arXiv:2605.30501.
[11] Zhang, H., Edelman, B. L., Francati, D., Venturi, D., Ateniese, G., & Barak, B. (2023). Watermarks within the sand: Impossibility of sturdy watermarking for generative fashions. arXiv preprint arXiv:2311.04378.
