An independent stress test of SynthID-Text on two Gemma 4 models. We kept the key and attacked the output blind: one paraphrase by a small open model removed the mark and kept the answer, and the larger model carried a much fainter mark to begin with.
SynthID-Text is a watermark: text from a model that applies it can later be verified by anyone holding the key. It is not a general test of whether a text was written by a machine. We applied it, with our own key, to Gemma 4 E4B and Gemma 4 31B, generated answers to 400 everyday prompts, and attacked them with code that never saw the key or the detector. At a nominal 1% false-positive operating point (measured at 0.9% on held-out unmarked text), the small model’s untouched answers were verified 69% of the time and the large model’s 30%. The gap follows how much choice each model had. The large model sampled from about a third as much entropy per token; changing only the sampling temperature moved each model’s rate with its entropy, and the two models were verified equally often (45%) once their entropy was similar, with no appreciable change in judged answer quality and no change in false positives. Running the small model in 4-bit or with a second key changed nothing. Flattening spacing alone cut the large model’s rate from 30% to 22%. One paraphrase by a small open model brought both to about 2%, no higher than unmarked text after the same treatment. An independent judge model found that 92% of those paraphrased answers escaped detection, kept at least four fifths of the original’s statements and contradicted nothing in it. Google DeepMind documents that thorough rewriting weakens the mark; this report measures how little effort that takes, and what ordinary handling does on the way.
Small model’s watermarked answers still verified after one paraphrase by a small open model (95% interval 0.9 to 3.6), in line with the 1% false-positive rate. No key or detector was needed.
Paraphrased answers that both escaped detection and kept at least 80% of the original’s key statements, as judged by Qwen3.5-9B.
Untouched answers verified for Gemma 4 E4B and Gemma 4 31B. The larger model sampled from 2.7 times less entropy per token, leaving the mark less room. At similar entropy the two models were verified equally often.
The large model’s verification rate after line breaks and spacing were flattened, with no words changed: what a paste into a plain-text field does.
A text watermark nudges a model’s word choices according to a secret key, so that anyone holding the key can later check a passage for the pattern. A positive result links the text to a particular watermarking configuration. A negative result says nothing about who wrote it: text from another model, or watermarked text that has been changed, looks the same as human writing. Google DeepMind’s SynthID-Text is the best-known scheme for open models. It is built into the transformers library and runs with any model, and its documentation says plainly that detection weakens when text is thoroughly rewritten or translated and that it is less effective on factual answers.
So the weakness to paraphrase is known. What we wanted to measure is how little effort defeats it in practice, whether the attacked text is still a usable answer, what everyday handling such as copying and reformatting does to the mark, and whether a stronger model carries it better or worse than a small one. This is an independent stress test of a documented limitation, not the discovery of a new one. Han, Li, Ni and Zulkernine (2025) have already tested SynthID-Text against paraphrasing, copy-paste insertion, back-translation and synonym substitution on 1.3-billion-parameter models, and proposed a sturdier variant, SynGuard. We extend that work to current Gemma 4 models, with an attacker that never touches the key or the detector, with measured content retention for the attacked text, formatting-only changes, and a temperature intervention.
The attack code runs in a separate directory, never imports the scoring code and never reads a key. It gets only the watermarked answers. Every attack was applied to all 800 watermarked answers, and the same attacks were applied to the 800 unwatermarked answers to check that they do not create false positives:
Using a Gemma model to remove a Gemma watermark is fair: the weights are public, and the protection is meant to rest on the key, not on the model.
An evader gains little if the rewrite loses the content. We used Qwen3.5-9B, from a different model family than every generator and rewriter, as an independent judge. It first listed up to 12 key statements from each original watermarked answer (facts, names, numbers, steps, recommendations; about 10 on average), then marked each one PRESENT, CHANGED or MISSING in the attacked version and counted claims that contradict the original. Two controls bracket the judge: flattened spacing changes no words and should keep nearly everything, and deleting a quarter of the words should lose a lot. No people reviewed the outputs.
On held-out unmarked text the detectors flagged 0.9% (interval 0.5 to 1.5), and every model, source and prompt type stayed between 0.5% and 2.0%. Bootstrapping the calibration texts moves the weighted threshold between z = 2.25 and 2.44, so a rate quoted at 1% is a nominal operating point with real uncertainty around it, not an exact floor.
The small model’s watermarked answers were verified 68.8% of the time (64 to 73), the large model’s 30.5% (26 to 35). Table 1 tests the obvious explanations for the gap.
| E4B, bf16 | E4B, 4-bit | E4B, second key | 31B, 4-bit | |
|---|---|---|---|---|
| Flagged, all 400 answers | 68.8% | 68.0% | 66.8% | 30.5% |
| Flagged, open prompts | 96.5% | 96.5% | 94.5% | 51.0% |
| Flagged, constrained prompts | 41.0% | 39.5% | 39.0% | 10.0% |
| Flagged, first 200 tokens (matched prompts) | 87.4% | 30.0% | ||
| Median scored tokens per answer | 300 | 296 | 287 | |
| Sampling entropy per token (nats) | 0.355 | 0.354 | 0.133 | |
| Candidate tokens left after top-k/top-p | 2.58 | 1.37 | ||
| Watermark signal per token (mean g-value above 0.5) | 0.0206 | 0.0195 | 0.0080 |
These measurements show entropy tracking the gap, but on their own they are a correlation across two models. Section 4 changes the entropy of one model at a time.
| Entropy (nats per token) | Answers | From 31B | Signal, all | E4B | 31B |
|---|---|---|---|---|---|
| 0.00 to 0.11 | 240 | 61% | 0.0033 | 0.0026 | 0.0038 |
| 0.11 to 0.18 | 240 | 60% | 0.0079 | 0.0072 | 0.0084 |
| 0.18 to 0.30 | 240 | 44% | 0.0134 | 0.0139 | 0.0129 |
| 0.30 to 0.47 | 240 | 2% | 0.0229 | 0.0230 | 0.0147 |
| 0.47 to 0.99 | 240 | 0% | 0.0325 | 0.0325 |
To test the entropy explanation directly, we changed the sampling temperature of each model and nothing else: same prompts, same key, same top-k and top-p, same detector and thresholds. Lower temperature makes a model more certain of its next token; higher temperature gives it more choice. Entropy here, as in Table 1, is that of the pre-watermark sampling distribution. To check that temperature does not move the detector’s false-positive rate on its own, each model also generated unwatermarked answers at each temperature. Answer quality was scored 1 to 10 by the same independent Qwen3.5-9B judge against the human reference answer.
| Model | Temp. | Entropy | Verified | Open | Constrained | False positives | Quality (1 to 10) |
|---|---|---|---|---|---|---|---|
| E4B | 0.5 | 0.154 | 44.8 40.0 to 49.6 | 72.5 | 17.0 | 1.2 | 8.79 8.6 to 9.0 |
| E4B | 0.7 | 0.228 | 59.2 54.4 to 64.0 | 90.5 | 28.0 | 2.8 | 8.80 8.7 to 9.0 |
| E4B | 1.0 | 0.355 | 68.8 64.0 to 73.1 | 96.5 | 41.0 | 1.5 | 8.72 8.6 to 8.9 |
| E4B | 1.3 | 0.501 | 76.2 71.8 to 80.2 | 98.5 | 54.0 | 2.0 | 8.81 8.7 to 9.0 |
| 31B | 1.0 | 0.133 | 30.5 26.2 to 35.2 | 51.0 | 10.0 | 1.0 | 9.14 9.0 to 9.2 |
| 31B | 1.3 | 0.185 | 45.5 40.7 to 50.4 | 73.0 | 18.0 | 1.0 | 9.14 9.0 to 9.3 |
Raising the temperature raised verification for both models, and lowering it lowered it, in step with entropy. For E4B, going from 0.5 to 1.3 roughly tripled the entropy (0.15 to 0.50 nats per token) and took verification from 45% to 76%. For 31B, raising the temperature from 1.0 to 1.3 lifted its entropy from 0.13 to 0.18 and its verification from 30% to 46%. The two models meet where their entropy meets: E4B at temperature 0.5 (0.154 nats) and 31B at 1.3 (0.185) were verified 44.8% and 45.5% of the time, and within each entropy group of Table 2 the two models carry about the same signal per token. Unwatermarked answers at the same temperatures were flagged between 1.0% and 2.8%, with no trend across temperature, so the rise in verification is watermark signal, not a shifted null distribution. The judge found no appreciable quality difference. Paired on the same prompts, average scores at each temperature differed from temperature 1.0 by +0.01 to +0.09 points on the 10-point scale, and every 95% interval lay inside a margin of half a point. That is a judge-based check within this range, not proof that temperature never affects quality. So within one model, more sampling freedom gives a stronger mark, and the intervention supports sampling entropy as a major driver of the size gap in section 3, rather than model size alone. It does not rule out a smaller independent contribution from the model. For operators the trade is concrete: a deployment that runs a model cold for consistent answers gets a much weaker watermark.
| Attack | E4B verified | 31B verified | Runs kept | False positives |
|---|---|---|---|---|
| No attack | ||||
| Untouched | 68.8 64.0 to 73.1 | 30.5 26.2 to 35.2 | 100% | 1.5 / 1.0 |
| Spacing and line breaks flattened | 64.8 59.9 to 69.3 | 22.0 18.2 to 26.3 | 82% | 2.5 / 0.8 |
| Word edits (on flattened text) | ||||
| Typos in 10% of words | 55.5 50.6 to 60.3 | 11.2 8.5 to 14.7 | 56% | 2.5 / 1.8 |
| Typos in 25% of words | 27.0 22.9 to 31.6 | 4.0 2.5 to 6.4 | 32% | 2.0 / 0.8 |
| Typos in 50% of words | 5.8 3.9 to 8.5 | 1.0 0.4 to 2.5 | 10% | 0.8 / 1.0 |
| Delete 10% of words | 55.0 50.1 to 59.8 | 10.0 7.4 to 13.3 | 60% | 2.5 / 1.0 |
| Delete 25% of words | 20.5 16.8 to 24.7 | 3.5 2.1 to 5.8 | 37% | 1.5 / 1.5 |
| Delete 50% of words | 2.8 1.5 to 4.9 | 0.8 0.3 to 2.2 | 14% | 1.8 / 1.2 |
| Swap neighbors, 10% of words | 48.0 43.1 to 52.9 | 7.8 5.5 to 10.8 | 49% | 1.5 / 1.0 |
| Swap neighbors, 25% of words | 15.0 11.8 to 18.8 | 2.0 1.0 to 3.9 | 22% | 2.0 / 2.2 |
| Swap neighbors, 50% of words | 3.2 1.9 to 5.5 | 1.5 0.7 to 3.2 | 11% | 1.2 / 1.8 |
| Less text | ||||
| Keep first 100 words | 49.2 44.4 to 54.1 | 9.8 7.2 to 13.1 | 82% | 1.2 / 0.5 |
| Keep first 50 words | 29.0 24.8 to 33.6 | 5.8 3.9 to 8.5 | 84% | 1.5 / 1.2 |
| Keep first 25 words | 12.8 9.8 to 16.4 | 2.5 1.4 to 4.5 | 99% | 1.5 / 0.8 |
| Marked text is 50% of a longer text | 30.8 26.4 to 35.4 | 7.8 5.5 to 10.8 | 41% | 2.2 / 1.5 |
| Marked text is 25% of a longer text | 20.2 16.6 to 24.5 | 4.2 2.7 to 6.7 | 21% | 1.5 / 2.5 |
| Marked text is 10% of a longer text | 8.2 5.9 to 11.4 | 4.8 3.1 to 7.3 | 8% | 2.8 / 2.0 |
| Rewriting | ||||
| 25% of sentences rewritten (small rewriter) | 63.0 58.2 to 67.6 | 19.5 15.9 to 23.7 | 78% | 1.2 / 0.8 |
| 50% of sentences rewritten (small rewriter) | 50.5 45.6 to 55.4 | 9.8 7.2 to 13.1 | 57% | 0.8 / 1.2 |
| 75% of sentences rewritten (small rewriter) | 23.5 19.6 to 27.9 | 5.2 3.5 to 7.9 | 36% | 0.8 / 1.5 |
| French and back (small rewriter) | 48.0 43.1 to 52.9 | 11.0 8.3 to 14.4 | 64% | 2.5 / 0.5 |
| French and back (large rewriter) | 44.2 39.5 to 49.1 | 9.5 7.0 to 12.8 | 58% | not run |
| Paraphrased once (small rewriter) | 1.8 0.9 to 3.6 | 1.8 0.9 to 3.6 | 12% | 0.8 / 2.2 |
| Paraphrased once (large rewriter) | 2.0 1.0 to 3.9 | 1.2 0.5 to 2.9 | 12% | not run |
| Paraphrased twice (small rewriter) | 4.5 2.9 to 7.0 | 0.8 0.3 to 2.2 | 18% | 1.5 / 1.2 |
| Paraphrased twice (large rewriter) | 5.5 3.7 to 8.2 | 2.5 1.4 to 4.5 | 25% | not run |
Applied to unwatermarked answers, the same attacks produced false positives between 0.5% and 2.8% per row. That is broadly consistent with the nominal 1% rate at this sample size (400 texts per row), though it does not show that transformations have no effect at all on false positives. It does show that the falls in Table 4 are lost watermark evidence, not attacks pushing ordinary text toward the threshold.
Each watermarked token depends on the four before it, so one changed word spoils the evidence at that point and at the next several. Typos or deletions in a tenth of the words left only about 56% to 60% of the original 5-token runs and cut the small model’s verification from 69% to about 55%; the large model fell to about 10%. Shorter text means less evidence: cut to 50 words, the small model’s answers were verified 29% of the time, at 25 words 13%. Mixed into human text, the mark fades with its share of the passage.
A full paraphrase kept only about 12% of the original 5-token runs and removed the mark. Whichever model rewrote and whichever wrote the original, verification fell to between 1% and 2%, inside the range of the false-positive rate, and paraphrasing twice changed nothing. The small rewriter did as well as the large one, so the attacker needs only a model that runs on a laptop.
Partial rewrites show the mark is local: it survives in whatever text is left untouched. Rewriting a quarter, half or three quarters of the sentences left 78%, 57% and 36% of the 5-token runs, and the small model’s answers were verified 63%, 51% and 24% of the time. Across all the attacks on the small model’s answers, verification rose from 4% when under a tenth of the runs survived to about 50% when more than half did; the few attacks that kept nearly every run were the short crops, where too few tokens remain to decide. The French round trip kept about 60% of the runs, and the small model’s answers were still verified 48% of the time against 69% untouched, because translating back tends to land on wording close to the original.
| Attack | Statements kept | 80%+ kept | Contradiction | Unverified, faithful |
|---|---|---|---|---|
| Spacing flattened (control: no words changed) | 97.5% | 99.1% | 2.9% | 34.2% / 74.5% |
| Paraphrased once, small rewriter | 96.6% | 97.6% | 5.2% | 91.8% / 91.8% |
| Paraphrased once, large rewriter | 96.2% | 97.5% | 6.2% | 90.2% / 89.8% |
| Paraphrased twice, small rewriter | 95.4% | 96.8% | 6.6% | 86.0% / 90.8% |
| French and back, small rewriter | 95.3% | 95.8% | 8.2% | 45.8% / 76.8% |
| 50% of sentences rewritten, small rewriter | 95.8% | 96.9% | 6.6% | 43.0% / 82.5% |
| 25% of words deleted (control: damaged text) | 72.7% | 49.4% | 20.5% | 30.8% / 34.8% |
The judge marks about 2% of statements as changed even when no words changed, which is its own noise. Against that, one paraphrase kept 96.6% of statements, and nearly every paraphrased answer kept at least four fifths of them; deleting a quarter of the words, by contrast, kept only about 73%. A paraphrase added a contradiction in about one answer in twenty, against one in thirty-five for the unchanged control. So the evasion is practical: 96% of the small model’s answers, and the same share of the large model’s, came out of a single paraphrase both unverifiable and substantially intact, and 92% did so with no contradiction of the original at all. A model judge is not a person, and the statements it lists are its own, so these numbers measure information retention, not every nuance of tone or emphasis.
The SynthID paper describes a trained Bayesian detector that learns, from examples, how strongly each layer of the watermark shows up in real text. We trained it (the BayesianDetectorModel in transformers) on clean text from the even-numbered prompts only, then scored every attack on the odd-numbered prompts, which it never saw, alongside the mean and weighted detectors with thresholds set on the same held-out unmarked texts.
| Attack | Gemma 4 E4B | Gemma 4 31B |
|---|---|---|
| Untouched | 62.5 / 67.5 / 66.0 | 13.5 / 25.5 / 25.0 |
| Spacing flattened | 55.0 / 64.0 / 63.5 | 8.0 / 17.5 / 21.0 |
| Typos in 10% of words | 42.0 / 55.0 / 55.0 | 3.5 / 10.5 / 9.5 |
| Keep first 50 words | 14.0 / 29.5 / 9.0 | 3.0 / 5.0 / 0.5 |
| Keep first 25 words | 7.0 / 14.5 / 0.0 | 1.0 / 1.0 / 0.0 |
| Marked text is 25% of a longer text | 11.0 / 19.5 / 23.5 | 2.5 / 3.5 / 7.0 |
| 50% of sentences rewritten | 34.0 / 49.0 / 50.0 | 3.0 / 9.0 / 9.5 |
| French and back (small rewriter) | 31.5 / 47.0 / 47.5 | 4.5 / 10.5 / 13.5 |
| Paraphrased once (small rewriter) | 0.0 / 1.0 / 1.0 | 0.5 / 1.5 / 2.0 |
The weighted detector is a free improvement over the mean: 5 to 15 points on clean and lightly edited text, and it nearly doubles what is verified on the large model. Our trained Bayesian detector matched the weighted one on full-length text, did slightly better when the marked passage was mixed into human text, and did worse on short excerpts. That last result belongs to our setup rather than to Bayesian detection in general: we scored with a summed log-likelihood ratio that grows with length and set its threshold mostly on full-length answers, and a length-aware threshold or different training data could behave differently. On paraphrased text all three sat at the false-positive rate. A better detector reads a faint mark more reliably; it cannot read one that is no longer there.
Line breaks, list markers and runs of spaces are tokens, and the watermark is applied to them like any other token. Flattening them to single spaces, which changes no words and kept 97.7% of statements, removed about a fifth of the 5-token runs and took the large model’s verification from 30% to 22% and the small model’s from 69% to 65%. This is what happens, with no intent to evade, when an answer is pasted from a chat window into an email, a ticket system, a CMS field or a plain-text form. Anyone who plans to verify their own model’s output after it has passed through other systems should test detection on text that has been through those systems, not on the raw output.
A watermark is a label, not a lock. SynthID-Text did what it promises for text that nobody changes: a clean answer from the small model on an open prompt was verified about 96% of the time. Removing it took one prompt to a free model that runs on a laptop, with no key and no way to check the result, and the rewritten answer still said what the original said. A business or school can use the mark to recognize its own model’s output when it is passed along unchanged. It cannot use a missing mark as evidence that a person wrote a text.
Confident sampling leaves less room. The 31B model’s answers were verified less than half as often as the small model’s at the default temperature, and only one in ten on constrained tasks such as summarizing or extracting from a passage. The temperature experiment supports sampling freedom as a major driver of that gap, rather than model size alone: turning E4B down to 0.5 brought it to the large model’s level, and turning 31B up to 1.3 lifted it, with no appreciable change in judged quality. Anyone relying on the watermark should test detection on their own prompts at their own sampling settings, and know that low-temperature or near-deterministic deployments carry the weakest marks.
What is new here. The SynthID paper, its documentation and Han et al. (2025) already show that thorough rewriting and translation weaken the mark. The measurements here add how cheap the rewrite can be, that it preserves the content, how far ordinary formatting changes go on their own, and a within-model test showing that the mark’s strength follows sampling entropy, with false positives checked at every temperature.
The code, configurations, prompts, generated and attacked texts and per-text scores for every detector are in the study repository, github.com/strifero/watermark-study, with instructions for repeating the study with a new key. We do not publish our watermark keys or the sampling tables built from them, which would let anyone forge or verify our mark. The prompts and answers are ordinary Dolly questions with nothing harmful in them.
Runs took place between 9 and 12 November 2026 on two NVIDIA RTX A4500 GPUs (20 GB each) in the trifero lab, about 17 GPU-hours in all. Generation, entropy and scoring used transformers 5.17.0 (SynthID-Text and the Bayesian detector from transformers.generation.watermarking), the 31B model and the E4B control in bitsandbytes 4-bit NF4. Rewrites and the fidelity judge used vLLM 0.31.0 across both cards. SynthID builds its sampling table with the GPU’s random number generator, and a table rebuilt on the CPU from the same seed is different and scores watermarked text as unmarked; the repository explains how to build the table once on a GPU and keep it with the key, which is key material and must not be published. The Bayesian detector was trained with Adam (learning rate 3e-3, batch 64, 400 epochs, best held-out loss kept).