A text-watermark robustness test is useful when you can connect its findings to the conditions that produced them. Which watermark was tested? What could the attacker query? What text did they generate, and how did the researchers decide that it counted?

Those questions determine whether a result helps you assess a system you use. Robustness here means how a watermark holds up under specified attacks. A finding about one scheme, model pair and set of writing tasks needs those details attached.

The 24 June 2024 arXiv version of Watermark Stealing in Large Language Models, by Nikola Jovanović, Robin Staab and Martin Vechev, provides a concrete example. The work was published at ICML 2024; the methods discussed below come from that June arXiv version. This is a reading of reported methods, with supporting repository material, rather than an independent replication.

Our earlier coverage of watermark detection provides context on what a detector signal can establish. Here, the narrower question is how to read an attack evaluation.

Identify the watermark and the two models

Start with the study’s spoofing setup: KGW2-SelfHash, a watermarked Llama-7B model and a Mistral-7B attacker model, evaluated on Dolly creative-writing prompts. Spoofing means generating text that triggers the target watermark detector without using the watermark’s secret key.

In the June version’s description, SelfHash uses the previous three tokens together with self-seeding to determine the watermark’s token preferences. That specificity matters: “an AI watermark” covers more than the particular construction being evaluated.

The repository’s example instructions pair Llama-7B and Mistral-7B with SelfHash. Its model configurations identify meta-llama/Llama-2-7b-chat-hf and mistralai/Mistral-7B-Instruct-v0.1. These are main-branch files checked on 6 November 2026, not an identified publication-time commit.

The useful inference is tied to a named research setup. Applying it to another deployment would require evidence about that deployment’s scheme, parameters, models and access controls.

Unpack what “black-box access” allows

A threat model describes what the attacker knows and can do. In section 3 of the June paper, the attacker knows that watermarking is present and knows the scheme’s parameters, but lacks the secret key. They can obtain full watermarked generations and use an auxiliary model to help estimate which token choices reflect the watermark.

The main setting, labelled B0/D0, withholds two things from the attacker:

  • B0: unwatermarked responses from the model owner.
  • D0: access to the watermark detector.

These conditions explain why the example is interesting. The attack procedure seeks to learn watermark preferences from generated text without stealing the key or consulting a detector. It still assumes known scheme parameters and access to watermarked outputs.

The researchers use a detector to evaluate the generated text. That measurement does not mean the attacker could query it. The paper separately defines D1, where the attacker receives binary detector answers. When reading a result, keep the access setting beside it: allowing an attacker to check candidate outputs changes their opportunities.

Separate the examples collected from the prompts tested

In section 6, the authors describe collecting 30,000 watermarked responses using C4 RealNewsLike prompts, with responses up to 800 tokens. They explicitly report no overlap between this collection and the later evaluation datasets.

For the Dolly example, the question is therefore whether preferences learned from one collection can be used on separate creative-writing prompts. That separation gives the evaluation more value than simply reusing the collection prompts.

It also leaves a boundary around the conclusion. The paper evaluates selected writing tasks, including creative writing, harmful queries, book reports and fake news. It reports average generated attack text of about 450 tokens, a different quantity from the collection’s maximum response length.

Those conditions do not establish performance on short workplace messages, code, multilingual documents or arbitrary chains of editing and translation. To assess a deployment used for those purposes, look for evaluation text that resembles the work you care about.

Check how the study judges useful output

Making a detector react is only part of a useful spoofing evaluation. The resulting text also has to satisfy the writing task.

The June paper’s metrics section includes GPT-4 ratings of accuracy, consistency and style, with a low-quality cutoff of 6.5. This adds a task-quality check to the detector measurement. A generated passage that produces a signal while failing its task has a different practical meaning from usable writing.

The quality assessment is the authors’ model-based judging procedure. It should be read with that attribution, rather than treated as an independent human verdict.

When reading a reported rate, also establish what happens to outputs that fail the quality check. Are they counted as failed attempts or removed before calculating the fraction? The exact denominator and final metric postprocessing remain unverified in this source inspection, so this walkthrough does not translate the reported metrics into an attack-success rate.

Repetition needs similar care. The June methods describe averaging over five attacks per prompt, while Figure 3 distinguishes average and selected-best curves. An average describes repeated attempts differently from a result obtained by selecting a candidate. Ask how selection works and what information the attacker needs to make it.

Ask which text was used to calibrate the detector

A detector threshold determines when a score becomes a positive signal. Its false-positive rate describes how often reference text is flagged under the calibration conditions.

Section 6 of the June paper describes calibration on non-adversarial text. The exact reference corpus, its size and language, the calibration procedure and associated uncertainty remain unverified in this inspection. That limits how confidently this walkthrough can connect the threshold to a new population of documents. It is not a finding that the final paper lacks those details.

Keep this reference population separate from the C4 prompts used to collect watermarked responses. A collection dataset does not automatically identify the text used to calibrate a detector.

Accidental false positives and deliberate spoofing answer different questions. The former concerns ordinary reference text triggering a signal; the latter concerns an attacker deliberately imitating it. Neither measurement alone gives the probability that a particular person used a particular model. That inference requires relevant population assumptions and evidence about the individual case.

Follow the resources through the whole procedure

The study separates collecting watermarked examples, learning from them and generating attack text. That separation is useful when interpreting a query budget.

A collection budget accounts for obtaining examples. Auxiliary-model computation, subsequent generation, repeated attempts and quality judging are additional resource components. A cost attached to the first stage cannot establish the total cost of carrying out the procedure.

The released spoofing configuration makes some choices inspectable, including the collection-query setting and multiple random seeds. It offers concrete implementation detail, while retaining the main-branch version limitation described above.

Trace released code through to the reported metric

The repository provides code, configurations, static data and a link to watermarked server outputs. Its README describes the query, learn and generate stages, along with dependencies and an OpenAI key for quality judging.

The same README says that paper metrics were postprocessed at chosen false-positive rates and that perplexity, a model-based measure of text predictability, was recomputed using Llama-13B. A raw threshold in a configuration file therefore cannot, by itself, reconstruct the calibrated results.

The output archive and final metric postprocessing were not verified here, and the experiments were not rerun. The materials make further examination possible; independent replication would require following the procedure through to its outputs and measurements.

For a study you intend to rely on, trace one result all the way through: the named scheme and models, attacker permissions, collection and evaluation text, quality rule, calibrated threshold, selection procedure and calculation code. Each connection tells you where the evidence applies and what another team would need to check.