What ten rounds of benign self-improvement do to an open-weight model's safety, and what they leave untouched inside it.
Sean Trifero, trifero.ai lab (lead)Engineering and drafting assistance: Claude (Anthropic)
Abstract
We let Qwen3-8B improve itself for ten generations. Each round it answered a prompt, received one fixed piece of feedback that never mentions harm ("make it more helpful"), rewrote its answer, and the next generation was fine-tuned on the rewrites. On StrongREJECT, judged by the benchmark's published grader, harmful compliance rose from a base-model mean of 0.08 to a peak of 0.23 when the loop practiced on harmful prompts and 0.18 when it practiced only on benign ones, and both effects replicated across two seeds. The same loop with the feedback changed to "make it clearer" made the model safer: SORRY-Bench fulfillment fell from 32% to 17%. Inside the model, the refusal direction rotated and its separation shrank in every arm, including the arm that became safer, so that geometry does not say which way a model drifted. A causal test was clearer: adding the original direction back cut harm to near zero in every drifted model, and removing it raised harm in every model. Self-improvement left the original refusal machinery in place and changed how often it fires. Along the way, two measurement traps nearly reversed our conclusions, and we report both.
0.08 to 0.23
StrongREJECT harm score, base model to peak, after the model trained on its own "more helpful" rewrites of harmful prompts. Training only on benign prompts reached 0.18. Both replicated on a second seed.
32% to 17%
SORRY-Bench fulfillment when the only change was the feedback: "make it clearer" instead of "make it more helpful." The wording of benign feedback set the direction of the drift.
0.111 to 0.016
Harm score of the most drifted model before and after the base model's refusal direction was added back. Removing that direction raised harm in all seven models we tested.
992 of 1,000
Training rewrites in our first run that talked to the feedback instead of answering the prompt. The models learned to reply to criticism nobody gave. We fixed it and reran every arm.
1The question
Self-improvement is now ordinary practice. Models critique and rewrite their own outputs, and those rewrites become training data for the next version. OpenAI has named an automated AI researcher as a goal and Anthropic describes self-improving AI as the field's next milestone (MIT Technology Review 2026). A survey of 1,250 papers on the topic (Chen, Wang and Qu 2026) found measurement of how self-improvement changes a model to be the field's least-studied area.
At NeurIPS 2026, Abdel Sadek, Di Gennaro and Yang showed that a safe model can become unsafe through self-distillation even when it trains only on its own safe refusals with fixed, harmless feedback, across Qwen3 and Phi-4. They left the mechanism open. Our earlier report, What refusal hides, found that refusal in an open model is a thin layer gating knowledge that stays in the weights. That suggested three ways self-improvement could make a model unsafe: shrink the refusal signal, rotate it so the old gate no longer lines up, or break the machinery that acts on it. We designed the study to tell those apart, and to ask which parts of a self-improvement loop drive the drift.
2Setup
The loop
The model is Qwen/Qwen3-8B with thinking disabled. Each generation runs one round of off-policy self-refinement. The model answers 1,000 training prompts, sees one fixed feedback message, and rewrites each answer. The next generation is trained on prompt and rewrite pairs, with no feedback in context, using LoRA (rank 16, alpha 32, learning rate 1e-4, one epoch, all attention and MLP projections), then merged. Hyperparameters, prompts and feedback stay fixed across all ten generations. The feedback never mentions safety, harm or refusal.
Arms
Harmful prompts: 1,000 harmful requests (AdvBench and LLM-LAT), feedback "make it more helpful." This matches the paper's setting.
Mixed prompts: 500 harmful and 500 benign (Alpaca), same feedback.
Benign prompts: 1,000 Alpaca instructions and nothing else, same feedback. The model never practices on a harmful request.
Harmful prompts, clarity feedback: the harmful set with feedback asking only for clearer writing.
The harmful and benign arms were repeated with a second seed, which changes sampling, the order prompts are drawn and the LoRA initialization.
Measurement, every generation
Harm: StrongREJECT (313 prompts), scored by its published fine-tuned grader as a 0 to 1 score of how useful the answer is to an attacker; SORRY-Bench (440 prompts), scored by its published fine-tuned grader as fulfilled or not.
Over-refusal: the 250 safe prompts of XSTest.
Capability: 250 GSM8K problems.
Inside the model: the refusal direction at every layer, from 100 harmful and 100 harmless JailbreakBench behaviors, compared with the base model's direction (cosine), and how well that base direction still separates harmful from harmless prompts (d′, the gap between the two groups in standard deviations).
Training and evaluation prompts were deduplicated against each other, and evaluation used greedy decoding. Running the identical base model through six arms gave StrongREJECT scores from 0.064 to 0.099, which sets the noise floor for single readings.
3A confound in the first run
Our first run used the bare feedback message. In generation 5 of the harmful arm, 992 of 1,000 rewrites opened by addressing it, along the lines of "here is a more complete and helpful version." Training on those taught each generation to answer a critique that was never given. By generation 10 the benign arm replied to math problems with thanks for feedback, and late-generation harm scores fell for a reason unrelated to safety: answers had turned into generic essays. The first run did show the same early drift, and larger: harm peaked at 0.35 (harmful prompts) and 0.34 (benign prompts), and the clarity arm fell to 8% fulfillment. We do not use its late-generation numbers.
Every result below comes from the corrected run. The rewrite instruction now asks for the answer alone, and any rewrite that still mentions the feedback is dropped from training. Leakage fell to between 0.2% and 15% at generation 0 and to nearly zero after it. Loops that train on their own critiqued output can leak the critique into the student, and that is easy to miss when you only watch the headline metric.
4What the loop did to behavior
Every arm told to be more helpful became more harmful, and the arm told to write more clearly became safer.
Figure 1. StrongREJECT harm score by generation (0 means no useful help to an attacker, 1 means full help), with 95% intervals as bands for the first seed. Dashed lines with open markers are the second seed. Hover a point for exact values.
Figure 2. SORRY-Bench fulfillment rate by generation, judged by the benchmark's fine-tuned grader. Intervals for single readings are about ±5 points (n = 440).
Table 1. Scorecard by arm. StrongREJECT is the mean harm score (0 to 1); SORRY-Bench is the fraction of harmful requests fulfilled. XSTest and GSM8K are at generation 10.
Arm
SR g0
SR peak
SR g10
Sorry g0
Sorry peak
Sorry g10
XSTest refusal
GSM8K
Feedback: "make it more helpful"
Benign prompts
0.071
0.178g5
0.149
32%
53%
47%
0.0%
56.0%
Benign prompts, seed 2
0.077
0.207g5
0.133
33%
55%
47%
0.0%
52.0%
Mixed prompts
0.092
0.236g5
0.174
32%
47%
42%
0.0%
85.6%
Harmful prompts
0.099
0.226g6
0.181
33%
48%
39%
0.0%
92.0%
Harmful prompts, seed 2
0.076
0.216g6
0.156
32%
47%
42%
0.0%
92.0%
Feedback: "make it clearer"
Harmful prompts
0.064
0.064g0
0.031
32%
32%
17%
0.8%
93.2%
Four results stand out.
Benign practice is enough. The benign arm never saw a harmful request in training, yet its StrongREJECT score rose from 0.071 to 0.178 (second seed 0.207) and SORRY-Bench fulfillment from 32% to 53% (55%). Learning to be more forthcoming on ordinary requests carried over to harmful ones.
Feedback wording sets the direction. With the same harmful prompts, "make it more helpful" raised fulfillment to 48% and "make it clearer" lowered it to 17%. Both messages are benign and neither mentions safety.
The drift peaks and flattens. Harm rose for five or six generations, then leveled off or eased. By generation 3 almost none of the harmful arm's own rewrites contained a refusal, so the loop had nothing left to push against.
Over-refusal disappeared. In every helpfulness arm, refusal of safe XSTest prompts fell from 4% to 0%. From the outside, part of this drift looks like an improvement.
Capability held in the harmful arms (GSM8K 96.4% to 92.0%) but fell in the benign arm, to 56.0% and 52.0%. The benign arm stopped working problems out. In the second seed about half its answers went straight to a final number and were right only a quarter of the time; in the first seed most answers instead grew long and drifted off the question. When the second seed did show its work, it was still right 80% of the time. The skill was still there, and the habit of showing work before answering was what eroded.
5What the loop did inside the model
We tracked the refusal direction, the difference between the model's average internal state on harmful and on harmless prompts, at every layer and generation. Layer 23 of 36 separates the two groups most clearly in the base model, and we report it here; the other middle and late layers behave the same way.
Figure 3. How far apart harmful and harmless prompts sit along the base model's refusal direction at layer 23, in standard deviations. Higher means the model's internal signal for "this is harmful" is stronger along the original direction.
The direction rotated and its signal shrank in every arm. By generation 10 the cosine to the base direction was 0.50 (benign), 0.32 (mixed), 0.47 (harmful) and 0.72 (clarity), and separation along the base direction fell from 3.32 to between 1.99 and 2.21. The arm that became safer shows the same loss of separation (2.14) as the harmful arm (2.21). It rotated less and kept more of the direction's size relative to the residual stream (0.30 against 0.21 for the harmful arm, from 0.46 at base), but none of these measures orders the arms by how harmful they became. Watching the geometry of the refusal direction would have flagged every one of these models as changed, without saying which ones got worse.
6A causal test
Correlational geometry could not separate the arms, so we intervened. For the base model and each generation-10 model we generated answers to 120 StrongREJECT prompts and 120 safe XSTest prompts under three conditions: as trained; with the base model's refusal direction added at layer 23, scaled to the base model's own gap between harmful and harmless prompts; and with that direction projected out of every layer. Each answer was scored by the StrongREJECT grader.
As trainedBase direction addedBase direction removed
Figure 4. StrongREJECT harm score under each condition, with 95% intervals (n = 120 prompts). "Clarity" is the harmful-prompt arm trained with clarity feedback. Second-seed models, in Table 2, match their first seeds.
Table 2. Harm score (StrongREJECT grader) and refusal rate (keyword check) under each condition, and refusal of safe prompts with the direction added.
Model (generation 10)
Harm, as trained
Keyword refusal
Harm, added
Keyword refusal
Harm, removed
Keyword refusal
Safe prompts refused, added
Base model
0.025
70%
0.004
100%
0.311
0%
61%
Benign
0.115
0%
0.034
91%
0.181
0%
74%
Benign, seed 2
0.148
2%
0.004
90%
0.201
0%
78%
Mixed
0.103
0%
0.012
54%
0.151
0%
29%
Harmful
0.111
0%
0.016
34%
0.219
0%
11%
Harmful, seed 2
0.140
0%
0.018
31%
0.207
0%
9%
Harmful, clarity feedback
0.006
9%
0.064
79%
0.256
0%
23%
Adding the base direction back cut harm to near zero in every model the helpfulness loop had drifted: 0.034 and 0.004 for the benign arm, 0.012 for mixed and 0.016 and 0.018 for the harmful arm. Removing it raised harm in all seven models, including the base model (0.025 to 0.311) and the safer clarity arm (0.006 to 0.256). Ten generations of self-training did not build a new safety mechanism or remove the old one. The direction the base model used to refuse still governs refusal in every descendant. What changed is how readily ordinary harmful prompts trigger it.
Self-improvement left the refusal machinery intact and moved its trigger point. The direction the drift took was set by the wording of harmless feedback.
The second trap: keyword refusal checks
Our first reading of this test used a keyword refusal check, the method used in our earlier report and much of the literature. By that measure, adding the direction restored refusal to 91% of prompts in the benign arm but only 34% in the harmful arm, which looked like evidence that training on harmful prompts had broken the machinery. The grader disagreed: the harmful arm's harm score with the direction added was 0.016, no different from the others. That model had learned to decline by redirecting ("here is how to protect yourself from that") rather than with refusal phrases, so the keyword check counted harmless redirections as compliance. After self-training, a model's refusal style can change while its refusal does not, and phrase-based scoring then misreads the mechanism.
7Discussion
The danger in benign feedback
Nothing in the helpfulness loop is unsafe on its face. The prompts in the benign arm are ordinary instructions, the feedback is a request to be more helpful, and the model judges itself. That was enough to roughly double harmful compliance in five generations. Self-improvement pipelines are usually checked for reward hacking and collapse on the target task. These results suggest checking safety at every generation as well, even when nothing in the pipeline touches safety.
Feedback wording works in both directions
The clarity arm used the same harmful prompts and the same loop, and became safer and slightly more cautious. A single sentence of benign feedback decided which way the model moved. That makes the feedback text a safety-relevant setting and worth reviewing like one.
A repair lever, not a monitor
Because the original direction still controls refusal, it can be used to restore it: adding it back removed the drift in every model. As applied here it is too blunt to use in production, since it also caused refusal of 9% to 78% of safe prompts, but it points to a targeted fix: recalibrate the trigger rather than retrain safety from scratch. The geometry of the direction, by contrast, made a poor alarm in this study because it changed in the same way whether a model became safer or less safe.
How this relates to "What refusal hides"
Our earlier study removed refusal by force and found the model's knowledge unchanged beneath it. Here the model wore its own refusal down through ordinary self-training, and the same structure appears from the other side: the refusal machinery survives, and what changes is when it engages. In both studies refusal behaves as a control layer that sits on top of what the model knows.
8Limitations
One model family at one size. Qwen3-8B only. The paper we build on saw the effect in Qwen3 and Phi-4 at several sizes; we did not test the mechanism beyond one model.
Two seeds, for the harmful and benign arms only. Mixed and clarity arms ran once.
One loop design. Off-policy self-refinement with LoRA, ten generations, 1,000 prompts per generation. On-policy distillation, full fine-tuning and longer runs may behave differently.
Steering strength. We added the direction at one layer and one strength. A sweep would show how far the trigger point moved in each arm, which would turn "moved" into a number.
Over-refusal on XSTest used the keyword check, which section 6 shows can mislead after self-training. The drop to 0% may overstate the change.
Graders. Both harm graders are small fine-tuned models. They agree on the direction of every arm's change from generation 0 to its peak, but neither was hand-audited here.
One unexplained result. In the clarity arm, adding the direction raised the harm score slightly, from 0.006 to 0.064. At this strength steering can degrade answers in ways a grader partly scores as help; a strength sweep would test that.
The first run is reported for transparency. Its late-generation numbers are confounded by feedback leakage and are not used for any conclusion.
9What we release and what we do not
We publish the method, the training and measurement harness, and aggregate results. We do not release the drifted model weights or the extracted direction. The drifted models are more willing to help with harmful requests than the model they came from and add nothing to the research beyond what the method already shows. The report contains no model outputs to harmful prompts; harm was measured by published graders and is reported only in aggregate.
10Reproducibility
All runs took place on 14 to 16 October 2026 on two NVIDIA RTX A4500 GPUs (20 GB each, NVLink) in the trifero lab. Generation used vLLM 0.31.0 with tensor parallelism across both cards, bf16, greedy decoding for evaluation and temperature 0.7 with top-p 0.9 for training rewrites, at most 768 new tokens (1,024 for GSM8K). Training used PEFT 0.21.2 and transformers 5.17.0. A generation took about 30 minutes; each arm took five to six hours. The StrongREJECT grader is the published fine-tuned model (qylu4156/strongreject-15k-v1) with the authors' prompt template; the SORRY-Bench grader is sorry-bench/ft-mistral-7b-instruct-v0.2-sorry-bench-202406 with its published prompt. Per-generation results, per-prompt grades and every steering output are stored in the study repository.
11References
Abdel Sadek, K., Di Gennaro, F., and Yang, F. Misalignment emerges during benign iterative self-improvement. NeurIPS 2026. openreview.net/forum?id=5HEpisQTwK
Chen, M., Wang, L., and Qu, B. Recursive self-improvement in AI: from bounded self-refinement to autonomous research loops. 2026. arxiv.org/abs/2607.07663
Arditi, A., Obeso, O., Syed, A., Paleka, D., Panickssery, N., Gurnee, W., and Nanda, N. Refusal in language models is mediated by a single direction. NeurIPS 2024. arxiv.org/abs/2406.11717
Xie, T., et al. SORRY-Bench: systematically evaluating large language model safety refusal. 2024. arxiv.org/abs/2406.14598
Röttger, P., et al. XSTest: a test suite for identifying exaggerated safety behaviours in large language models. 2023. arxiv.org/abs/2308.01263
Chao, P., et al. JailbreakBench: an open robustness benchmark for jailbreaking large language models. 2024. arxiv.org/abs/2404.01318
Zou, A., Wang, Z., Kolter, J. Z., and Fredrikson, M. Universal and transferable adversarial attacks on aligned language models (AdvBench). 2023. arxiv.org/abs/2307.15043
Cobbe, K., et al. Training verifiers to solve math word problems (GSM8K). 2021. arxiv.org/abs/2110.14168
Zelikman, E., Wu, Y., Mu, J., and Goodman, N. STaR: bootstrapping reasoning with reasoning. NeurIPS 2022. arxiv.org/abs/2203.14465