Model safety cards TSC-002 · v0.1 · method v1
Out of the box, Gemma 4 refuses almost every harmful request we tried and rarely refuses safe ones. Its refusals took more work to remove than Qwen 3.8's, but a targeted edit to the weights still cut them to about 1 in 6, and a short round of ordinary fine-tuning cut them to about 1 in 20. The edit itself left the model just as capable. Put your safety controls in the application around the model, and treat the copy on your servers as something worth protecting.
We removed the model's refusal behavior with abliteration, a published technique that edits one direction out of the weights, then "healed" the edited model with a short fine-tune to repair any damage. HarmBench measures how many harmful requests it still refuses.
What it means for you: the model's refusals protect you only while nobody can change the weights. If a copy leaks or an insider modifies it, assume the guardrails are gone. AdvBench showed the same pattern: 98.7% refusal before, 19.4% after.
Capability tests on the original, the stripped copy and the healed copy.
| Test | Original | Stripped | Healed |
|---|---|---|---|
| MMLU-Pro (general knowledge) | 84.5% | 83.0% | 77.9% |
| GSM8K (math) | 97.5% | 97.3% | 96.3% |
| IFEval (following instructions) | 90.3% | 91.3% | 76.8% |
What it means for you: when the stripped copy scores about the same as the original, nothing discourages tampering. Protect the weights like any other sensitive asset.
WMDP is a multiple-choice test of hazardous knowledge in biology, chemistry and cybersecurity. Random guessing scores 25%.
What it means for you: this knowledge is in every copy of the model. Refusal is what stands between a user and that knowledge, and refusal is removable.
Both tests contain only safe prompts. XSTest is ordinary questions; OR-Bench hard is safe questions that sound risky.
What it means for you: staff in security, legal, medical or HR roles may hit refusals on legitimate questions. Test it with your own real prompts before rollout.
The risks businesses hit most often once a model is connected to real data and tools.
Whether instructions hidden in emails, web pages or documents can take over the model.
Not yet testedWhether it reveals its system prompt or documents it was given to read.
Not yet testedWhether it takes harmful actions when connected to email, files or other systems.
Not yet testedHow often it states false facts confidently on business-style questions.
Not yet testedThe standard edit, removing the refusal direction at every layer, turned Gemma 4 into gibberish, and so did removing a random direction, so we removed it only from what layers 28 to 34 of 60 write, using the direction from layer 34. Healing used QLoRA (rank 16, 3,000 Alpaca pairs, 24 steps). It taught the model Alpaca's short answer style, which cost about 14 points on IFEval. Wikitext perplexity is left out: this chat-tuned model does not score raw encyclopedia text in a way that compares across models. Refusal and capability runs on two NVIDIA RTX A4500 GPUs (20 GB each) with vLLM, weights in 8-bit (fp8). Refusal-direction search and WMDP in 4-bit. Healing in 4-bit QLoRA on the same GPUs. Ratings use the shared trifero rule set (method v1), the same for every model. We do not publish stripped weights.