We trained Elroy, our 360M-parameter code model, on worked solutions from four Qwen3.5 reasoning models to see whether a model this small can learn to reason by imitation.
Large reasoning models pass their skill to smaller ones when the smaller model is trained on their written reasoning, and distilled models from 1.5B parameters up have shown large gains. We asked whether the same works far below that size. Elroy is a 360M-parameter code model we trained from scratch, which means we know every token it has seen. We had Qwen3.5 0.8B, 2B, 4B and 9B solve 26,681 Python problems in thinking mode, kept only solutions that passed their unit tests, and fine-tuned Elroy three ways on the same verified code: code alone, code after a short explanation, and code after the teacher’s full reasoning. Code alone raised Elroy’s held-out accuracy from 32.9% to as much as 42.9%, 10 to 13 points above a matched control, but GPT-4o reference solutions to the same problems did just as well (42.4%). Adding a short explanation cut the gain by more than half. Adding the full reasoning erased it, and with less data it fell below the control. The size of the teacher made no difference to that pattern, and no training set raised HumanEval or MBPP above Elroy’s starting score. At this scale, Elroy copied the form of reasoning without its substance.
Held-out gain from training on the teachers’ verified code alone, against a control with the same training schedule. The three teachers land within about three points of each other.
Code from the 9B reasoning teacher against the dataset’s own GPT-4o reference solutions. The teacher’s value was correct code, which the dataset already had.
Effect of the full reasoning traces at 1,200 problems, against the same control. At 3,800 problems they were neutral. They never helped.
Problems that all four teachers solved with reasoning short enough for Elroy’s 2,048-token context. Each teacher stays short on different problems.
DeepSeek showed in early 2025 that a model as small as 1.5B parameters gains sharply on math when fine-tuned on the written reasoning of a much larger model. Li and colleagues then found that models of 3B parameters and below learn poorly from long reasoning by strong teachers, and better from short reasoning or from smaller teachers. Below 1B parameters there is little evidence either way.
Elroy, documented in our earlier report, is a 360M-parameter Llama-style model we wrote and pretrained ourselves on about 20B tokens of educational Python and English. Because we built its corpus and its instruction data, we know what it saw before this study, and its instruction data was already filtered against the benchmarks we use. We asked three questions: does reasoning from a teacher raise a model this small above what correct answers alone give; does a short explanation work better than full reasoning; and does a teacher closer to Elroy’s size teach better.
KodCode-V1 contains about 484,000 synthetic Python problems, each with a function signature, a reference solution and pytest tests; the solutions and tests were generated by GPT-4o and checked against each other. We kept instruction-style problems from five function-level subsets (Prefill, LeetCode, Algorithm, Data_Structure, Evol), rated easy or medium, with exactly one target function and a statement under 1,200 characters. We dropped 810 problems that KodCode flags as close to a public benchmark and 79 more that Elroy’s own filter matched to HumanEval or MBPP text or solutions. That left 26,681 problems for the teachers and 500 held out for evaluation.
Qwen3.5 0.8B, 2B, 4B and 9B share a tokenizer, a training recipe and a thinking mode, so they differ mainly in size. Each solved every problem once in thinking mode (temperature 0.6, top-p 0.95, top-k 20) with a 2,048-token limit, because longer reasoning cannot fit in Elroy’s context next to a problem and a solution. We ran every solution against its tests and kept only those that passed. Passing reasoning was screened again for benchmark text, which removed 4 to 15 traces per teacher.
Every kept problem became three training examples with the same verified function at the end:
So the formats differ only in how much reasoning comes before identical code. For comparisons between teachers we drew equal-size sets from each teacher’s own solved problems: 3,800 problems for the 2B, 4B and 9B, and 1,200 for all four teachers. A fifth condition used KodCode’s GPT-4o reference solutions for the 4B teacher’s problems, as code only.
Each run fine-tuned Elroy’s decontaminated chat model for two epochs at learning rate 5e-5, the settings of its original instruction tuning. Every set was mixed one to one with examples from Elroy’s original instruction data, which kept the narrow new data from overwriting what it already knew. The control is that replay data alone, at the same size, so every comparison below holds the training schedule fixed. Every condition ran with two seeds.
We converted Elroy to the standard Llama layout so vLLM could serve it, and checked the conversion against the original code: greedy decoding reproduced the published scores (HumanEval 22.0% against 22.6% published, HumanEval+ 21.3% against 21.3%, MBPP 35.4% against 35.0%). Results below use temperature 0.6 with four samples per problem, averaged over both seeds, on the 500 held-out KodCode problems, HumanEval and HumanEval+ (164 each) and MBPP (257). Intervals are 95% paired bootstrap intervals over problems.
Every teacher thought past the 2,048-token limit on most problems. The 4B was the most accurate when it finished; the 9B finished fewer problems than the 2B or 4B because it reasons longer.
| Teacher | Finished | Passed | Pass rate | Removed | Kept | Median tokens |
|---|---|---|---|---|---|---|
| Qwen3.5-0.8B | 3,697 | 1,284 | 35% | 4 | 1,275 | 824 |
| Qwen3.5-2B | 10,389 | 6,461 | 62% | 15 | 6,390 | 769 |
| Qwen3.5-4B | 9,588 | 7,004 | 73% | 14 | 6,981 | 1,044 |
| Qwen3.5-9B | 6,366 | 3,908 | 61% | 9 | 3,892 | 1,089 |
The teachers stayed short on different problems. Only 182 problems had usable reasoning from all four, so a shared problem set was not possible. Each teacher’s set reflects the problems it found easy enough to solve briefly, which is what any distillation run under a length limit would get.
The teacher closest to Elroy’s size did not teach better. With 1,200 problems, code from the 0.8B gave 35.3% against 36.8% from the 9B, and its full reasoning cost 4.2 points against the control, in line with the 4B and 9B.
| Training data | HumanEval | HumanEval+ | MBPP | KodCode held-out | KodCode vs control |
|---|---|---|---|---|---|
| Elroy before distillation | 18.9% | 16.8% | 29.5% | 32.9% | |
| Replay only (control) | 17.5% | 16.1% | 25.0% | 29.4% | |
| Code only | |||||
| Qwen3.5-2B teacher | 17.8% | 16.8% | 27.4% | 39.8% | +10.3 [+7.6, +13.0] |
| Qwen3.5-4B teacher | 18.1% | 17.4% | 26.7% | 40.1% | +10.6 [+7.8, +13.5] |
| Qwen3.5-9B teacher | 18.4% | 17.3% | 27.8% | 42.9% | +13.4 [+10.7, +16.2] |
| Short reasoning | |||||
| Qwen3.5-2B teacher | 16.8% | 15.9% | 27.9% | 32.5% | +3.1 [+0.6, +5.5] |
| Qwen3.5-4B teacher | 16.2% | 14.4% | 25.7% | 34.2% | +4.8 [+2.2, +7.3] |
| Qwen3.5-9B teacher | 16.0% | 14.0% | 25.0% | 34.4% | +5.0 [+2.5, +7.5] |
| Long reasoning | |||||
| Qwen3.5-2B teacher | 18.4% | 17.8% | 26.2% | 29.8% | +0.4 [−2.0, +2.8] |
| Qwen3.5-4B teacher | 17.5% | 16.5% | 25.0% | 27.4% | −2.0 [−4.3, +0.3] |
| Qwen3.5-9B teacher | 18.3% | 17.2% | 25.6% | 28.6% | −0.9 [−3.0, +1.3] |
| No teacher | |||||
| GPT-4o reference solutions, code only | 19.2% | 18.4% | 27.0% | 42.4% | +12.9 [+10.0, +16.0] |
| Training data | HumanEval | HumanEval+ | MBPP | KodCode held-out | KodCode vs control |
|---|---|---|---|---|---|
| Elroy before distillation | 18.9% | 16.8% | 29.5% | 32.9% | |
| Replay only (control) | 16.2% | 14.2% | 25.9% | 30.0% | |
| Code only | |||||
| Qwen3.5-0.8B teacher | 18.3% | 16.6% | 25.7% | 35.3% | +5.3 [+2.6, +8.0] |
| Qwen3.5-2B teacher | 18.4% | 16.6% | 26.8% | 36.6% | +6.6 [+4.1, +9.2] |
| Qwen3.5-4B teacher | 18.5% | 17.1% | 24.3% | 35.0% | +5.0 [+2.2, +7.8] |
| Qwen3.5-9B teacher | 17.9% | 16.8% | 27.5% | 36.8% | +6.8 [+4.2, +9.5] |
| Short reasoning | |||||
| Qwen3.5-0.8B teacher | 16.2% | 15.1% | 24.6% | 32.0% | +1.9 [−0.4, +4.3] |
| Qwen3.5-2B teacher | 17.5% | 15.7% | 24.9% | 30.9% | +0.9 [−1.5, +3.2] |
| Qwen3.5-4B teacher | 13.9% | 12.3% | 23.7% | 30.7% | +0.7 [−1.7, +3.1] |
| Qwen3.5-9B teacher | 16.4% | 15.4% | 23.8% | 31.0% | +1.0 [−1.4, +3.4] |
| Long reasoning | |||||
| Qwen3.5-0.8B teacher | 14.5% | 13.0% | 25.7% | 25.9% | −4.2 [−6.4, −1.9] |
| Qwen3.5-2B teacher | 16.5% | 15.5% | 27.0% | 29.0% | −1.0 [−3.1, +1.1] |
| Qwen3.5-4B teacher | 16.5% | 15.5% | 25.2% | 24.7% | −5.3 [−7.7, −3.0] |
| Qwen3.5-9B teacher | 17.1% | 15.5% | 26.7% | 25.4% | −4.6 [−6.9, −2.3] |
| No teacher | |||||
| GPT-4o reference solutions, code only | 18.8% | 17.0% | 26.8% | 36.4% | +6.4 [+3.8, +9.0] |
Our first run had no replay and used greedy decoding, the standard way to score small code models. Elroy trained on the 4B teacher’s full reasoning ran out of context without writing any code on 30% of HumanEval problems, usually by repeating one line:
Small distilled reasoners are known to loop under greedy decoding, which is why DeepSeek recommends sampling for theirs. With sampling at temperature 0.6, looping fell to 2% or less at 3,800 problems and at most 9% anywhere (the 0.8B teacher’s reasoning), and the results above use sampling throughout. Looping was not the main problem. When the reasoning finished, it read like the teacher’s but did not do the teacher’s work. Here Elroy, asked whether any two numbers in a list are closer than a threshold, reviews its plan, approves it, and writes a function that compares single numbers to the threshold:
The reasoning also cost length. Replies with full reasoning averaged 675 to 881 tokens on held-out problems against 80 to 92 for code alone, eight to ten times the compute per answer for lower accuracy.
| Format | HumanEval | MBPP | KodCode held-out | Never finished | Mean tokens |
|---|---|---|---|---|---|
| Code only | 18.3% | 28.4% | 42.0% | 1% | 194 |
| Short reasoning | 14.0% | 24.5% | 40.4% | 2% | 296 |
| Long reasoning | 14.0% | 19.1% | 28.0% | 30% | 1,060 |
Without replay, even code alone cost Elroy ground on HumanEval and MBPP while it gained on KodCode, which is why every result above mixes in its original instruction data.
HumanEval and MBPP ask for the same kind of short Python function as KodCode but were written by different people in a different style. No training set raised them above Elroy’s score before distillation. Against the replay-only control a few differences of about three points clear the interval, such as the 9B teacher’s code on MBPP, but the control itself lost ground to the extra training, so those gains only win it back.
| Training data | HumanEval | HumanEval+ | MBPP |
|---|---|---|---|
| Code only, 9B teacher | +1.0 [−1.6, +3.5] | +1.2 [−1.4, +3.9] | +2.8 [+0.1, +5.4] |
| GPT-4o reference solutions | +1.8 [−1.2, +4.8] | +2.3 [−0.7, +5.3] | +2.0 [−0.7, +4.8] |
| Short reasoning, 9B teacher | −1.4 [−3.9, +1.0] | −2.1 [−4.6, +0.5] | −0.0 [−2.5, +2.4] |
| Long reasoning, 9B teacher | +0.8 [−1.4, +3.0] | +1.1 [−0.8, +3.1] | +0.6 [−1.7, +2.8] |
Li and colleagues found that models of 3B parameters and below learn better from short reasoning than from long. At 360M we found the next step down: short reasoning still costs accuracy and long reasoning removes the whole benefit of the correct code it comes with. The teacher’s reasoning is not information Elroy can use. It is text to imitate, and imitating it takes the capacity and context the code needs.
Code from a 9B reasoning model and code from the dataset’s own GPT-4o solutions gave the same gain. For a model this small, a reasoning teacher is useful as a source of correct, tested answers, and only to the extent that such answers are not already available.
A student with a short context only ever sees the problems its teacher can solve briefly, and different teachers are brief on different problems. Of 26,681 problems, 182 had usable reasoning from all four teachers. Any distillation run into a small model is selecting its training problems this way, whether or not it reports it.
We publish the method, the generation, training and evaluation code, and all aggregate results. KodCode is licensed CC BY-NC 4.0, so models trained on it inherit a non-commercial restriction; we do not release the fine-tuned Elroy weights from this study. The original Elroy weights remain available as before.
Everything ran on 21 and 22 October 2026 on two NVIDIA RTX A4500 GPUs (20 GB each) in the trifero lab; training and evaluation were scheduled through Slurm. Teacher generation used vLLM 0.31.0 and took about 13.5 hours of wall time across both cards (the 9B with tensor parallelism). Elroy fine-tuning used Elroy’s own training code with bf16 autocast, batch 4 with 8-step accumulation; training took 2 to 12 minutes per run, and all 50 runs with their evaluations took under five hours of wall time on the two cards. The replay pool was Elroy’s original instruction set rebuilt with its decontamination filter (82,191 examples). Evaluation ran Elroy through vLLM after converting it to the Llama layout, with tests executed in sandboxed subprocesses. Per-sample outputs for every condition are stored in the study repository.