Elroy, from scratch

1The question

Vendors describe language models with adjectives. We wanted to describe one with measurements: what it costs to build a small model end to end, what the public training data actually contains, how capability appears during training, and how far a published benchmark number can be trusted, including our own. The model was scoped to one job, short Python programs and beginner explanations, so that every measurement could be checked against a clear standard.

The project was also built to be reproduced. Every number in this report comes from code and logs in a public repository, organized as nine chapters that a reader with one or two GPUs can follow.

2Method

Hardware, measured rather than assumed

All training ran on one machine: two RTX A4500 cards (20 GB each, Ampere, NVLink), an AMD Ryzen 9 7900, 32 GB of RAM and a 1 TB NVMe drive, under PyTorch 2.11 with CUDA 12.8. Plans were made from measured figures, which diverged from the specification in both directions.

The energy figure is estimated from rated draw at the 100% utilization both cards reported for the whole run; no meter was attached. At $0.27/kWh it is about $21.

Data

Four public sources, tokenized into 27.5B tokens of shards from which 20.0B were sampled by mix weight: Stack-Edu Python (45%, classifier-filtered educational code), StarCoderData Python (20%), FineWeb-Edu (25%) and Cosmopedia v2 (10%). Stack Overflow was excluded because its CC-BY-SA license would impose share-alike terms on the weights. The final 10% of training used an anneal mix of 60% Stack-Edu, 25% Cosmopedia and 15% FineWeb-Edu.

Two properties of the data were found by inspection. 48% of StarCoderData files begin with a <reponame>/<filename>/<gh_stars> metadata line that is not Python; left in, it would have taught the model to open every file with noise. After those lines were stripped, about 6% of code files still failed Python 3's parser, mostly Python 2, and were dropped.

Tokenizer

A byte-level BPE with 32,768 entries (256 bytes, 32,496 merges and 16 reserved special tokens), trained on an 800 MB sample in the training-mix proportions. Digits are split one per token and runs of whitespace are kept whole, so a Python indent is a single token. On held-out Python it encodes 3.69 bytes per token, against 2.16 for GPT-2's 50k vocabulary and 3.80 for GPT-4's 100k vocabulary: 41% fewer tokens than GPT-2 for the same code. On English it is 8% less efficient than GPT-2, the trade the plan asked for.

Model and training

A Llama-style decoder with RMSNorm, rotary position embeddings, SwiGLU feed-forward layers, no biases and tied embeddings: 24 layers, width 1,024, 16 heads, feed-forward width 3,072 and a 2,048-token context, 360.8M parameters. Next-token prediction, with half of all code documents rearranged into fill-in-the-middle form. AdamW (betas 0.9 and 0.95, weight decay 0.1), gradient clipping at 1.0, bf16 autocast, 1,000 warmup steps to a peak learning rate of 4e-4, cosine decay to 4e-5, then linear decay to zero over the final 10% as the mix switched to the anneal mix. Data-parallel across both GPUs with torch.compile, 38,146 optimizer steps of 524,288 tokens.

Instruction tuning

Two epochs at a learning rate of 5e-5 on one GPU (93 minutes), with the loss masked to the assistant reply. Data: Magicoder OSS-Instruct and evol-codealpaca-v1, filtered to replies whose code parses as Python 3 and fits in 2,048 tokens (36,964 and 47,002 examples), plus 45 handwritten examples repeated four times covering the model's identity, its limits and beginner concepts. 82,464 training examples and 25.2M supervised tokens.

Evaluation

• Benchmarks: HumanEval (164 problems), HumanEval+ (the same problems with roughly 80 times more tests) and the sanitized MBPP test split (257 problems), pass@1 with greedy decoding and one attempt.
• Sandbox: each completion runs in a subprocess with a 10-second timeout and a 4 GB memory limit.
• Chat format: each problem is posed as a user message and the first code block in the reply is extracted. MBPP prompts include the three test assertions, as in the original paper.
• Statistics: 95% intervals for single proportions are normal approximations. Checkpoints are compared problem by problem with exact McNemar tests, because every checkpoint answers the same problems.

3Pretraining

The run completed without a loss spike or divergence. After warmup the gradient norm exceeded the clipping threshold on 6 of 37,146 steps, with a median of 0.16. A restart at step 13,001 reproduced the pre-restart loss to four decimal places, because the data loader position and every random-number state are part of the checkpoint.

4What the base and chat models can do

The base model completes a function from its signature and docstring. Its competence has a sharp edge: all 22 of its HumanEval passes fall in the first 61 problems, which are roughly the easier ones. Asked a question in prose, it continues the text instead of answering it; asked to continue a sentence about Providence, Rhode Island, it called the city the largest in the United States and repeated variations of that claim for 200 tokens.

Instruction tuning taught three things: the chat template means a question is being asked, the reply should be an answer, and the end-of-turn token means stop. Held-out reply loss fell from 1.33 to 1.03 in the first 200 steps, which is the template, and to 0.94 by the end of the second epoch. Most of the benchmark gain is format: the base model's HumanEval failures include 15 completions that ran past the function and hit the token limit, and the chat model has 4.

What fine-tuning did not change is the shape of the failures. Most are code that runs, looks idiomatic and returns the wrong answer. The model's best-known example writes a correct is_prime and then, in the usage examples beneath it, states that is_prime(2) and is_prime(5) return False. The conversational layer is thin in the same way: 45 handwritten examples gave the model a stable identity, and three greeting examples against 82,000 programming questions left a lowercase "hi" with an empty reply.

5Was the chat score earned?

Two days after release, a reader pointed out that evol-codealpaca-v1 is the dataset Magicoder redistributes as a decontaminated version, and that the project had never checked its instruction data for benchmark overlap. The concern was specific. A doubling from 25M supervised tokens is where leaked solutions tend to surface, and the small HumanEval to HumanEval+ drop we had read as evidence of fully correct solutions is also what memorized canonical solutions would produce. Three outcomes were possible: the score was earned, the model was reciting answers, or the data was teaching the benchmark some other way. We separated them in three steps.

• Copy test: compare each chat-model pass with the canonical solution (difflib ratio on whitespace-normalized lines). A recited answer scores above 0.9.
• Data scan: search each instruction set for HumanEval problem text (the first 80 characters of each docstring, whitespace-normalized) and for canonical solutions (half or more of a solution's 13-token n-grams, comments and docstrings removed).
• Removal with a control: drop every match, retrain with nothing else changed, and retrain once more on the original data with only the seed changed, so the effect of the data can be told apart from the effect of a rerun.

The outputs are not copies

None of the chat model's 44 HumanEval passes reaches 0.9 similarity to its canonical solution. The highest is 0.73, on a problem whose canonical solution is one line, and most are below 0.3. By this test the model is not reciting answers.

The data contains the benchmark

The matches are benchmark problems in practice-exam form: reworded, extended with an extra requirement or ported to Java, with full solutions. HumanEval/43, pairs_sum_to_zero, appears with its original docstring, its doctests and a working solution under an added instruction to make it linear time. Evol-Instruct grows a dataset by repeatedly rewriting seed problems, and some of the seeds were benchmark problems. Switching to the filtered copy would not have fixed it.

Removing it

We wrote a filter that drops any instruction example containing a HumanEval or MBPP problem statement or half of a reference solution, using the same tests and covering all 427 sanitized MBPP problems rather than the test split alone. It removed 452 source examples touching 63 HumanEval and 64 MBPP problems; 271 of them had survived the existing Python and length filters, so the training split went from 82,464 to 82,198 examples. The decontaminated model was fine-tuned from the same base checkpoint with every other setting unchanged.

On HumanEval the two runs on the original data agree to within one problem (44 and 43 passes, 40 in common), while the decontaminated run is six to seven problems lower. Against the published run it lost 8 problems and gained 1 (p = 0.039); against the seed-2 run the difference is similar and not significant on its own (p = 0.11). On MBPP the picture reverses: the two original-data runs differ by 11 problems, and the decontaminated run sits on the lower one. Only 10 MBPP problems matched removed data, and the published 38.9% is best read as a favorable draw from run-to-run variance.

If the contamination worked by memorization, the lost passes would sit on the problems that appeared in the data. They do not. Against the mean of the two original-data runs, the decontaminated model is about four points lower on both subsets: 2.5 of 63 problems and 4 of 101. Practice on HumanEval-style questions helped on HumanEval-style questions in general.

We report the decontaminated checkpoint as the model's capability: 22.6% on HumanEval, 21.3% on HumanEval+ and 35.0% on MBPP, with run-to-run variance of up to about four points on MBPP.

6Errors found by reading

Five errors in this project would have produced wrong published numbers or a failed run. None raised an exception that stopped anything; each was found by reading a log, a sample or a failure record instead of a summary.

• Sandbox memory. HumanEval+ first scored 3.0% for the base model. 14 of the 22 solutions that pass HumanEval had died with MemoryError, because the expanded test files exceed a 1 GB subprocess limit. At 4 GB the score is 11.0%.
• MBPP prompt format. The chat model first scored 2.5% on the first 40 MBPP problems. The task text does not name the function, so the model chose its own name and every hidden test failed with a NameError. Showing the test assertions, as the MBPP paper does, gave 38.9% for that run.
• Milestone checkpoints. Permanent checkpoints every 5B tokens never fired, because the boundary was checked on steps that never wrote a checkpoint. Found at step 12,864 when the file was missing.
• Resume memory. The restart to fix that bug failed after one step with an out-of-memory error, because the checkpoint was loaded onto the GPU beside the live model. It was invisible at the 42M-parameter scale the code had been tested at.
• Benchmark contamination. Section 5. Found by an outside reader, which is the best argument we have for publishing every completion and every log.

7Discussion

A workstation is enough for 2021

With current open data and published architecture choices, a model at the 2021 frontier for its size trains on hardware a small firm can buy, in under a week, for about $21 of electricity. The data work dominated both the effort and the outcome: the run was uneventful because the corpus had been read before it was tokenized.

A benchmark score has four inputs

A published score is a property of a model, its training data, its evaluation harness and a random seed, and a single run reports all four as one number. In this project the harness moved one score from 3.0% to 11.0% and another from 2.5% to 38.9%, the training data moved HumanEval by about four points without any detectable copying, and the seed moved MBPP by four points. None of this is visible in a headline figure. Any model provider, including us, should be asked for the decontamination method, the harness and the number of runs behind a score.

Decontamination by name is not decontamination

The redistribution offered as the clean option removed 89 rows and kept 52 of 60 contaminated HumanEval problems. Exact-match filters miss rewritten problems, and rewriting is how Evol-Instruct data is made. A filter that also matches solution n-grams and problem-text prefixes is cheap; ours runs in minutes and is in the repository.

The failure that matters in practice

The model's typical error is code that runs, looks right and is wrong, delivered after fine-tuning in a confident, helpful register. Larger models make this error less often, not differently, and it becomes harder to notice.

8Limitations

• Few runs. Each fine-tuning condition rests on one or two runs. The direction of the HumanEval effect holds against both controls; its size is uncertain by a few points.
• Filter recall. The filter uses docstring prefixes and solution n-grams and will miss paraphrases that share neither, so the decontaminated score is an upper bound on clean performance.
• Pretraining data was not scanned. Solutions to many HumanEval problems have since been published on GitHub, and some will be in StarCoderData and Stack-Edu.
• One pretraining run at one scale. Validation loss was still falling at 20B tokens; the run stopped for time, not convergence.
• Energy is estimated from rated power, not metered.
• Scope. Evaluation covers Python function synthesis only. No English benchmarks, no multi-turn evaluation, and the model was not trained to refuse anything.

9Reproducibility

Pretraining ran from 14 to 20 September 2026 and fine-tuning, decontamination and the seed control from 20 to 22 September, on the two RTX A4500 cards described above with PyTorch 2.11, CUDA 12.8 and Python 3.12. The code, the nine-chapter write-up, every benchmark completion and failure record and the training logs are at github.com/strifero/elroy under Apache-2.0, including the contamination check (scripts/check_contamination.py) and the filter (elroy/decontam.py). The base and chat weights are at huggingface.co/strifero/elroy-350m-base and huggingface.co/strifero/elroy-350m-chat. The chat weights released there are the original run that scored 26.8%; the decontaminated checkpoint this report measures is not yet published.

We thank the reader whose review prompted section 5. They identified the problem, proposed the first test, and were right.

10References

1. Chen, M., et al. Evaluating large language models trained on code. 2021. arxiv.org/abs/2107.03374
2. Austin, J., et al. Program synthesis with large language models. 2021. arxiv.org/abs/2108.07732
3. Liu, J., Xia, C. S., Wang, Y., and Zhang, L. Is your code generated by ChatGPT really correct? Rigorous evaluation of large language models for code generation. NeurIPS 2023. arxiv.org/abs/2305.01210
4. Wei, Y., Wang, Z., Liu, J., Ding, Y., and Zhang, L. Magicoder: empowering code generation with OSS-Instruct. 2023. arxiv.org/abs/2312.02120
5. Luo, Z., et al. WizardCoder: empowering code large language models with Evol-Instruct. 2023. arxiv.org/abs/2306.08568
6. Allal, L. B., et al. SmolLM2: when smol goes big, data-centric training of a small language model. 2025. arxiv.org/abs/2502.02737
7. Li, R., et al. StarCoder: may the source be with you! 2023. arxiv.org/abs/2305.06161
8. Penedo, G., et al. The FineWeb datasets: decanting the web for the finest text data at scale. 2024. arxiv.org/abs/2406.17557
9. Touvron, H., et al. LLaMA: open and efficient foundation language models. 2023. arxiv.org/abs/2302.13971
10. Su, J., et al. RoFormer: enhanced transformer with rotary position embedding. 2021. arxiv.org/abs/2104.09864
11. Shazeer, N. GLU variants improve transformer. 2020. arxiv.org/abs/2002.05202
12. Zhang, B., and Sennrich, R. Root mean square layer normalization. NeurIPS 2019. arxiv.org/abs/1910.07467
13. Bavarian, M., et al. Efficient training of language models to fill in the middle. 2022. arxiv.org/abs/2207.14255
14. Sennrich, R., Haddow, B., and Birch, A. Neural machine translation of rare words with subword units. ACL 2016. arxiv.org/abs/1508.07909