SLM-125M-e4
The 125.8M legal language model, continued for four epochs on a rebuilt 2.5B-token corpus — a controlled test of whether extra epochs keep improving a small model or start to overfit and forget. The finished result is below.
This model is the base of the 125M family — every fine-tune above starts from these weights. Each stage has its own site with its training details, cost and evaluation.
Text completion
The completion will appear here.
- temperature
- How random the output is. Low stays safe and predictable; high gets more varied — and more likely to wander.
- top-p
- Each step, sample only from the most likely words whose probabilities add up to p. Lower = more focused.
- top-k
- Each step, only consider the k most likely next words. Lower = more focused.
- max tokens
- How long the completion can get, in tokens (roughly ¾ of a word each).
What this is
base completerThis is a base model, not a chatbot. It only ever learned to predict the next token, so it continues text instead of answering questions — hand it the opening of a sentence and it carries the thought forward.
The honest quality number is held-out validation perplexity: 9.44 (lower is better), after 4 epochs on the rebuilt 2.5B-token corpus. That is better than the same model at 2 epochs (9.86), and every domain improved rather than regressing — extra in-distribution epochs behaved as pure improvement, not forgetting. It has a strong legal register — citation phrasing, procedural language — but at 125M parameters and a 1,024-token context it holds almost no world knowledge, so anything factual would need retrieval (RAG).
Corpus: SEC filings (~35%), US case law (~34%), educational web (~31%). The interactive demo below is not connected to a live server yet — the numbers and charts are the finished result.
Architecture
- Architecture
- LlamaForCausalLM (continued from base)
- Layers
- 12
- Hidden size
- 768
- Attention heads
- 12 (head dim 64)
- KV heads
- 12 — full MHA
- MLP
- SwiGLU, inner 3,072
- Positional
- RoPE, θ = 10,000
- Normalization
- RMSNorm, ε = 1e-5
- Embeddings
- tied input/output
- Tokenizer
- identical to base (16,384 BPE)
Training recipe
- Initialized from
- Ace-2504/slm-125m-e2 (weights only)
- Optimizer state
- reset — fresh schedule
- This run (e4)
- 9,440 steps = 2 epochs (→ 4 total)
- Tokens this run
- 4.95B (9,440 × 524,288)
- Tokens across lineage
- 12.45B (v1 2.04B + ext 0.52B + e2 4.95B + e4 4.95B)
- Learning rate
- cosine · peak 3e-4 → floor 3e-5
- Why cosine
- a fresh peak-then-decay each continuation cycle
- Global batch
- 524,288 tokens / step
- Windows per step
- 512 (micro-batch 32 × grad-accum 16)
- Optimizer
- AdamW β = (0.9, 0.95), wd 0.1
- Precision
- bfloat16 · SDPA attention
- Throughput
- ~113k tokens/sec on A100-40GB (Modal)
$70.14 across 27.5 h on A100-40GB
Every leg of the lineage, not just the last one — this model was pretrained from random weights, so nothing here is inherited free.
| Stage | Detail | Hours | Cost |
|---|---|---|---|
| v1 pretraining | 2.04B tokens, 1 epoch — from random weights | 5.10 | $11.54 |
| Extension | +515M new tokens the base had never seen | 1.29 | $2.88 |
| e2 | 2 epochs on the rebuilt 2.5B corpus | 12.53 | $27.47 |
| e4 (this model) | 2 further epochs — 4 in total | 8.60 | $28.25 |
| Total | 27.5 | $70.14 |
The fine-tunes built on this base carry this figure forward, so a 125M QA or RLAIF model costs this plus its own training — the base is not free when you trained it yourself.
What changed
forgettingOnce continued pretraining has already recovered from the initial disruption, do further epochs on the same in-distribution data keep improving a small model — or start to overfit and forget?
Further epochs behaved as pure improvement, not forgetting. Validation perplexity fell from 9.86 to 9.44, and every domain's held-out loss improved rather than regressing — the transient rise while the fresh learning rate peaked was fully recovered and surpassed.
| domain | share of corpus | ppl · 2 epochs | 4 epochs | change |
|---|---|---|---|---|
| SEC filings | 35% | 5.74 | 5.46 | -2.82% |
| US case law | 34% | 9.80 | 9.32 | -2.21% |
| Educational web | 31% | 23.53 | 22.17 | -1.88% |
Measured per domain on the base model's own held-out validation set. Neither checkpoint ever trained on it and both share the identical tokenizer, so this is a like-for-like comparison. Lower perplexity is better — every domain improved from 2 to 4 epochs, so the extra passes were pure gain on this in-distribution corpus, not catastrophic forgetting.
What it read
training corpusThe rebuilt 2.5B-token corpus the model trained on for all four epochs — a near-even three-way split of SEC filings, US case law and educational web, deduplicated and decontaminated, with a stricter OCR gate than the original run.
Corpus composition
- SEC filings34.7% · 860MPleIAs/SEC
- US case law34.2% · 847MHFforLegal/case-law
- Educational web31.0% · 768MHuggingFaceFW/fineweb-edu
Dedup + decontamination
The rebuilt 2.5B-token corpus after deduplication and 13-gram decontamination against the held-out evaluation sets.
| source | kept | near-dup | exact | contam. |
|---|---|---|---|---|
| case-law | 241,356 | 1643 | – | 28,274 |
| sec | 45,035 | – | 1989 | 175 |
| fineweb-edu | 690,923 | – | 77 | 0 |