Short, dated observations from our research and engagements. Each one links to the full write-up.
On our claims-extraction task every LoRA rank from 4 up hit 100% exact match on familiar templates. On three unseen formats, rank 1 managed 85% and rank 16 97%. If you choose rank on an in-distribution validation set you will pick too low. The experiment.
Our compute model puts a 60M-token full fine-tune of a 7B model at 2.6 H100-hours at 40% MFU — about $5–$11. We corrected our own cost guide after running the numbers; earlier per-run figures were too high. The budget lines that matter are experiment count, RL sampling and evaluation. The model.
In a tabular DPO simulation with a known optimum, β=1 was supposed to keep the policy within KL 0.31 of the reference. With 256 preference pairs it drifted to 1.48. We now gate preference-tuning releases on measured divergence, not on β. Derivation and simulation.
For a 152k-token vocabulary at 4,096 tokens, fp32 logits take 4·s·V bytes ≈ 2.3 GB per sequence. For Qwen2.5-0.5B with activation checkpointing that is most of the 2.5 GB activation budget. Chunked cross-entropy fixes it. Memory model.
To detect +3 points at 85% accuracy with 80% power, an unpaired comparison needs about 2,046 items. When the two models share their hard items — typical for extraction tasks — a paired McNemar test needed 1,165. Score candidates on identical items and analyse the differences. Simulation.
A 45-minute briefing with a senior research engineer. You leave with a feasibility read, a data plan and a fixed quote, whether or not you hire us.
Book a briefing →