Skip to content

Small models, measured in the open.

We train and post-train small open models and publish what each step changes, including what gets worse.

See the researchContact

OpenGrad · Qwen3.5-2B

Tool-call F1 rose. Bare maths questions went unanswered.

Tool-call F1

1,277 held-out tool-use prompts

  • Base0.63
  • Post-trained0.75

GSM8K solved

1,319 questions, asked without worked examples

  • Base67.4%
  • Post-trained0%

From scratch · 151M

GRPO taught the answer format, not the maths.

GSM8K answer format

200 questions, after SFT and after GRPO, with DPO between

  • After SFT5.5%
  • After GRPO98.5%

GSM8K solved

The same 200 questions

  • After SFT2 of 200
  • After GRPO3 of 200

In both, the number the training aimed at rose and the one beside it did not follow.

OpenGradStudy 001 published

What tool-use training cost a 2B model.

OpenGrad post-trains Qwen3.5-2B to decide when to call a tool, then measures what else changed. The checkpoint we promoted calls tools better and does worse on general benchmarks than the model it started from. Finding that cost $11.39 of GPU time.

General ability, before and after tool-use training

  • IFEval strict

    refuses 19%

    67.8% Base
    45.1% After SFT

    −22.7 pts

  • GSM8K 8-shot

    refuses 0%

    70.4% Base
    56.3% After SFT

    −14.1 pts

  • MMLU-Pro 5-shot

    refuses 0%

    49.0% Base
    37.0% After SFT

    −12.0 pts

Base Qwen3.5-2B After our tool-use SFT

Upstream benchmarks and scorers, one engine configuration for every stage. On the bottom two rows the model refuses nothing, so those drops are lost ability rather than refusals. Two gaps carry a caveat about the token limit. On IFEval, 47 base and 67 SFT answers of 541 hit it, and no interval was computed. The MMLU-Pro gap is a point estimate that lies between 6.4 and 33.1 points depending on how its truncated answers resolve. DPO on top of SFT changed every score by under one point.

Why the promotion gate missed it

What the 1,277 held-out examples asked the model to do

  • Call a tool453
  • Say it can't453
  • Ask to clarify371
  • Just answerNo examples0
With no examples that ask for a direct answer, the gate could not see a model that stopped answering. Study 002 sets out to explain the refusals before trying to fix them; it has no trained model yet.

Compressed for llama.cpp: nine sizes, two pass the main gate

Tool-use decisions unchanged from the uncompressed model on llama.cpp

  • Q2_K0.90 GiB53.4%
  • IQ3_M0.99 GiB77.1%
  • Q3_K_M1.02 GiB77.1%
  • IQ4_XS1.11 GiB88.3%
  • Q4_K_S1.13 GiB87.9%
  • Q4_K_M1.19 GiB87.2%
  • Q5_K_M1.31 GiB95.0%
  • Q6_K1.45 GiB98.0%
  • Q8_01.87 GiB98.7%

Passes the main gate

Each GGUF scored on the same 1,277 examples against gates fixed before any format existed. Q6_K is the smallest that passes the main gate; no format met the stricter release bar. llama.cpp is one of the engines in OpenWeights, the Experimental Machines app OpenGrad grew out of.

The training data

OpenGrad ToolPolicy Canonical v2

173,237 records from four public function-calling datasets, mapped to one schema. Each record keeps its source and licence, so attribution travels with the row. It carries no single licence: its sources use two.

Read Study 001The auditWeightsGGUF buildsDatasetOpenGrad Source

Records each source contributed.

Tool-choice contrastIn development, nothing measured yet

Same question, different tools.

A test set built for the gap OpenGrad found. Each question is asked three times, word for word, with different tools on offer. In one family the right response is always a direct answer; in the other it changes with the tools while the question does not.

Answer directly

One question, unchanged, asked with

  • No toolsAnswer
  • One irrelevant toolAnswer
  • Three to five irrelevant toolsAnswer

Decide about a tool

One question, unchanged, asked with

  • The right tool, every parameter knownCall it
  • The right tool, a parameter missingAsk
  • The right tool swapped for a wrong oneSay it can't
The correct response to each version. Planned: 900 items over 300 questions. Code under Apache-2.0, the questions under CC-BY-4.0. The rubric Tool-choice contrast Source

From scratchRun complete

A 151M model, trained from scratch.

Five stages, applied end to end to one small model on one H100: pretraining and midtraining on about 10B tokens, then SFT, DPO and GRPO, in 13.7 hours of training. The training code is not public yet.

  1. Pretrain7.26 h

    loss, lower is better

    9.992.99

    9.99 to 3.04

  2. Midtrain1.76 h

    loss, lower is better

    3.132.08

    3.13 to 2.29

  3. SFT3.78 h

    loss, lower is better

    1.781.27

    1.78 to 1.36

  4. DPO0.39 h

    preference accuracy

    0.840.44

    0.50 to 0.59

  5. GRPO0.49 h

    reward; a well-formed wrong answer scores 0.2

    0.280.01

    0.01 to 0.23

Step logs from the run, downsampled. Loss is cross-entropy.

Scroll the table sideways for every benchmark.

Benchmarks after each stage
AfterPerplexityHellaSwagARC-eMMLUGSM8K
Pretrain20.8538.5%45.8%24.0%1.0%
Midtrain19.7037.3%45.1%26.6%4.0%
SFT25.1137.4%48.1%23.8%1.0%
DPO25.0737.3%47.9%24.3%2.5%
GRPO25.1037.3%47.7%23.8%1.5%

The multiple-choice differences between stages are within noise: HellaSwag and MMLU were scored on 1,000 questions each, ARC-e on 570 and GSM8K on 200. Perplexity rises at SFT because it is measured on web text while the model is being trained for dialogue.

What changed was behaviour. Midtraining taught the GSM8K answer format, SFT lost it, and GRPO brought it back. After GRPO 3 of 200 answers were correct, against 2 after SFT, with DPO between.

Taglish sentimentResults on GitHub

Sentiment in Tagalog, Taglish and English.

A three-class sentiment model for Filipino reviews and posts, fine-tuned from XLM-RoBERTa. Its data is de-duplicated across splits, near-duplicates included, and an int8 ONNX build runs on a CPU.

Macro-F1 on the test set, by language

  • Tagalog0.813
  • Taglish0.817
  • English0.700
0.729 overall, against 0.632 for a TF-IDF baseline.

Median time to classify one text, Apple M5 CPU

  • PyTorch11.8 ms
  • ONNX int85.1 ms
The int8 build is 287 MB instead of 1.1 GB and loses 0.001 macro-F1.

Source and results

The people behind it.

Working on small models?

Tell us the model, what you trained it to do, and what you measured.