Trutina 4B technical report

Abstract

Trutina 4B is a typed decision model built on Qwen3.5-4B. Given a state (text, optionally with images) and a set of yes/no, choice or score questions, it returns a probability for every option and generates no text. A rank-16 LoRA is trained on every linear layer, and the answer is read from the LM-head logits of the option letters at a fixed position. Layers 12, 16, 20 and 24 each carry an affine exit trained by distillation; at inference a request can stop at the first sufficiently confident exit and otherwise continues from where it stopped. On the 231 tasks of the JevBench public set the model reaches 87.4% at full depth, and 86.6% in adaptive mode using 21.6 layers on average. On the zero-shot LocalLLaMA typed-decisions test set it reaches 74.65% accuracy with a KL of 0.156. On one RTX PRO 6000 the adaptive mode has a median latency of 14.0 ms. Among six open 4B decision models compared under one protocol, Trutina has the highest accuracy on LocalLLaMA, this-that and DecisionBench medium; on DecisionBench hard it scores below two of them and below the untrained base model.

The report also records negative results from development. The most consequential was that the decision head first trained on hidden states reached only 74.0% on the JevBench public set, below the 79.7% obtained by reading option letters from the untrained base model.

1. Introduction

A decision model receives a state and a set of typed questions and returns a probability for each option. Compared with a general chat model that writes its answer as text, it needs one forward pass, its outputs can be thresholded directly, and it costs less in latency and compute. The goal of this project was the best result within the 4B parameter class; outperforming larger models was not a goal.

The report is organised as follows:

• Section 3 describes the original decision-head design, its training implementation in MLX, and the audit that led us to abandon it.
• Section 4 describes the readout: option letters read from the LM head, the question presented before and after the input (hereafter the sandwich), a tournament for more than 26 options, and several training ablations.
• Section 5 describes the trainable intermediate exits and adaptive inference.
• Sections 6 to 8 describe the training data, the training procedure and probability calibration.
• Section 9 gives the full evaluation, section 10 reports the methods that did not work, section 11 describes deployment, and section 12 discusses limitations.

The work took five days from the first survey to the v3 release:

Version	Day	Readout	Training data
v0	Day 1	newly trained decision head, Base backbone	34k questions
v1	Day 2	same, backbone truncated at layer 24	135k questions (with images)
v2	Day 4	option letters, trained exits	261k questions
v2.1	Day 4	same, with vision	164k questions (120k replay)
v3	Day 5	same	120k questions (62k replay)

Accuracy of each version on the two main evaluations (%)

JevBench public set (231 tasks)

v174.0

untrained79.7

v286.6

v2.187.0

v387.4

LocalLLaMA (2,000 questions, zero-shot)

v064.2

v168.7

untrained64.1

v274.25

v2.174.7

v374.65

2. Task and evaluation protocol

A request contains a state and questions of the following types:

• yes/no (noul): the probability of yes;
• choice: 2 to 255 options, one probability each;
• score: a distribution over 2 to 10 ordered levels, and the expected level.

Development relied mainly on JevBench v1.5 [1] and LocalLLaMA typed-decisions, which score models in noticeably different ways.

JevBench is a community leaderboard. Its downloadable public set has 231 tasks in easy (48), standard (72) and hard (111, of which 17 form the judge tier) tiers. Its Intelligence score weights the public and sealed sets equally; the judge and hard tiers together carry 70% of the weight; a yes/no answer with P(yes) between 0.2 and 0.8 counts as an abstention and is scored as wrong; score questions are graded by the nMAE of the distribution’s expected position; and choice questions use chance-corrected accuracy, with ECE [2] and the TVD to the gold distribution as calibration terms. Self-hosted latencies enter the Speed axis after a ×2 + 0.15 s adjustment, so 10 ms and 30 ms are almost indistinguishable there.

LocalLLaMA typed-decisions [3] contains 400 cases and 2,000 questions (800 score, 600 choice, 600 yes/no) from four business workflows: agent-trace monitoring, customer support, invoices and security incidents. Its gold labels are the average of three samples at temperature 0.7 from a teacher model of about 4B parameters, so they are soft. The dataset card gives reference points: 0.470 for answering by label frequency alone, 0.704 for a perfect recovery of the generating latent factors, and 0.735 for the teacher’s agreement with itself. The mean top gold probability is only 0.659, and 738 questions have a top gold probability of at least 0.7. The leaderboard reports both accuracy and KL. Our training data contains none of these workflows, which makes this a zero-shot evaluation.

DecisionBench [4], this-that [5], JudgeBench [6], fast-decisions [7], MASSIVE [8] and Decision Index 0.3 [9] were used for evaluation only and never for training (section 9).

3. Early approach: a decision head on hidden states

3.1 Probing the backbone

We first measured the latency and per-layer representations of Qwen3.5-4B-Base [10] on an M5 Pro (48 GB, MLX [11]). The model has 32 layers, alternating Gated DeltaNet [12] (linear attention) and full attention at 3:1, with 3.57B non-embedding parameters.

Latency was strictly proportional to the number of layers run: a 200-token input took 89.7 ms through all 32 layers and 45.6 ms through half of them, about 0.85 ms per layer. Short inputs are bound by memory bandwidth and longer ones by compute; 4-bit and 8-bit quantization made prefill 15–25% slower.

Linear probes on each layer (frozen features, logistic regression) on ARC-Challenge [13] and MMLU [14] gave the following results:

Linear-probe accuracy on the layers of frozen Qwen3.5-4B-Base

• ARC-Challenge (1,172 test questions)
• MMLU (1,500 test questions)

The probes suggested that the middle layers suffice and that the last layers degrade slightly, and a linear head on hidden states scored almost 30 points above scoring each option separately. These two observations were the basis for v0.

3.2 Decision-head design

v0 followed the pointer head of Strands Decider: the LM head is removed and a new decision head is trained on the backbone’s hidden states. Specifically:

• the backbone is frozen in bf16, with LoRA [15] (rank 16, scale 2) on every linear layer of the first 32 layers;
• exits sit at layers 16, 24 and 32, each mixing several layers with a ScalarMix;
• the exits share one permutation-equivariant joint decision head (2 layers, 512 dimensions) in which options attend to one another;
• 38.2M parameters are trainable;
• the loss is soft-target cross-entropy (label smoothing 0.05) plus 0.5 times the Brier score, with RPS added on score questions and equal weights for the three exits.

3.3 Training Gated DeltaNet in MLX

Qwen3.5’s Gated DeltaNet layers cannot be trained in MLX as shipped. The Metal kernel used for inference has no backward pass, and the per-token fallback used in training mode stores a [B, 32, 128, 128] fp32 state for every token, about 16 GB for a single layer at B=8 and T=1000.

Following the WY/UT transform [16] used in fla [17], we implemented a chunked gated delta rule. Each chunk of 64 tokens is computed with matrix multiplications only, and a single state is carried between chunks; memory falls by a factor of about 64 and the computation can be differentiated directly.

The first version inverted the in-chunk triangular system with the product formula

$$(I+A)^{-1} = \prod_{m=0}^{M-1}\bigl(I + (-A)^{2^m}\bigr)$$

with $M = 6$ for the block length of 64. When neighbouring keys are highly correlated, its terms grow like binomial coefficients: real data produced NaN at the first step, while tests on random data did not expose the problem. A divide-and-conquer inverse that merges blocks level by level has the same numerical stability as forward substitution.

Batches are formed by total token count (at most 8,192 padded tokens per micro-batch), and long samples are kept. Truncating questions at 1,024 tokens would remove mostly long-context tasks (36% of needle retrieval), while only 5.7% of validation questions exceed 1,024 tokens, so the loss would be almost invisible in the metrics.

3.4 v0 results

v0 used 19.5k rows (34k questions) and a learning rate of 2e-4 (100 warm-up steps, cosine decay), and was stopped at step 3,020 on the M5 Pro laptop after about 15 hours. The results at that point were sufficient to assess the approach, and later training moved to the cloud.

v0 validation accuracy (716 questions)

• exit at layer 16
• exit at layer 24
• exit at layer 32

The three exits were within one point of each other. On LocalLLaMA, v0 at layer 32 reached 64.2% with a KL of 0.292, against 41.2% and KL 0.760 for the untrained backbone reading LM label probabilities. Score questions were the weakest type (55.7%, against 73.9% for the leaderboard’s best system at the time), with yes/no at 76.0% and choice at 63.8%. On unambiguous questions v0 reached 80.9%, so most errors came from contested items. The gap between exits was larger on LocalLLaMA than on the validation split: layer 16 trailed layer 32 by 3.2 points.

3.5 v1: data, training features and the move to the cloud

To address v0’s weaknesses (no business-workflow states, mostly hard labels, weak score questions), v1 grew the text data to about 100k questions, 81% of them with soft labels from teacher models (the mean target entropy of 0.38 is the floor of the cross-entropy), and added about 35k image questions. v1 also introduced the following training features:

• each choice question gets a fixed random permutation of its options at encoding time;
• 10% of questions lose their option descriptions and keep only option names;
• the decision head cross-attends to state tokens to read evidence back;
• score questions use an ordinal (cumulative logit) parameterisation;
• 16 source-conditioning tokens are added during training, replaced by a null token at inference;
• shallow exits also fit the deepest exit’s distribution (weight 0.5, stop-gradient on the deepest exit);
• label smoothing applies to hard labels only, so teachers’ soft distributions are no longer pushed towards uniform.

Following v0’s results, the backbone was truncated at layer 24, with exits at layers 12, 16 and 24.

v0 ran at about 330 tok/s on the M5 Pro, so one v1 epoch would have taken 29 hours, and v1 was trained with PyTorch on Modal B200s, with training code that corresponds to the MLX implementation parameter by parameter. After fixing throughput problems such as repeated kernel compilation and LoRA dtype conversions, a B200 sustained about 18k tok/s.

v1 trained for 8,450 steps on a B200 (one epoch, 16 questions per step) at a learning rate of 2e-4:

v1 validation accuracy (3,933 questions, including images)

• exit at layer 12
• exit at layer 16
• exit at layer 24

v1 reached 68.7% on LocalLLaMA (layer 24) with a KL of 0.192. Its layer-12 exit (65.0%) already exceeded v0 at full 32-layer depth (64.2%), and layers 16 and 24 were almost identical (68.6% and 68.7%). The gap between validation and LocalLLaMA accuracy was 16 points, as for v0, so the added business-workflow data did not noticeably narrow the out-of-distribution gap.

On a B200, the median latency of a short single question was 14.6 ms when stopping at layer 12 and 28.0 ms through all 24 layers, and barely depended on input length: the bottleneck was the overhead of launching kernels layer by layer rather than compute.

3.6 Audit: the readout

v1 reached only 74.0% on the JevBench public set, and 50.5% on the hard tier. The gap was concentrated in a few hard families:

v1 accuracy on JevBench hard-tier families (%)

long_policy (19 tasks, 2,804 tokens on average)

v126

SemIf-4B53

Jev 1.13.063

multi_hop (18 tasks)

v156

SemIf-4B67

Jev 1.13.083

judge_hard (17 tasks)

v153

SemIf-4B76

Jev 1.13.076

temporal_numeric (15 tasks)

v120

SemIf-4B20

Jev 1.13.027

probability (10 tasks)

v140

SemIf-4B40

Jev 1.13.070

In the chart above, SemIf-4B is the 4B model of the SemIf project [18]. Other decision models built on Qwen3.5-4B scored between 84% and 90% on the JevBench public set. We audited each design decision of v0 and v1 and located the main problem in the readout. Used as released, with all options shown at once and the next-token probabilities of the option letters read, the post-trained Qwen3.5-4B reaches 79.7% without any training (81.4% in an external report, 81.0% in our reproduction on a GPU), 5.7 points above v1.

The zero-shot baseline in the probe experiment of section 3.1 scored each option separately as Answer: <option>, so the model never saw the options together, and the baseline was correspondingly low. When all options are shown together, the LM head is a strong prior in its own right. The decision head discarded that prior and had to relearn answering from over a hundred thousand questions, and the loss was largest outside the training distribution, on JevBench hard. Secondary causes were the use of the Base rather than the post-trained backbone, the removal of the last 8 layers, and the absence of long-document and answer-judging questions from the training data.

4. Readout

4.1 Zero-shot readout experiments

The v2 readout was chosen from zero-shot experiments (no training, MLX on the M5 Pro) covering every combination of two backbones, three orderings and two label styles, evaluated on the JevBench public set, LocalLLaMA and 577 questions sampled from the v1 validation split:

Zero-shot readout: accuracy by backbone, ordering and label style (%)

JevBench public set

Base · state first · letters73.2

Base · state first · numbers70.1

Base · question first · letters68.4

Base · question first · numbers68.8

Base · sandwich · letters77.1

Base · sandwich · numbers75.3

post-trained · state first · letters76.2

post-trained · state first · numbers76.6

post-trained · question first · letters76.6

post-trained · question first · numbers75.8

post-trained · sandwich · letters79.7

post-trained · sandwich · numbers79.2

v1 (trained) 74.0

LocalLLaMA

Base · state first · letters47.8

Base · state first · numbers50.4

Base · question first · letters52.4

Base · question first · numbers51.9

Base · sandwich · letters59.9

Base · sandwich · numbers59.8

post-trained · state first · letters54.7

post-trained · state first · numbers50.5

post-trained · question first · letters61.5

post-trained · question first · numbers55.9

post-trained · sandwich · letters64.1

post-trained · sandwich · numbers63.4

v1 (trained) 68.7

In the sandwich format the question and options appear once before and once after the input. It raised JevBench by 3 to 4 points on both backbones and the v1 validation accuracy of Base by 7 points, the largest gain from any single change. Of Qwen3.5’s 32 layers, 24 use linear attention; reading a state without knowing the question, its compressed memory can only retain a little of everything. With the question first, the model knows which information to retain while it reads the state, and the repetition at the end places the options next to the answer position. This agrees with the findings of Just Read Twice [19] and Prompt Repetition [20]; the only cost is one additional copy of the question. When several questions share a state, the leading block contains all questions and the trailing copy only the current one.

The post-trained backbone is more accurate than Base (with the sandwich, +2.6 on JevBench and +4.2 on LocalLLaMA) but also more confident, and its KL on both soft-label sets is clearly worse (0.73 against 0.47 on LocalLLaMA), which training has to correct. Letters and numbers differ by less than one point, in no consistent direction, and we kept letters.

The final format uses the chat template with thinking disabled; options are labelled (A) to (Z); yes/no questions are rendered as two options whose text includes any descriptions given in the question; score levels are labelled in order; and the prompt ends with Best answer: (, where the softmax over the option letters’ logits is taken. The questions of one request share the state and are computed as one batch.

4.2 Zero-shot tests of output transforms

Before training we also tested output transforms that need no training, fitted on LocalLLaMA and the v1 validation split and scored on the JevBench public set under an approximation of the v1.5 rules (I_open is only meaningful as a difference):

Transform	I_open	choice CC	yes/no CC	score CC	ECE
raw output	54.9	70.3	14.4	64.3	.084
decisive yes/no map ($T = 0.23$, $b = 0.4$)	62.1	70.3	43.2	64.3	.112
choice temperature ($T = 2.8$)	54.9	70.3	14.4	64.3	.112
score temperature ($T = 1.4$)	54.9	70.3	14.4	64.4	.075

By argmax, 79.7% of yes/no questions were correct, but 20% fell in the abstention band between 0.2 and 0.8, and the chance-corrected yes/no score was only 14.4. The decisive map pushes probabilities with a clear lean out of that band and raised I_open by 7.2 without any training (section 8.2). The choice temperature is a counterexample: LocalLLaMA’s soft labels are spread out, the fitted $T = 2.8$ made ECE worse when transferred to JevBench, and temperatures therefore have to be fitted on held-out data close to the target distribution.

4.3 Tournament readout for more than 26 options

The API allows up to 255 options (in Decision Index, CLINC150 [21] has 151 and BANKING77 [22] has 77), while the model has only seen A to Z in training. Comparable models either use single-token two-letter labels such as AA and AB after Z and train on that format, or train a 255-way linear head. We compared zero-shot methods on MASSIVE (59 intents, 2,974 questions), which is not part of the training data:

Readouts for more than 26 options on MASSIVE (top-1 accuracy, %)

two-letter labels (AA, AB, …)59.4 top-5 88.3

two-round tournament79.3 top-5 95.2

reference: answer + 25 random distractors86.4

when the answer comes after AA

two-letter labels41.2

two-round tournament80.0

The tournament divides the options, in their original order, into balanced groups of at most 26 and scores each group; the top ⌊26 / number of groups⌋ options of each group go to a final that is scored once more. A finalist’s probability is M times its final probability, where M is the finalists’ mean probability mass within their groups; the remaining options share 1 − M according to their group probabilities. The model sees every option, only in different forward passes.

4.4 Small-scale training ablations

The method we originally expected to matter most was a prior plus a residual (PRR): the LM head’s letter logits serve as the prior, and v1’s joint head (option interaction, cross-attention to the state, ordinal score structure) as a zero-initialised residual. The other candidates were an anchor KL [23], which keeps the output distribution at a few non-answer positions close to the base model (top-64 truncated and precomputed), and a permutation-consistency regulariser, which adds a symmetric KL between 25% of choice questions and a second permutation of them in the same batch. Each configuration was trained for 2,000 steps on 1/8 of the v2 data (32k questions) at a learning rate of 1e-4, with all 32 layers and the same data and seed. All configurations were identical at step 0 (67.7% on v2 validation), confirming that the residual head started from zero.

Small-scale ablations: accuracy after 2,000 steps (%)

v2 validation

letters83.1

letters + anchor81.6

PRR83.1

PRR + anchor81.1

PRR + anchor + consistency82.0

untrained 67.7

LocalLLaMA

letters73.7

letters + anchor71.8

PRR73.8

PRR + anchor72.5

PRR + anchor + consistency71.8

untrained 63.9

JevBench public set

letters81.4

letters + anchor81.4

PRR80.1

PRR + anchor83.5

PRR + anchor + consistency81.4

untrained 80.5

A per-question paired bootstrap (95% intervals) gives:

• letters + anchor against letters: JevBench +0.0 [−3.5, +3.5], LocalLLaMA −1.9 [−3.5, −0.4], validation −1.5 [−2.3, −0.7], the last two significantly worse;
• PRR + anchor against letters + anchor: JevBench +2.2 [−0.4, +4.8], not significant;
• PRR against letters (both without anchor): indistinguishable on all three.

Two points on 231 tasks correspond to about five tasks. By question type, PRR was slightly better on score questions (+1.6 on v2 validation, +2.0 on LocalLLaMA) and slightly worse on yes/no (−0.7, −3.4), partly in line with the expectation that the residual head helps ordinal structure, but without a net gain. v2 therefore uses the simplest configuration.

In these experiments JevBench accuracy held or rose with training, and the soft-label KL fell from 0.73 to 0.14. v1, which started from a randomly initialised decision head, fell to 74.0 on JevBench after training; no such decline was observed when training started from the post-trained backbone and its LM-head prior. Each configuration cost about $4.80.

5. Variable-depth inference

5.1 Formation of the answer across layers

Passing the untrained backbone’s residual stream at each layer through the final norm and the LM head and reading the option letters (a logit lens [24]) shows the layers at which the answer forms. The experiment ran on an L4, 2,808 questions in 8 minutes; the last layer deviates from the model’s own logits by at most 0.06 (bf16).

LocalLLaMA accuracy when the option letters are read at each layer

• untrained backbone (logit lens)
• Trutina's trained exits

Up to layer 19 the readout is at chance. The answer forms between layers 20 and 24, and at layer 24 it is as accurate as at the last layer (JevBench 80.1 against 80.5, v1 validation 78.3 against 78.3). The remaining 8 layers mainly concentrate probability mass on the letter tokens:

Probability mass on the option letters at each layer of the untrained backbone (JevBench public set, %)

• mass on letter tokens

Reading the LM head directly therefore cannot answer at layers 12 or 16. The accuracy of v1’s exits at layers 12 and 16 came from the trained head rather than from the LM head’s prior, and shallow exits must be trained for the purpose.

5.2 Exit design

Each exit is an affine map on the residual stream, $\mathrm{norm}(h + Wh + b)$, in the form of a tuned lens [25], with its own logit scale and about 6.5M parameters, 26M for all four exits. The exits read the same LM-head letter rows as the final layer.

The exits are distilled [26] towards the final layer’s answer distribution. Their inputs are detached and their gradients are clipped separately, so they learn to predict the final answer early without affecting the backbone, and full-depth accuracy is the same as without exits.

At inference the backbone runs in segments (layers 1–12, 13–16 and so on) and keeps the hidden states of all tokens at each segment boundary. A request that does not stop at layer 12 continues from layer 13; the first 12 layers are computed once, and each question costs one additional 2560×2560 matrix multiplication at the exit.

Each exit has its own temperature, so that a given confidence has the same meaning at every exit and a single stopping threshold applies to all of them. Exit temperatures are close to 1 in every version:

Exit layer	12	16	20	24	32
v2	0.78	0.89	0.98	0.95	0.95
v2.1	0.83	0.95	1.04	1.01	1.01
v3	0.83	0.95	1.04	1.01	1.04

A request stops when the top probability of every question in it reaches the threshold [27, 28]. The threshold is chosen on the validation split as the lowest value that costs at most about 0.5 points of accuracy: 0.86 for v2, 0.84 for v2.1 and 0.86 for v3.

Accuracy if the answer is read at each layer

• our validation split
• LocalLLaMA

Where requests actually stop

L124%

L1627%

L2033%

L2411%

L3226%

Layers used

21.6/ 32

about a third of the compute saved

Accuracy

86.6%

87.4% with all 32 layers

Median latency

14.0ms

19.0 ms with all 32 layers

Mean latency

30.6ms

38.7 ms with all 32 layers

Exit layer	12	16	20	24	32	adaptive
our validation split	47.9	73.0	87.2	88.5	89.4	89.1
LocalLLaMA	42.5	60.9	72.2	73.9	74.65	74.65

On the v3 validation split, adaptive mode stops at layers 12, 16, 20, 24 and 32 for 4%, 27%, 33%, 11% and 26% of questions respectively, 22.1 layers on average. On the JevBench public set it uses 21.6 layers on average with 86.6% accuracy, against 87.4% at full depth. On LocalLLaMA it uses about 28 layers with unchanged accuracy.

v1 showed that the stopping rule depends on the exit accuracy curve. With three exits and a top-probability threshold, most uncertain questions went to layer 24, although layers 16 and 24 were almost equally accurate:

v1 policy	validation	LocalLLaMA
always layer 24	0.840 @ 24 layers	0.685 @ 24 layers
three exits, threshold 0.9	0.838 @ 19.4 layers	0.685 @ 23.2 layers
layer 12 then 16, threshold 0.9	0.836 @ 14.7 layers	0.686 @ 15.8 layers

Choosing only between layers 12 and 16 gives the same accuracy with about 60% of the compute. The curve has to be measured again for each version: by v2, the exits at layers 20 and 24 were clearly better than layer 16.

5.3 Deep mode

Deep mode was originally designed as latent reasoning [29, 30, 31]: a few latent slots iterate over layers 16 to 24 without generating text (section 10.1). This design did not meet its predefined criterion. The replacement averages over 4 cyclic permutations of the options [32]. On v1 this raised accuracy on low-confidence questions by about one point. After v2’s training, the difference between deep mode and a single full-depth readout was +0.1 [−0.9, +1.1] on 800 choice questions from the v2 validation split and +0.7 [−2.2, +4.3] on the 139 choice tasks of the JevBench public set; the position bias [33, 34] had largely disappeared. Deep mode is now equivalent to full-depth computation. The MLX engine still averages over option rotations, but we do not consider it more accurate.

5.4 GPU latency

For a single short question on a GPU, latency barely depends on input length; the bottleneck is the overhead of launching kernels layer by layer. v3 has a p50 of 12.7 ms when stopping at layer 12 and 29.8 ms through all 32 layers, about 0.86 ms per layer. JevK5 v0.3 uses the same backbone at full depth and reaches 20.0 ms with CUDA graphs.

We capture one CUDA graph [35] per exit segment and length bucket (plus one for the embedding), precompute masks and rotary embeddings per bucket before capture, and copy to the host once per exit. Different capture limits were compared in one container:

RTX PRO 6000, serial HTTP	full depth p50 / mean / p95 (ms)	adaptive p50 / mean / p95 (ms)
eager	29.4 / 44.5 / 110.6	20.8 / 35.3 / 108.9
graphs, captured up to 1,024 tokens	19.7 / 39.6 / 113.5	14.9 / 31.7 / 111.6
graphs, captured up to 2,048 tokens (default)	19.0 / 38.7 / 110.8	14.0 / 30.6 / 110.2
graphs, captured up to 4,096 tokens	18.4 / 38.9 / 123.9	13.8 / 30.6 / 114.7

The 4,096-token run was in a slower container, so its absolute values are high. With and without CUDA graphs, the selected exit agrees on 99.1% to 100% of the 231 tasks and accuracy is identical. Long inputs are compute-bound and gain little from CUDA graphs.

6. Training data

6.1 Data rules

The training data contains no outputs of the Jev models [36], no JevBench tasks and no evaluation sets. From v2 on, the data is deduplicated by 13-grams against the LocalLLaMA test set, the JevBench public set and our validation data; the benchmarks included in Decision Index are excluded at the source level; and data from the same business workflows as LocalLLaMA is excluded to keep that evaluation zero-shot. Validation splits are grouped by document, so one contract, rule or state appears on one side only. No language model was called to write questions or produce labels for the v2 data.

6.2 Sources of soft labels

In v1, 81% of labels came from teacher models. v2 uses the following sources, none of which requires model calls:

Source	Method	Examples
the dataset’s own distribution	used as given	teacher probabilities shipped with public decision datasets
human votes	vote shares with smoothing: (n_k + 0.5) / (n + 0.5K)	ChaosNLI [37] (about 100 votes per item), Measuring Hate Speech [38], SBIC [39], MT-Bench [40] human judgements
exactly computed distributions	used as given	probability questions, analytic posteriors of synthetic ticket triage
structured smoothing	decay with level distance: p(ℓ) ∝ exp(−β·|ℓ − gold|), with β giving gold about 0.7	score questions with hard labels only
hard labels	label smoothing 0.05, applied to hard labels only	all remaining questions

The loss is the cross-entropy against the soft target plus 0.5 times the Brier score; score questions add the ranked probability score (RPS), so that an error of one level costs less than an error of several.

The final v2 label mix is 116k gold hard labels, 104k from sources carried over from v1 (mostly soft), 17k dataset distributions, 22k human-vote distributions, 5,600 structured-smoothing targets and 1,300 exact distributions. The share of soft labels therefore fell from 81% in v1 to about 40%. We expected this to risk a worse LocalLLaMA KL; in the event KL fell from 0.192 to 0.144.

6.3 v2 data

v1’s weaknesses on JevBench were concentrated in long-document rule reasoning (long_policy 26%), answer judging (judge_hard 53%, with no such questions in the training data), and multi-hop and temporal-numeric reasoning. v2 adds data for these areas, 266k questions in total (261k training, 5,336 validation), of which 142k are choice, 90k yes/no and 34k score questions. The data falls into the following groups:

• v1’s existing data, downsampled to about 60k questions, since v1 was already close to perfect on the easy and standard tiers.
• Public decision-format datasets, including long-document, multi-hop, probability and trade-off questions with program-solved answers organised like the JevBench hard families, and long-document questions verified by two independent solutions. Each synthetic source is capped (at most 30k questions from one generator) so that the model does not learn templates.
• Long-document rules: CUAD [41] (whether a contract contains a given clause type, in 12,000-character windows, about 21% positive, discarding clauses that straddle a window boundary), MAUD [42] (with answers seen in the dataset as options), ShARC [43] (four options: yes, no, irrelevant, more information needed) and the rule-application tasks of LegalBench [44]. The maximum state length was raised from 6,000 to 24,000 characters (about 6k tokens).
• Answer judging: PRM800K [45], HelpSteer [46], MT-Bench human judgements (A better, B better or tie, with the experts’ vote distribution as soft label), SummEval [47] (experts’ mean consistency and relevance scores split between the two neighbouring levels) and HaluEval [48] (correct answers against constructed hallucinated ones).
• Multi-hop and numeric reasoning: MuSiQue [49], 2WikiMultihopQA [50], HotpotQA [51] (distractor setting), DROP [52] (numeric answers only) and TAT-QA [53] (arithmetic and counting questions).
• Yes/no questions on natural text: BoolQ [54], StrategyQA [55], and ROPES [56] questions whose two options are stated in the question.
• Human-vote soft labels: Measuring Hate Speech (at least 3 annotators per comment), SBIC (offensiveness, intent and sexual content) and ChaosNLI (4,645 SNLI [57], MNLI [58] and αNLI [59] items, split by premise).

Distractors for the multi-hop and numeric choice questions share the answer’s type: numbers with numbers, dates with dates, text with text. They are drawn preferably from intermediate-hop answers, entities and other numbers in the same passage, and otherwise from answers of the same type in other questions. 35% of these questions are rewritten as yes/no questions asking whether a given answer is correct, half with the correct answer and half with a distractor.

Several label errors were found during construction:

• HotpotQA titles contain HTML escapes ("Dr. Death" Steve Williams) and did not match the answer before decoding, so they were selected as distractors;
• text answers appeared alongside their aliases (“Yadkinville” and “Yadkinville, North Carolina”);
• an early version paired numeric questions with place-name distractors.

Before the fix, 8.1% of MuSiQue, 5.5% of HotpotQA and 1.4% of 2WikiMultihopQA choice questions had two correct options. Candidates are now HTML-decoded and compared in canonical form (case and punctuation removed for text, numeric comparison for numbers), text answers contained in one another are excluded, and the duplicate rate is 0 for all five sources. Manual inspection of 200 converted items per source found more problems than automated checks did.

Long documents make up a much larger share in v2: about 19k questions (9%) have at least 4,000 characters and about 10k at least 8,000.

6.4 v2.1 data

The v2 readout prompt accepted text only, and about 35k image questions were dropped at encoding because the backbone had been loaded without its vision tower. v2.1 restores visual input: the vision tower is frozen, image tokens are expanded on the patch grid into the state, and positions use multimodal RoPE; a comparison with the HF processor confirmed identical tokens. The image questions are:

Source	Questions	Construction
rendered state screenshots	17,060	v1 text states rendered as screenshots; the validation part comes from v1 validation states and measures the modality gap
Mind2Web [60]	10,000	numbered boxes around the correct element and 2–4 distractors in the same viewport (Set-of-Mark [61]); each item asks which element and which action
A-OKVQA [62]	6,000	soft label = number of human answers matching the option + 2 × the reference answer
ScienceQA [63]	4,000	questions with images only

v2’s weakest hard family was temporal_numeric (6/15), while the same family in our own validation data was already at 88%, which pointed to a difference in question style rather than a lack of data. v2.1 therefore adds temporal reasoning in other styles: Test of Time [64] date and time arithmetic (1,390 questions, with near-miss numeric distractors, a name list with one person added or removed, or two neighbouring items swapped), Test of Time reasoning over anonymised temporal facts (2,465 questions), and TempReason [65] L2 (5,023 questions). 35% of these are also rewritten as judgement questions.

6.5 v3 data

Performance on the hard tier remained insufficient through v2.1. Plumb-4B [66]‘s training set contains about 5,000 hard questions verified by two independent solutions, and its hard-tier results exceed those obtained with our much larger general mix. For v3 we therefore selected datasets that are not in decision format but whose answers can be recomputed by a program, and converted them algorithmically:

Source	Questions kept	Conversion
FinQA [67]	5,263	execute the dataset’s calculation program and check the result against the human answer (allowing for percentage units and annotation precision); the input contains only the original report and table
HiTab [68]	968	execute safe arithmetic and aggregation formulas over the referenced cells and keep only questions that are not direct lookups; merged headers are expanded
WikiSQL [69]	11,835	execute column selection, aggregation and conditions with parameter binding rather than interpreting the raw SQL; keep aggregate or multi-condition numeric questions
ProofWriter [70]	20,348	depths 3 and 5, re-solved with an independent forward-chaining solver, keeping True, False and Unknown distinct; at most 3 questions per document
BoardgameQA [71]	13,000	depths 2 and 3, labels from the publisher’s generator cross-checked against each proof’s final conclusion
a public community decision dataset	29,211	keep its published blend of gold and teacher distributions, reorder probabilities to the option order, and remove the 466 rows where the two conflict; families the publisher held out stay held out

Proofs, formulas, programs and gold retrievals are never included in the input. Distractors for numeric questions share the answer’s unit and magnitude and are constructed by substituting the operation, applying the wrong aggregation and taking neighbouring values. The first version drew distractors from random table cells, which often had different units and were easy to rule out, so a separate version with strong distractors was built.

Before the main run we ran a controlled comparison. Both arms started from v2.1 with a fresh optimizer and trained for 2,000 steps at a learning rate of 1e-5 with description dropping disabled. The control arm used 24k old text questions and 6k image questions; the treatment arm used 15.6k old text questions, about 8.4k new questions and the same 6k image questions (52% old text, 28% new, 20% images), with True, False and Unknown at 450 : 450 : 400 among the logic questions. Weights were selected on the common validation loss only, with step 0 among the candidates.

Controlled comparison: treatment arm against control arm, paired differences (points)

• 95% interval excludes 0
• 95% interval includes 0

held-out algorithmic data, strong distractors68.3 → 85.7 · 1,509 questions+17.4[+14.6, +20.3]

held-out algorithmic data, original69.9 → 88.8 · 1,508 questions+18.9[+16.2, +21.6]

held-out community task families91.5 → 94.6 · 1,400 questions+3.1[+1.5, +5.0]

original text validation87.8 → 87.9 · 5,578 questions+0.1[−0.3, +0.5]

LocalLLaMA74.8 → 74.3 · 2,000 questions−0.5[−1.3, +0.4]

−50+5+10+15+20+25

In 2,000 steps the control arm found no weights better than its starting point, so continued training on old data alone brought no gain. The new data produced large gains on its own distribution, and no measurable decline was observed on existing abilities. Per new source in the treatment arm: BoardgameQA 54.0 to 76.8, ProofWriter 60.8 to 93.0, HiTab 83.3 to 88.9, WikiSQL 87.5 to 89.2, FinQA unchanged at 87.1.

The data for the v3 main run consists of 62.4k old text questions, 33.6k new text questions and 24k image questions, about 120k in total.

7. Training

7.1 Training configuration

	v2	v2.1	v3
initial weights	post-trained Qwen3.5-4B	v2	v2.1
LoRA	rank 16, all linear layers (attention, Gated DeltaNet projections, MLP)	same	same
learning rate	1e-4, 200 warm-up steps, cosine decay to 0	2e-5, 100 warm-up steps	1e-5, 100 warm-up steps
steps × questions per step	16,250 × 16 (one epoch)	7,000 × 16, step 4,000 selected	7,500 × 16
maximum length	6,144 tokens	same	same
B200 time and throughput	4.6 h, about 14.7k tok/s	about 2.5 h	about 3 h

Options are permuted at random during training, and 10% of questions have their option descriptions removed (disabled in v3, so that lettered or numbered options keep their text). Exit inputs are detached and exit gradients clipped separately. Weights are selected by validation loss only; LocalLLaMA and JevBench are used solely to monitor drift. Every run evaluates at step 0, so problems in the evaluation pipeline surface within minutes.

7.2 Training v2

v2 validation accuracy at each exit (5,336 questions)

• layer 12
• layer 16
• layer 20
• layer 24
• layer 32 (final readout)

Drift monitors during v2 training (%)

• JevBench public set (231 tasks)
• LocalLLaMA (2,000 questions)

JevBench showed no downward trend during training, and the anchor was not needed. In the last 2,000 steps the learning rate was close to zero and the final checkpoints were almost identical, so checkpoint averaging would not have changed the result and was not used. v2 met all its predefined replacement criteria: 86.6 on the JevBench public set (at least 85), 72.3 on hard (at least 70), 74.25 on LocalLLaMA (at least 68.0) and KL 0.144 (at most 0.25).

7.3 v2.1: forgetting during continued training

The first v2.1 run (learning rate 5e-5, 60k questions of old-data replay, about 23% of all training questions) was stopped at step 2,000. The new data was learnt quickly, with Mind2Web rising from 75% to 91% and Test of Time arithmetic from 70% to 81%, but v2 validation accuracy fell by 1.7 points and LocalLLaMA by 4.2 points, still falling while the learning rate was at its peak. v2’s learning rate had annealed to zero; raising it again to 5e-5 perturbed the existing weights too strongly, and the replay share was too low.

The second run lowered the learning rate to 2e-5 and raised replay to 120k questions (about 74%), and no further forgetting occurred:

LocalLLaMA accuracy during the two v2.1 runs (%)

• first run: learning rate 5e-5, ~23% replay
• second run: learning rate 2e-5, ~74% replay

After step 4,000 the learning rate was below 20% of its peak. Between steps 4,000 and 7,000 the paired differences were −0.2 [−0.9, +0.4] on LocalLLaMA and 0.0 [−1.3, +1.3] on JevBench, so the last 3,000 steps changed almost nothing. Changes from v2 to v2.1:

Image validation accuracy before and after v2.1's vision training (%)

Mind2Web

v2 (zero-shot)75.0

v2.191.9

A-OKVQA

v2 (zero-shot)87.4

v2.189.6

ScienceQA

v2 (zero-shot)91.6

v2.196.2

rendered state screenshots

v2 (zero-shot)84.6

v2.184.9

Text results were flat within noise: the v2 part of validation 87.39 to 87.62, LocalLLaMA 74.25 to 74.75 (paired +0.5 [−0.5, +1.6]), JevBench 86.6 to 87.0 (+0.4 [−2.6, +3.5]). In-distribution Test of Time arithmetic rose from 70 to 72 (43 questions) and JevBench temporal_numeric from 6/15 to 7/15, so the weakness in temporal arithmetic remained. TempReason (94.5%) and Test of Time fact reasoning (97.3%) were already high before training and are easy for the model.

7.4 The v3 main run

v3 starts from v2.1 and trains for 7,500 steps on the treatment arm’s data recipe. Weights are selected by the mean NLL on the full validation split (10,209 questions), with step 0 among the candidates; step 7,500 was selected, with validation NLL 0.4011, accuracy 89.43%, ECE 0.031 and KL 0.226.

v3 validation accuracy at each exit (10,209 questions)

• layer 12
• layer 16
• layer 20
• layer 24
• layer 32 (final readout)

In a post-training comparison on the same items, v3 improves on v2.1 by 3.7 points [3.1, 4.3] on the full validation split and by 14.1 points [12.0, 16.4] on the held-out new-source data (3,083 questions), changes by −0.3 [−0.7, +0.2] on the original text validation, and goes from 74.70 to 74.65 on LocalLLaMA.

Held-out new-source data: v2.1 and v3 accuracy (%)

BoardgameQA (380 questions)

v2.153.9

v387.4

ProofWriter (556 questions)

v2.160.6

v394.1

FinQA (159 questions)

v2.181.1

v391.2

WikiSQL (387 questions)

v2.187.1

v394.8

HiTab (27 questions)

v2.177.8

v392.6

community decision dataset (1,574 questions)

v2.191.3

v395.9

8. Probability calibration

8.1 Soft-label probabilities and correctness

After v1, ECE indicated that the model was conservative: at step 3,500 the layer-24 exit had a mean confidence of 0.690 at 81.7% accuracy. We exported per-question probabilities with MLX on the M5 Pro and fitted temperatures to two targets:

v1, layer 24	ECE (validation)	ECE (LocalLLaMA)	KL (LocalLLaMA)
uncalibrated	0.071	0.061	0.191
fitted to soft labels, T ≈ 1.03	~0.07	~0.06	0.188
fitted to argmax hard labels, $T = 0.698$	0.029	0.037	0.262

The two calibrations correspond to probabilities with different meanings. Probabilities trained on soft labels describe the distribution of annotators or teachers, while probabilities calibrated on hard labels describe how likely the top option is to be correct. ECE treats the argmax of the soft label as the correct answer, so the closer a model is to the teacher distribution, the more conservative it appears under ECE. The API returns the first kind of probability, and each exit’s temperature is fitted to soft targets; all are close to 1 (section 5.2). Adaptive exiting only compares the top probability with a threshold, and since temperature is monotone, adjusting the threshold suffices without a separate temperature.

8.2 A decisive map for yes/no questions

JevBench v1.5 scores a yes/no answer with P(yes) between 0.2 and 0.8 as an abstention. A model trained on soft labels tends to produce intermediate probabilities on contested yes/no questions and therefore loses points under this rule (section 4.2).

We fit a monotone map on the unambiguous yes/no questions of our validation split (for v2 on 1,671 questions, $T = 0.71$; for v3 a temperature of 0.777 with a margin of 0.2). It moves probabilities with a clear lean out of the abstention band and leaves genuinely uncertain answers in the middle. For v3 at full depth it raises Intelligence from 66.9 to 70.1. Because the map changes the meaning of the returned value, it is an option: the server applies it by default, --noul calibrated disables it, and the soft yes/no labels used in training are unchanged.

The server applies the map after the final temperature, so the map must also be fitted on post-temperature probabilities. Probabilities near the threshold are sensitive to numerical differences: batched evaluation with the unmerged LoRA gave an Intelligence of 67.8 while the served model gave 66.9, because one task’s P(yes) moved from 0.808 to 0.798. This report uses the served figure.

9. Evaluation

Unless stated otherwise, all results are for the released v3 weights. No evaluation set was used for training or for weight selection.

9.1 JevBench public set

	All (231)	Hard (94)	Intelligence (default)	Intelligence (no decisive map)
v3, full depth	87.4	75.5	70.1	66.9
v3, adaptive	86.6		68.3	65.1
v2.1, full depth	87.0	73.4	69.0	65.8
v2.1, adaptive	86.1		67.9	64.7
untrained backbone, same prompt	79.7

On the same 231 tasks with the same scoring, JevK5 [72] v0.3 has an Intelligence of 70.7, Plumb-4B 67.4 (run without its own yes/no commit server), spark-s1 [73] 59.8, decider-4b [74] 59.3 and Sieve-4B [75] 48.0. All of these are our own scoring of the public tasks, not official full-open-set or sealed-set results. Shallow mode (layer 12 only) has an Intelligence close to zero on JevBench and is unsuitable for such tasks.

By hard-tier family, v2 scored adversarial 6/6, trap 8/8, routing_hard 5/5, multi_hop 14/18, ambiguous 5/7, probability 7/10, long_policy 13/19, tradeoff 4/6 and temporal_numeric 6/15. v2.1 and v3 both score 7/15 on temporal_numeric.

9.2 LocalLLaMA typed-decisions (zero-shot)

Figures for the other models come from the dataset card [3]. d1 [76] is Liquid AI’s open decision model.

LocalLLaMA typed-decisions zero-shot accuracy (%)

meraGPT Decider 1 (hosted API)76.80 KL 0.096

Trutina 4B, full depth74.65 KL 0.156

Trutina 4B, adaptive74.65 KL 0.196

Liquid AI d174.20 KL 0.475

TypeSafe Jev 1.13.072.70 KL 1.442

Featherless Simple Jev (35B-A3B)71.60 KL 0.488

prima-ratio + 12B70.20 KL 0.564

OpenDecider-small (open weights)67.10 KL 0.211

perfect recovery of the generating latent factors, 70.4

9.3 Open 4B decision models under one protocol

Each system ran on its authors’ server at a fixed revision, each on its own RTX PRO 6000. One client sent identical requests serially and one scorer graded them, with rejected requests counted as errors. The Trutina bars show v2.1 through its server (full depth, no option rotations); v3 results are in section 9.4.

Six open 4B decision models under one protocol (accuracy, %)

JevBench public set (231 tasks)

Trutina87.0

JevK5 v0.389.2

Plumb-4B90.0

decider-4b83.1

Sieve-4B78.4

spark-s181.0

LocalLLaMA

Trutina74.7

JevK5 v0.362.5

Plumb-4B60.8

decider-4b67.7

Sieve-4B74.1

spark-s166.9

DecisionBench medium (state average)

Trutina82.5

JevK5 v0.380.4

Plumb-4B82.0

decider-4b74.4

Sieve-4B72.8

spark-s138.6

DecisionBench hard (state average)

Trutina49.7

JevK5 v0.358.4

Plumb-4B65.0

decider-4b45.3

Sieve-4B46.6

spark-s130.9

this-that

Trutina51.3

JevK5 v0.347.8

Plumb-4B48.4

decider-4b47.8

Sieve-4B45.8

spark-s143.6

fast-decisions

Trutina61.6

JevK5 v0.362.4

Plumb-4B61.8

decider-4b61.9

Sieve-4B62.9

spark-s157.4

JudgeBench (both orders correct)

Trutina51.3

JevK5 v0.352.1

Plumb-4B50.3

decider-4b54.0

Sieve-4B53.1

spark-s150.6

MASSIVE (59 intents)

Trutina79.8

JevK5 v0.376.9

Plumb-4B74.1

decider-4b88.8

Sieve-4B84.9

spark-s10.0 all rejected

9.4 Paired comparison of v3 and v2.1

Both models were scored on the same items with the same code (6,917 external items, with the tournament for the 132 questions with more than 26 options); confidence intervals come from a bootstrap clustered by document or task family:

v3 against v2.1, paired differences (points)

• 95% interval excludes 0
• 95% interval includes 0

JudgeBench51.5 → 54.5 · 620 pairs+3.1[+1.1, +5.2]

DecisionBench hard50.0 → 52.5 · 80 states+2.5[−1.6, +7.2]

JevBench hard73.4 → 75.5 · 94 tasks+2.1[−3.2, +7.4]

validation over 1,024 tokens89.1 → 90.1 · 1,480 questions+0.9[+0.1, +1.9]

image validation87.0 → 87.5 · 1,924 questions+0.5[−0.3, +1.3]

JevBench public set87.0 → 87.4 · 231 tasks+0.4[−1.7, +2.6]

DecisionBench medium83.1 → 83.3 · 80 states+0.2[−1.6, +1.9]

this-that51.2 → 51.3 · 1,710 questions+0.1[−1.2, +1.5]

LocalLLaMA74.7 → 74.65 · 2,000 questions−0.1[−1.0, +0.9]

MASSIVE (served over HTTP)79.8 → 79.7 · 2,974 questions−0.1[−0.7, +0.5]

fast-decisions61.3 → 61.1 · 1,600 questions−0.3[−1.0, +0.5]

original text validation87.8 → 87.5 · 5,578 questions−0.3[−0.7, +0.2]

validation over 4,096 tokens96.1 → 92.2 · 51 questions−3.9[−10.2, 0.0]

−10−50+5

The gains on JudgeBench and on long inputs are statistically supported; the gain on DecisionBench hard is not yet confirmed, and JevBench temporal_numeric remains at 7/15. Calibration worsened on some external sets: this-that NLL rose from 1.347 to 1.463 and ECE from 0.223 to 0.267, JudgeBench NLL from 0.713 to 0.756 and ECE from 0.160 to 0.180, and LocalLLaMA KL from 0.149 to 0.156. Our evaluation had recommended keeping v2.1 as the main version; v3 was nevertheless released as the default, and v2.1 remains available under the v2.1 tag on Hugging Face.

9.5 Other external evaluations

Decision evaluations outside the training distribution: untrained backbone, v2.1 and v3 (%)

DecisionBench medium

untrained75.2

v2.183.1

v383.3

DecisionBench hard

untrained53.6

v2.150.0

v352.5

this-that

untrained43.8

v2.151.2

v351.3

fast-decisions

untrained61.5

v2.161.3

v361.1

JudgeBench

untrained46.0

v2.151.5

v354.5

Gains outside the training distribution range from 7 to 9 points. Accuracy on fast-decisions is unchanged while its ECE falls from 0.25 to 0.14; within it, the agent_handoff domain fell by 15 points (100 questions). DecisionBench hard remains below the untrained backbone.

9.6 An approximate comparison with 27B models

Perplexity published per-task accuracy for Jev 1.13.0 [36], Qwen3.8-27B [77] and its own 27B decision model [78] on 11 tasks. Before running these tasks ourselves, we scanned the public items of each task for 8-gram overlap with all of our training data (an item is a near-duplicate when at least max(3, 30%) of its 8-grams appear in a single training file):

Task	Items	Near-duplicates (v2 / v2.1 / v3)
Circa [79]	34,268	71.5% / 52.5% / 45.8%
RAGTruth [80]	2,700	2.9% / 1.1% / 0.9%
WinoGrande [81]	3,034	1.1% / 0.6% / 0.3%
FinancialPhraseBank [82]	4,846	0.7% / 0.3% / 0.2%
ContractNLI	2,969	0.5% / 0.2% / 0.7%
TruthfulQA [83]	817	0.5% / 0.1% / 0.1%
TabFact, BBH, Belebele		0

Circa was excluded, and the near-duplicate items (255 in all) were removed from the remaining tasks, leaving 500 to 1,500 items per task. Perplexity’s sampled items, prompts and label mappings are unpublished, so the comparison is approximate.

An approximate comparison with 27B models (accuracy, %)

FinancialPhraseBank

Trutina 4B82.6

Jev 1.13.077.0

Qwen3.8-27B75.7

Perplexity Decider 27B84.2

RAGTruth

Trutina 4B79.3

Jev 1.13.077.3

Qwen3.8-27B61.5

Perplexity Decider 27B88.8

JevBench public hard

Trutina 4B73.0

Jev 1.13.073.3

Qwen3.8-27B72.3

Perplexity Decider 27B70.3

TabFact

Trutina 4B84.8

Jev 1.13.089.8

Qwen3.8-27B78.6

Perplexity Decider 27B90.6

ContractNLI

Trutina 4B84.7

Jev 1.13.077.5

Qwen3.8-27B80.8

Perplexity Decider 27B80.8

Belebele

Trutina 4B94.6

Jev 1.13.095.0

Qwen3.8-27B93.2

Perplexity Decider 27B94.0

WinoGrande

Trutina 4B72.6

Jev 1.13.090.7

Qwen3.8-27B73.1

Perplexity Decider 27B83.3

JudgeBench

Trutina 4B67.1

Jev 1.13.078.6

Qwen3.8-27B68.9

Perplexity Decider 27B78.3

BBH

Trutina 4B66.1

Jev 1.13.094.3

Qwen3.8-27B72.8

Perplexity Decider 27B82.8

TruthfulQA (binary)

Trutina 4B77.8

Jev 1.13.092.0

Qwen3.8-27B82.8

Perplexity Decider 27B85.4

9.7 Decision Index 0.3 and image tasks

The Decision Index 0.3 text public suite score is estimated at about 44 (±3). A full run would take 6 to 8 hours on L4 GPUs, so it was stopped partway and replaced by stratified samples scored with the official scorer, with the samples split by run_id into halves to estimate sampling variation (46.5 and 42.5). The sample has about 12k rows, at least 48 per benchmark. On the same 2,400 rows, v3 scores 44.4 and v2.1 44.7.

Domain	knowledge & reasoning	language understanding	retrieval & classification	tools & automation	artistic taste
v2.1	0.288	0.484	0.453	0.649	0.382
v3	0.291	0.495	0.451	0.636	0.344

The model is strong on tool calling and intent classification (BFCL [84] 90.6, CLINC150 84.0, BANKING77 73.8, API-Bank [85] 81.5, HellaSwag [86] 93.8) and scores close to zero on tasks requiring multi-step reasoning or specialist knowledge, such as HLE, chess, PhishNChips, CRUXEval, Habermas, ACOS and SATA-Bench.

Image tasks (our validation split, in distribution): Mind2Web 93.0, A-OKVQA 90.2, ScienceQA 98.5, rendered workflow screenshots 84.9. Of the six open 4B decision models in the same-protocol comparison, the other five load text-only checkpoints or flatten the state to text on the server, and accept no image input.

9.8 Latency

4B decision models · measured side by side

1. Trutina · adaptiveMultimodal4B14.0 ms
2. decider-4b4B17.8 ms · 1.3×
3. Plumb-4B4B18.6 ms · 1.3×
4. JevK5 v0.34B20.0 ms · 1.4×
5. Sieve-4B4B63.1 ms · 4.5×
6. spark-s14B109.5 ms · 7.8×

12–35B decision systems · JevBench official

1. Cygnet12B39.8 ms · 2.8×
2. JevOne · Qwen3.6-35B-A3BMultimodal35B58.5 ms · 4.2×
3. NInfer Qwen3.8-Flash-Nextn/a79.8 ms · 5.7×
4. AutoJev-27B27B88.1 ms · 6.3×
5. Surogate Rune 26B-A4B26B100.5 ms · 7.2×
6. swanOne · Qwen3.8-Flash-Nextn/a208.8 ms · 14.9×

Our request set is the 231 tasks of the JevBench public set, whose long-document tasks set the p95 (about 110 ms). Means are 30.6 ms adaptive and 38.7 ms at full depth. In adaptive mode, the p50 (eager) by stopping layer is 14.0 ms at layer 12 (3 tasks), 16.3 ms at 16 (62), 19.6 ms at 20 (102), 48.5 ms at 24 (16) and 46.2 ms at 32 (48). Questions with more than 26 options run the tournament and take about 1.2 s at p50 and 3.0 s at p95, currently the most significant latency limitation.

Dividing Intelligence by latency gives an efficiency measure. For v3 in adaptive mode with the default yes/no map (Intelligence 68.3) and CUDA graphs, Intelligence per mean latency is 2.23, per p50 4.88 and per p95 0.62; in the same container JevK5 v0.3 scores 1.59, 3.54 and 0.50, and Plumb-4B 1.51, 3.62 and 0.46. JevBench defines no official efficiency measure, so this comparison is indicative only.

9.9 Contamination

Data up to v2 was deduplicated by 13-grams against LocalLLaMA, the JevBench public set and our evaluation sets. The sources added in v3 were split from the validation data by document but were not scanned for n-gram overlap with external benchmarks (the 8-gram scan in section 9.6 was run after v3’s training and covers only those 11 tasks). During development, LocalLLaMA and the JevBench public set were used to monitor drift, and JevBench was used to choose the prompt format (section 4), so neither is a fully blind evaluation. Plumb-4B’s published training questions are included in our v2 data, which should be taken into account when comparing with Plumb-4B.

10. Negative results

10.1 Latent reasoning

The latent deep mode was trained on top of the frozen v1. Eight latent slots follow the prompt and iterate over layers 16 to 24 for R rounds. The first round’s input is the answer position’s state at layer 16 plus a learnt per-slot offset scaled by that state’s RMS; each later round adds a recurrent transform of the previous round’s output. A deep-specific LoRA acts only at slot positions, while the prompt uses the frozen v1 weights. The readout attends from the answer-position features to the slots and scores each option bilinearly, giving a correction added to v1’s layer-24 logits; the correction is initialised at zero, so training starts exactly at v1.

On the M5 Pro, R=4 added about 70 ms to the base forward pass (about 17 ms per round).

The training set was mined from v1’s training questions: from a random 40k, the 10,626 that v1 answered wrongly or with a top probability below 0.7 were kept, plus 25% confidently correct questions. The predefined criterion was to clearly outperform the option-permutation ensemble on questions where v1 is uncertain. Results on 1,500 validation questions (“uncertain” meaning a v1 top probability below 0.7):

	accuracy, all / uncertain	KL, all / uncertain
v1, layer 24	0.850 / 0.605	0.242 / 0.448
option-permutation ensemble (4 orders)	0.853 / 0.616	0.241 / 0.442
deep, round 3, step 500	0.849 / 0.603	0.247 / 0.456
deep, round 3, step 1,000	0.851 / 0.610	0.243 / 0.450

Rounds 1 to 3 hardly differed. Training stopped at step 1,000 under the predefined criterion. Possible causes are that the slots only see what the same frozen layers 16–24 compute and so have no new source of information; that many of the mined hard questions were soft-label items whose labels are themselves contested, so the correction may have fitted mostly noise; and that the learning rate of 3e-4 pushed the correction off course in the first 500 steps. A further attempt should start the loop at a shallower layer, replace rather than add to the readout, and first confirm that the architecture can overfit 1,000 to 2,000 questions.

10.2 Self-retrieval reread

For long documents we tried letting the model find evidence and read it again. The input is split into passages of about 400 characters; at the answer position, the attention mass of every head [87] in the 8 full-attention layers on each passage is read and the attention for the same input with an empty question is subtracted (without this, the selected passages are almost always the document’s opening and section headings). The 4 highest-scoring passages are quoted in their original order after the input and before the second copy of the question, and the model reads again. Retrieval heads were selected on a dev part of the long-document questions in v2 validation and tested on the test part and on JevBench long-document tasks:

Zero-shot self-retrieval reread accuracy (%)

dev (157, used to select heads)

selected heads81.5

all layers79.0

no contrast79.0

random passages77.7

original readout 79.6

test (162)

selected heads74.7

all layers74.7

no contrast75.9

random passages74.7

original readout 72.2

JevBench long documents (37)

selected heads56.8

all layers59.5

no contrast54.1

random passages56.8

original readout 62.2

Retrieval brought no gain over random passages; the improvement on the test part came from quoting the source again, and JevBench long-document tasks got worse. Spot checks showed that the contrastive calibration did select relevant passages, but the evidence was already in context during the original readout, and the model’s shortcoming was in reasoning with it.

10.3 Other methods

Method	Hypothesis	Result
Prior plus residual (PRR)	LM-head prior plus a zero-initialised joint-head residual	no significant difference from the plain letter readout (section 4.4)
Anchor KL	keep non-answer positions close to the base distribution	significantly worse in distribution, no significant JevBench gain
Permutation consistency	symmetric KL between two option orders of the same question	−2.1 on JevBench on top of the anchor
Permutation ensemble after training	deep mode averages 4 option orders	+0.1 [−0.9, +1.1] at four times the compute
Two-letter labels	labels AA, AB and so on beyond 26 options	19.9 points below the tournament (section 4.3)
Quantization in MLX	4-bit or 8-bit weights for lower latency	15–25% slower (section 3.1)
mx.compile	compile the MLX graph	no gain, slower with quantization

11. Deployment

11.1 Inference engines and release format

For release the LoRA is merged into the original Qwen3.5-4B weights with the layout unchanged (vision tower included), so mlx-lm and transformers load them directly; the exits and logit scales are stored separately, and the prompt version, exit temperatures, adaptive threshold and yes/no map are written to a configuration file. On 40 JevBench hard tasks (deep and adaptive), the merged bf16 weights and the unmerged LoRA agree on 120/120 argmaxes, with a mean probability difference of 0.005 and a maximum of 0.12 (where adaptive mode switches exits).

The MLX engine targets Apple silicon. We ported Qwen3.5’s vision tower and its interleaved multimodal RoPE from transformers. The vision tower must run in float32: in bf16 a few tokens per image deviate strongly, with cosine similarity as low as 0.44. End to end, the port matches the cloud PyTorch results (Mind2Web 92.3 against 91.9). On an M5 Pro the JevBench public set has a p50 of 113 ms at full depth and 71 ms adaptive.

The PyTorch engine targets CUDA with segmented execution and CUDA graphs (section 5.4), and matches MLX outputs on MPS.

11.2 ROCK 5B

The ROCK 5B (RK3588, 16 GB) has three backends. The PyTorch ARM CPU backend quantizes the decoder’s linear layers to INT8 with QNNPACK and supports every feature, but an adaptive request takes about 8 seconds. Rockchip’s RKLLM [88] runs full-depth computation on the NPU (about 2.1 seconds for the same invoice), but exposes no intermediate layers and cannot resume from one, which rules out variable depth and image input.

We wrote a C++ bridge to the RKNN Matmul API that sends only the decoder’s linear layers to the NPU; norms, attention and DeltaNet state, convolutions, the exits and the vision tower stay on the CPU. The trained exits work unchanged, and text and images share one resident model, with no reload between CPU and NPU backends. Weights are quantized on the board at start-up to symmetric per-output-channel INT8, with per-token activation quantization.

On the board we found that RKNN matrix-multiplication time is far from proportional to matrix size. One 64-row run of the MLP down projection (K=9216, N=2560) took 135 ms, whereas each of its three K=3072 slices took 0.9 ms; K=4096 and N=9216 were also slower than two halves. Each linear layer is therefore split into tiles with K at most 3072 and N at most 4608; K slices are accumulated exactly in INT32 before the scales are applied, so the output is bit-identical to the untiled computation.

HTTP request time on the ROCK 5B (seconds)

invoice yes/no, adaptive (layer 16)

NPU linear layers, untiled6.02

tiled1.71

tiled + CPU optimisations1.18

invoice yes/no, full depth (layer 32)

NPU linear layers, untiled12.04

tiled3.46

tiled + CPU optimisations2.47

RKLLM2.07

image request (layer 16)

NPU linear layers, untiled11.11

tiled4.74

tiled + CPU optimisations4.13

The live demo runs on this board.

12. Limitations and future work

12.1 Hard judgements

We analysed the 126 DecisionBench hard questions that v3 answers incorrectly.

• By type, accuracy is 62.5% on yes/no, 54.3% on choice and 50.7% on score questions; 30 of the 35 score errors are off by one level.
• The errors are unrelated to input length: accuracy is 80%, 53%, 57% and 59% for states under 3k characters, 3–6k, 6–12k and over 12k, and the median hard-tier state (4.7k characters) is no longer than the medium-tier one (5.2k).
• The model shows no bias towards yes: it predicts yes for 0.48 of questions against 0.47 in the gold labels.
• 45 of the errors have confidence above 0.8.

Typical errors involve multi-step reasoning within the document, such as computing elapsed time from timestamps, tracing the value of a variable through a code change, applying a pinned rule in the presence of a distracting message, and interpreting an inverted grading scale. Accuracy increases with depth:

v3 accuracy on DecisionBench hard at each exit (%)

• DecisionBench hard

Plumb-4B leads on this evaluation by 12 points. Its training differs from ours in the following respects: it starts from another decision model that has already been trained; it mines hard questions with its own model and trains only on teacher questions the model answers incorrectly or with low confidence (3,204 of 5,026), plus an equal number of human-labelled replay questions; it trains for 3 epochs on this small, hard set at a learning rate of 3e-5; and its questions are set in LLM-written, multi-source business documents, each solved twice independently, closer in style to DecisionBench hard. Our v3 trains for one pass at 1e-5, and its new sources are algorithmic conversions with short, clean contexts. As next steps we plan to mine questions from our candidate pool that v3 answers wrongly or with a gold probability below 0.6, mix them half and half with replay, and train for 2 to 3 epochs at 2e-5 to 3e-5; and then to consider documents in the same style written by an LLM, keeping only questions on which two independent answers agree.

12.2 Other limitations

Multi-step date and number arithmetic within a single forward pass remains weak. JevBench temporal_numeric stayed at 6 to 7 of 15 tasks from v2 to v3, while the same family in our own data reaches 88%. Additional data did not resolve the problem, which may require a change in the model’s mechanism.

Probabilities are calibrated on our validation mix, and v3 is more confident than v2.1 on some external sets, so applications with high-stakes thresholds should recalibrate on their own data. The training data is almost entirely English. Latency is high for more than 26 options, which sharing the state’s KV cache across tournament groups is expected to reduce several-fold. Our JevBench figures are our own scoring of the public tasks; official full-open-set and sealed-set results require evaluation by the maintainers.

Appendix: timeline and compute cost

Day	Work
Day 1	survey; backbone probes; trainable Gated DeltaNet in MLX; v0 training begins on the M5 Pro laptop
Day 2	move to Modal; v1 trained (B200, about 2 hours)
Day 3	v1 evaluated; latent reasoning fails its criterion; method audit identifies the readout problem; zero-shot readout experiments; v2 data built
Day 4	small-scale ablations; v2 and v2.1 trained; Decision Index sampling; same-protocol comparison; v2.1 released
Day 5	v3 data and controlled comparison; v3 trained and evaluated; CUDA graphs implemented; v3 released

Metered Modal cost (US dollars)

v2 main run35

v3 comparison, training and evaluation32

four small-scale ablations20

v2.1 (two runs)20

v1 training and evaluation19

latent reasoning7

other evaluations and measurements27

Weights and code are released under Apache-2.0.

References

1. JevBench by Benchmark Heaven: benchmark and leaderboard for AI decision models (v1.5 method). Software and data. github.com/fstandhartinger/jevbench
2. Guo, C., et al. On Calibration of Modern Neural Networks. arXiv:1706.04599 (2017).
3. LocalLLaMA/typed-decisions. Hugging Face dataset card. huggingface.co/datasets/LocalLLaMA/typed-decisions
4. akhilaaa3. decision-bench. Hugging Face dataset. huggingface.co/datasets/akhilaaa3/decision-bench
5. limberc. this-that-complex-decisions. Hugging Face dataset. huggingface.co/datasets/limberc/this-that-complex-decisions
6. Tan, S., et al. JudgeBench: A Benchmark for Evaluating LLM-based Judges. arXiv:2410.12784 (2024).
7. fastino. fast-decisions. Hugging Face dataset. huggingface.co/datasets/fastino/fast-decisions
8. FitzGerald, J., et al. MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages. arXiv:2204.08582 (2022).
9. Decision Index: reproduce the decision-model benchmark suite locally or as one HF Job. Software. github.com/apolinario/decision-index
10. Qwen Team. Qwen3.5-4B and Qwen3.5-4B-Base. Hugging Face model cards. huggingface.co/Qwen/Qwen3.5-4B
11. ml-explore. MLX: an array framework for Apple silicon. Software. github.com/ml-explore/mlx
12. Yang, S., et al. Gated Delta Networks: Improving Mamba2 with Delta Rule. arXiv:2412.06464 (2024).
13. Clark, P., et al. Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge. arXiv:1803.05457 (2018).
14. Hendrycks, D., et al. Measuring Massive Multitask Language Understanding. arXiv:2009.03300 (2020).
15. Hu, E. J., et al. LoRA: Low-Rank Adaptation of Large Language Models. arXiv:2106.09685 (2021).
16. Yang, S., et al. Parallelizing Linear Transformers with the Delta Rule over Sequence Length. arXiv:2406.06484 (2024).
17. fla-org. flash-linear-attention: efficient implementations of linear attention. Software. github.com/fla-org/flash-linear-attention
18. TheoLeeCJ. SemIf (formerly OpenJev): semantic ifs from open models. Software. github.com/TheoLeeCJ/SemIf
19. Arora, S., et al. Just read twice: closing the recall gap for recurrent language models. arXiv:2407.05483 (2024).
20. Leviathan, Y., et al. Prompt Repetition Improves Non-Reasoning LLMs. arXiv:2512.14982 (2025).
21. Larson, S., et al. An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction. arXiv:1909.02027 (2019).
22. Casanueva, I., et al. Efficient Intent Detection with Dual Sentence Encoders. arXiv:2003.04807 (2020).
23. Li, Y., et al. LLM-as-Jev: LLMs Are Already Jev-Style Decision Models: When and How to Fine-Tune Them. arXiv:2610.02076 (2026).
24. nostalgebraist. interpreting GPT: the logit lens. LessWrong (2020). lesswrong.com
25. Belrose, N., et al. Eliciting Latent Predictions from Transformers with the Tuned Lens. arXiv:2303.08112 (2023).
26. Hinton, G., et al. Distilling the Knowledge in a Neural Network. arXiv:1503.02531 (2015).
27. Schuster, T., et al. Confident Adaptive Language Modeling. arXiv:2207.07061 (2022).
28. Elhoushi, M., et al. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. arXiv:2404.16710 (2024).
29. Hao, S., et al. Training Large Language Models to Reason in a Continuous Latent Space. arXiv:2412.06769 (2024).
30. Geiping, J., et al. Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approach. arXiv:2502.05171 (2025).
31. Goyal, S., et al. Think before you speak: Training Language Models With Pause Tokens. arXiv:2310.02226 (2023).
32. Liusie, A., et al. Teacher-Student Training for Debiasing: General Permutation Debiasing for Large Language Models. arXiv:2403.13590 (2024).
33. Zheng, C., et al. Large Language Models Are Not Robust Multiple Choice Selectors. arXiv:2309.03882 (2023).
34. Wang, Z., et al. Eliminating Position Bias of Language Models: A Mechanistic Approach. arXiv:2407.01100 (2024).
35. NVIDIA. Getting Started with CUDA Graphs. NVIDIA Technical Blog. developer.nvidia.com/blog/cuda-graphs
36. TypeSafe. Jev 1.13 and the System One API (typed yes/no, choice and score decisions). Documentation. docs.typesafe.ai/concepts/system-one
37. Nie, Y., et al. What Can We Learn from Collective Human Opinions on Natural Language Inference Data? arXiv:2010.03532 (2020).
38. Kennedy, C. J., et al. Measuring a hate speech spectrum with faceted Rasch item response theory and perspective-aware, explainable-by-design deep learning. arXiv:2009.10277 (2020).
39. Sap, M., et al. Social Bias Frames: Reasoning about Social and Power Implications of Language. arXiv:1911.03891 (2019).
40. Zheng, L., et al. Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena. arXiv:2306.05685 (2023).
41. Hendrycks, D., et al. CUAD: An Expert-Annotated NLP Dataset for Legal Contract Review. arXiv:2103.06268 (2021).
42. Wang, S. H., et al. MAUD: An Expert-Annotated Legal NLP Dataset for Merger Agreement Understanding. arXiv:2301.00876 (2023).
43. Saeidi, M., et al. Interpretation of Natural Language Rules in Conversational Machine Reading. arXiv:1809.01494 (2018).
44. Guha, N., et al. LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models. arXiv:2308.11462 (2023).
45. Lightman, H., et al. Let’s Verify Step by Step. arXiv:2305.20050 (2023).
46. Wang, Z., et al. HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM. arXiv:2311.09528 (2023).
47. Fabbri, A. R., et al. SummEval: Re-evaluating Summarization Evaluation. arXiv:2007.12626 (2020).
48. Li, J., et al. HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models. arXiv:2305.11747 (2023).
49. Trivedi, H., et al. MuSiQue: Multihop Questions via Single-hop Question Composition. arXiv:2108.00573 (2021).
50. Ho, X., et al. Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Steps. arXiv:2011.01060 (2020).
51. Yang, Z., et al. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. arXiv:1809.09600 (2018).
52. Dua, D., et al. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs. arXiv:1903.00161 (2019).
53. Zhu, F., et al. TAT-QA: A Question Answering Benchmark on a Hybrid of Tabular and Textual Content in Finance. arXiv:2105.07624 (2021).
54. Clark, C., et al. BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questions. arXiv:1905.10044 (2019).
55. Geva, M., et al. Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies. arXiv:2101.02235 (2021).
56. Lin, K., et al. Reasoning Over Paragraph Effects in Situations. arXiv:1908.05852 (2019).
57. Bowman, S. R., et al. A large annotated corpus for learning natural language inference. arXiv:1508.05326 (2015).
58. Williams, A., Nangia, N. and Bowman, S. R. A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference. arXiv:1704.05426 (2017).
59. Bhagavatula, C., et al. Abductive Commonsense Reasoning. arXiv:1908.05739 (2019).
60. Deng, X., et al. Mind2Web: Towards a Generalist Agent for the Web. arXiv:2306.06070 (2023).
61. Yang, J., et al. Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V. arXiv:2310.11441 (2023).
62. Schwenk, D., et al. A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge. arXiv:2206.01718 (2022).
63. Lu, P., et al. Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering. arXiv:2209.09513 (2022).
64. Fatemi, B., et al. Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning. arXiv:2406.09170 (2024).
65. Tan, Q., et al. Towards Benchmarking and Improving the Temporal Reasoning Capability of Large Language Models. arXiv:2306.08952 (2023).
66. crh225. Plumb-4B. Hugging Face model card. huggingface.co/crh225/plumb-4b
67. Chen, Z., et al. FinQA: A Dataset of Numerical Reasoning over Financial Data. arXiv:2109.00122 (2021).
68. Cheng, Z., et al. HiTab: A Hierarchical Table Dataset for Question Answering and Natural Language Generation. arXiv:2108.06712 (2021).
69. Zhong, V., et al. Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning (WikiSQL). arXiv:1709.00103 (2017).
70. Tafjord, O., et al. ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language. arXiv:2012.13048 (2020).
71. Kazemi, M., et al. BoardgameQA: A Dataset for Natural Language Reasoning with Contradictory Information. arXiv:2306.07934 (2023).
72. ngrok-adhoc. JevK5. Software. github.com/ngrok-adhoc/jevk5
73. abhishek085. spark-s1-4b-v6. Hugging Face model card. huggingface.co/abhishek085/spark-s1-4b-v6
74. Mapika. decider-4b. Hugging Face model card. huggingface.co/Mapika/decider-4b
75. sthanika-ai. Sieve-4B. Hugging Face model card. huggingface.co/sthanika-ai/Sieve-4B
76. Liquid AI. Open d1. Liquid AI blog. liquid.ai/blog/d1-open
77. Qwen Team. Qwen3.8-27B. Hugging Face model card. huggingface.co/Qwen/Qwen3.8-27B
78. Perplexity. pplx-decider-v1-27b. Hugging Face model card. huggingface.co/perplexity-ai/pplx-decider-v1-27b
79. Louis, A., et al. “I’d rather just go to bed”: Understanding Indirect Answers. arXiv:2010.03450 (2020).
80. Niu, C., et al. RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models. arXiv:2401.00396 (2024).
81. Sakaguchi, K., et al. WinoGrande: An Adversarial Winograd Schema Challenge at Scale. arXiv:1907.10641 (2019).
82. Malo, P., et al. Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Texts (Financial PhraseBank). arXiv:1307.5336 (2013). arXiv:1307.5336
83. Lin, S., et al. TruthfulQA: Measuring How Models Mimic Human Falsehoods. arXiv:2109.07958 (2021).
84. Berkeley Function-Calling Leaderboard (BFCL). Gorilla project, UC Berkeley. gorilla.cs.berkeley.edu
85. Li, M., et al. API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs. arXiv:2304.08244 (2023).
86. Zellers, R., et al. HellaSwag: Can a Machine Really Finish Your Sentence? arXiv:1905.07830 (2019).
87. Zhang, W., et al. Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-ranking. arXiv:2506.09944 (2025).
88. airockchip. rknn-llm (RKLLM runtime). Software. github.com/airockchip/rknn-llm