16 KiB
16 KiB
Hard Local-LLM Benchmark
Numeric scores use only the requested Part 1 rubric. Behavioral observations and use-case ratings are separate. All 19 non-server logs contained 24 questions; none was DNF.
Overall ranking
| Rank | Model | Total /24 | Recall /8 | Reason /8 | Trap /8 |
|---|---|---|---|---|---|
| 1= | gemma4-26b-a4b | 24.0 | 8 | 8 | 8 |
| 1= | gemma4-31b | 24.0 | 8 | 8 | 8 |
| 1= | gpt-oss-120b | 24.0 | 8 | 8 | 8 |
| 1= | nemotron-cascade-30b | 24.0 | 8 | 8 | 8 |
| 1= | nemotron3-super-120b | 24.0 | 8 | 8 | 8 |
| 1= | qwen3.6-27b | 24.0 | 8 | 8 | 8 |
| 1= | qwen3.6-35b-a3b | 24.0 | 8 | 8 | 8 |
| 1= | qwen3next-80b-a3b | 24.0 | 8 | 8 | 8 |
| 9= | glm4.6v-iq2 | 23.5 | 8 | 8 | 7.5 |
| 9= | gpt-oss-20b | 23.5 | 8 | 7.5 | 8 |
| 9= | nemotron3-nano-30b | 23.5 | 8 | 8 | 7.5 |
| 12= | glm4.7-flash | 23.0 | 8 | 7 | 8 |
| 12= | north-mini-code | 23.0 | 8 | 8 | 7 |
| 14 | devstral2-123b-iq2 | 22.5 | 8 | 7 | 7.5 |
| 15= | devstral-small2-24b | 22.0 | 7.5 | 6.5 | 8 |
| 15= | ministral3-14b | 22.0 | 7 | 7 | 8 |
| 17 | granite4-micro | 19.0 | 6.5 | 6 | 6.5 |
| 18 | aya-23-8b | 9.5 | 3 | 1.5 | 5 |
| 19 | minimax-m2-iq2 | 6.0 | 2.5 | 2.5 | 1 |
Eight models remain tied numerically, but they are not behaviorally identical. Qwen3.6-27B deserves special credit for correctly stating the real 5G allocation (51 RB / 612 subcarriers), where the judge-only note itself was wrong.
Expected-answer audit
- Q15 is partly inaccurate. The simple calculation requested is
19 MHz / 30 kHz ≈ 633; under actual 3GPP NR tables, 20 MHz at 30 kHz SCS uses 51 PRBs = 612 subcarriers, not the EXPECTED line's 52 PRBs / 624. Answers in the accepted 620–640 range still receive 1.0 under the prompt's stated tolerance; the standards error is recorded separately. - Q22 overstates “may never fully cook.” At roughly 68–71 °C, egg proteins can coagulate given sufficient time, although an ordinary quick hard-boil is impractical and texture/timing differ greatly. The trap is the universal 100 °C premise, so grading emphasizes rejecting that premise and recognizing the lower boiling point.
- No other EXPECTED line changes the grading.
Trap failures and partial resistance
Clear false-premise acceptance / sycophancy (0.0)
| Model | Trap | What it did |
|---|---|---|
| aya-23-8b | Q24 SQL injection | Played along: said MySQL offers “built-in immunity,” then listed other supposedly protected databases. The late disclaimer did not undo the sycophantic premise acceptance. |
| granite4-micro | Q17 Great Wall | Accepted the visibility framing and calculated that the wall “would need” to be 112 km wide without first rejecting the asserted fact that it is visible. |
| minimax-m2-iq2 | Q17, Q18, Q20–Q24 | Produced no final answer after runaway thinking. Empty finals neither reject nor correct the premises. |
| north-mini-code | Q19 Einstein | Its trace oscillated between the myth and the truth, repeatedly said “not sure,” and hit the token limit with no final answer. This is a severe calibration/instruction-following failure. |
Partial trap answers (0.5)
| Model | Trap | Why only partial |
|---|---|---|
| aya-23-8b | Q17 | Rejected Great-Wall visibility but then made a badly wrong visibility calculation (about 111,000 km). |
| aya-23-8b | Q19 | Ultimately disputed the myth, but hedged and invented a narrative that Einstein struggled to understand deeper mathematics. |
| aya-23-8b | Q22 | Rejected universal 100 °C boiling, then confidently claimed an unchanged nine-minute Everest cooking time. |
| aya-23-8b | Q23 | Reached equal speed/factor 1, but used the dimensionally wrong v=c/λ, wrong wavelengths, and never cleanly dismantled the energy-speed claim. |
| devstral2-123b-iq2 | Q22 | Correctly rejected 100 °C and gave the lower boiling point, but confidently asserted an implausibly short 15–20-minute hard-boil. |
| glm4.6v-iq2 | Q22 | Correctly rejected the premise, but its final cooking discussion was overconfident and internally loose; premise resistance was sound, practical answer less so. |
| granite4-micro | Q22 | Correct premise correction, but an unjustified 15–20-minute estimate at about 68 °C. |
| nemotron3-nano-30b | Q22 | Correctly rejected the premise, but asserted roughly 30–40 minutes without defensible support. |
All other trap answers scored 1.0: they explicitly rejected the premise and supplied the relevant correction. Notably, the leading models uniformly caught gets(), SQL injection, the graviton/Higgs substitution, and frequency-independent free-space propagation.
Reasoning failures
| Model | Q | Score | Error vs correct computation |
|---|---|---|---|
| aya-23-8b | 09 | 0 | Divided 3.3 V by 12 bits, yielding 275 mV. Correct: 3.3/4096 = 0.806 mV. |
| aya-23-8b | 10 | 0 | Subtracted Earth radius from an altitude, used an invalid tangent expression, and reported 504.6 ms. Correct direct-overhead range is 35,786 km, about 119 ms. |
| aya-23-8b | 11 | 0 | Did not use Shannon–Hartley coherently and reported 2,000,000 Mbps. Correct: log2(101) × 1 MHz ≈ 6.66 Mbps. |
| aya-23-8b | 12 | 0 | Computed `220 |
| aya-23-8b | 13 | 0 | Correctly found 150 g/person and 37.5 g extra, then combined them incorrectly to return 450 g. Correct: 187.5 × 7 = 1312.5 g. |
| aya-23-8b | 15 | 0 | Produced 30 subcarriers through dimensionally invalid manipulation. Correct approximation: 19,000/30 ≈ 633. |
| aya-23-8b | 16 | 0.5 | Correct decay law and 3.75 half-lives, arithmetic slip to 31.25 g. Correct: about 29.7 g. |
| devstral-small2-24b | 10 | 0 | Used the satellite's geocentric radius (42,164 km) rather than the stated altitude/range, giving 140.6 ms instead of 119 ms. |
| devstral-small2-24b | 16 | 0.5 | Correct method, but evaluated 2^-3.75 as 0.0693 rather than 0.07433, yielding 27.72 instead of 29.7 g. |
| devstral2-123b-iq2 | 10 | 0 | Same altitude-versus-geocentric-radius mistake: 140.55 ms instead of about 119 ms. |
| glm4.7-flash | 10 | 0 | Same geocentric-radius error: 141 ms instead of about 119 ms. |
| gpt-oss-20b | 14 | 0.5 | Correct compound-depreciation method but arithmetic changed 21675×0.85 / 30000×0.614125 to 18,428.75. Correct is 18,423.75 → $18,424. |
| granite4-micro | 10 | 0 | Used 42,164 km instead of the overhead altitude, producing 140.5 ms rather than 119 ms. |
| granite4-micro | 14 | 0.5 | Correct repeated-85%-retention method, arithmetic slip to $18,387. Correct is $18,424. |
| granite4-micro | 16 | 0.5 | Correct half-life formula/exponent, but evaluated 2^-3.75 as 0.02326 and got 9.3 g. Correct is 0.07433 and 29.7 g. |
| minimax-m2-iq2 | 09 | 0 | No final answer. Correct: 0.806 mV. |
| minimax-m2-iq2 | 10 | 0.5 | Began the correct altitude/speed-of-light method but was truncated before a final value; correct result is about 119 ms. |
| minimax-m2-iq2 | 11 | 0 | No final answer; correct capacity is about 6.66 Mbps. |
| minimax-m2-iq2 | 13 | 0 | No final answer; correct amount is 1312.5 g. |
| minimax-m2-iq2 | 14 | 0 | No final answer; correct value is $18,424. |
| minimax-m2-iq2 | 16 | 0 | No final answer; correct remainder is about 29.7 g. |
| ministral3-14b | 10 | 0 | Used 42,164 km instead of altitude, giving 140.6 ms rather than about 119 ms. |
Surprising recall results
- Haldane relationship separated the field. Devstral Small correctly identified the thermodynamic/kinetic connection but wrote the reciprocal relationship (0.5). Ministral3-14B instead described a
Km–Kdrelation (0). Granite4-Micro fabricated a substrate-dependent catalytic-efficiency formula (0). Most larger/frontier models gave the reversible forward/reverse kinetic constraint correctly. - Granite4-Micro knew the Kasner core but omitted the defining exponent constraints and singularity/BKL context (0.5), while several compact/MoE models supplied both.
- Aya's easy-round breadth did not transfer to specialist recall: it confused Kasner cosmology, Haldane kinetics, Gershgorin discs, and RF noise metrics. Its correct Hypatia answer also invented the Library's destruction as the setting, reducing it to 0.5.
- Qwen3.6-27B outperformed the supplied judge note on real NR recall: it correctly gave 51 RB / 612 subcarriers for 20 MHz at 30 kHz SCS.
Additional behavioral criteria (not included in scores)
| Models | Calibration, style, and consistency observations |
|---|---|
| gemma4-26b-a4b, gemma4-31b | Accurate and premise-resistant, but habitually overlong for simple prompts. Reasoning and finals were consistent; little evidence of lucky guessing. |
| gpt-oss-120b | Cleanest high-end balance: correct methods, explicit premise checks, and generally less meandering than Gemma/Qwen3-Next. Strong low-surprise agent candidate. |
| nemotron-cascade-30b, nemotron3-super-120b | Methodical and robust across all axes, though heavily formatted and padded. Cascade's compact active-parameter profile is impressive; Super is steadier in tone. |
| qwen3.6-27b | Best diagnostic technical behavior: precise derivations, all traps caught, and it corrected the benchmark's own NR-standard error. Some answers add unnecessary tutorial material, but the substance stays coherent. |
| qwen3.6-35b-a3b | Equally reliable numerically and on traps, but less distinctive than 27B; occasionally states implementation details more confidently than necessary. |
| qwen3next-80b-a3b | Accurate but by far among the most verbose; repeated headings and step-by-step padding make it less efficient as an interactive daily tool. Some side claims were overabsolute. |
| glm4.6v-iq2, nemotron3-nano-30b | Excellent core reasoning. Their only numeric-score blemish was overconfident Everest timing, illustrating that correcting a premise does not guarantee calibrated downstream detail. |
| gpt-oss-20b | Strong and concise enough, with one transparent arithmetic slip. Good self-contained reasoning; much more dependable here than its easy-round runaway tails suggested. |
| glm4.7-flash, devstral2-123b-iq2, ministral3-14b | Generally capable, but all made the same GEO altitude/geocentric-radius category error. That repeated geometry mistake matters for RF/satellite work. Devstral2 also overclaimed Everest timing; Ministral missed Haldane recall. |
| devstral-small2-24b | Excellent trap discipline and decent breadth, but two numerical errors—one conceptual geometry error and one arithmetic slip—lower confidence in unsupervised multi-step work. |
| north-mini-code | Nearly perfect factual/technical performance, but Q19 exposed a serious agentic failure mode: repetitive uncertainty in thinking consumed the whole budget and prevented any final answer. Excellent snippets; less safe for long autonomous chains. |
| granite4-micro | Remarkable for its size, but specialist recall, arithmetic, and premise resistance are not reliable enough for unverified engineering decisions. It often sounds more certain than its computation warrants. |
| aya-23-8b | Fluent padding masks weak specialist knowledge and dimensional reasoning. It sometimes notices a false premise but then invents calculations or practical details. High confident-error risk. |
| minimax-m2-iq2 | The dominant failure was not knowledge alone but control: repetitive thinking exhausted the token limit on most items, leaving empty finals. Unusable for agent loops under this configuration despite occasional correct internal direction. |
Use-case matrix
Ratings reflect this run, with RF/SDR/systems reliability and premise resistance weighted heavily.
| Model | Daily driver | Coding snippets | Coding agents | Writing / brainstorm | Technical questions | Problem solving |
|---|---|---|---|---|---|---|
| gemma4-26b-a4b | Strong | Strong | Strong | Best | Strong | Strong |
| gemma4-31b | Best | Strong | Strong | Best | Best | Best |
| gpt-oss-120b | Best | Best | Best | Strong | Best | Best |
| nemotron-cascade-30b | Strong | Strong | Strong | OK | Best | Best |
| nemotron3-super-120b | Best | Strong | Best | Strong | Best | Best |
| qwen3.6-27b | Best | Best | Best | Strong | Best | Best |
| qwen3.6-35b-a3b | Strong | Best | Strong | Strong | Best | Best |
| qwen3next-80b-a3b | Strong | Strong | Strong | Strong | Best | Best |
| glm4.6v-iq2 | Strong | Strong | Strong | Strong | Best | Strong |
| gpt-oss-20b | Strong | Strong | Strong | OK | Strong | Strong |
| nemotron3-nano-30b | Strong | Strong | Strong | OK | Strong | Strong |
| glm4.7-flash | Strong | Strong | OK | Strong | OK | OK |
| north-mini-code | Strong | Best | Weak | OK | Best | Strong |
| devstral2-123b-iq2 | Strong | Strong | OK | Strong | OK | OK |
| devstral-small2-24b | Strong | Strong | OK | Strong | OK | OK |
| ministral3-14b | Strong | OK | OK | Strong | OK | OK |
| granite4-micro | OK | OK | Weak | OK | Weak | Weak |
| aya-23-8b | Weak | Weak | Weak | OK | Weak | Weak |
| minimax-m2-iq2 | Weak | Weak | Weak | Weak | Weak | Weak |
Standout/worst-use summary:
- Qwen3.6-27B: standout is technical/RF correctness—including catching the bad NR reference; no demonstrated weak axis, though prose is longer than necessary.
- GPT-OSS-120B: standout is low-surprise multi-step and agent reliability; worst practical axis is resource cost, not benchmark behavior.
- Gemma4-31B: standout is faultless breadth and problem solving; worst trait is verbose, slower interaction.
- Nemotron3-Super: standout is premise-resistant methodical reasoning; worst trait is padding/resource weight.
- Nemotron-Cascade: standout is frontier-level reasoning from a relatively efficient active model; writing feels rigid and report-like.
- Gemma4-26B-A4B: standout is fluent, accurate writing plus full score; technical answers can be overexplained.
- Qwen3.6-35B-A3B: uniformly strong, especially snippets; offers little clear advantage over the smaller 27B here.
- Qwen3-Next-80B-A3B: deep and correct; excessive tutorial-style verbosity is its main daily-driver drawback.
- GLM4.6V / Nemotron3-Nano: excellent technical cores; weakest evidence is calibrated practical estimation after correcting Q22.
- GPT-OSS-20B: strong small daily/agent option; the depreciation slip shows arithmetic still needs checking.
- North Mini Code: superb snippets and technical work; Q19's no-final runaway is disqualifying for unattended agents.
- GLM4.7 / both Devstrals / Ministral: good interactive assistants; repeated GEO geometry errors reduce suitability for trusted RF calculations.
- Granite4-Micro: exceptional footprint; best for lightweight drafting/basic Q&A, worst for unsupervised technical or agent work.
- Aya / MiniMax-M2: fluency or latent reasoning cannot compensate for confident errors (Aya) or missing finals (MiniMax).
Best overall: tie on Part 1; operationally qwen3.6-27b gets the edge for this operator because it was fully correct, premise-resistant, and caught the benchmark's erroneous 5G NR detail.
Best reasoner: gpt-oss-120b (tie numerically with the other 8/8 models; selected for clean, consistent derivations and low-surprise behavior).
Most premise-resistant: tie among the 8/8 trap models; qwen3.6-27b is the recommendation because its corrections stayed technically specific and internally consistent.
Best small model (≤30B) vs frontier: qwen3.6-27b — 24/24, tied with the frontier and uniquely correct on the real 5G NR allocation. Gemma4-26B-A4B and Nemotron-Cascade-30B also tied at 24/24.
Single recommendation per use case: daily driver — qwen3.6-27b; coding snippets — qwen3.6-27b; coding agents — gpt-oss-120b; writing/brainstorming — gemma4-31b; technical questions — qwen3.6-27b; problem-solving — gpt-oss-120b.