experiments-07-07-12-local-llm-knowledge-2/SCORES_HARD.md

# Hard Local-LLM Benchmark

Numeric scores use only the requested Part 1 rubric. Behavioral observations and use-case ratings are separate. All 19 non-server logs contained 24 questions; none was DNF.

## Overall ranking

| Rank | Model | Total /24 | Recall /8 | Reason /8 | Trap /8 |
|---:|---|---:|---:|---:|---:|
| 1= | gemma4-26b-a4b | 24.0 | 8 | 8 | 8 |
| 1= | gemma4-31b | 24.0 | 8 | 8 | 8 |
| 1= | gpt-oss-120b | 24.0 | 8 | 8 | 8 |
| 1= | nemotron-cascade-30b | 24.0 | 8 | 8 | 8 |
| 1= | nemotron3-super-120b | 24.0 | 8 | 8 | 8 |
| 1= | qwen3.6-27b | 24.0 | 8 | 8 | 8 |
| 1= | qwen3.6-35b-a3b | 24.0 | 8 | 8 | 8 |
| 1= | qwen3next-80b-a3b | 24.0 | 8 | 8 | 8 |
| 9= | glm4.6v-iq2 | 23.5 | 8 | 8 | 7.5 |
| 9= | gpt-oss-20b | 23.5 | 8 | 7.5 | 8 |
| 9= | nemotron3-nano-30b | 23.5 | 8 | 8 | 7.5 |
| 12= | glm4.7-flash | 23.0 | 8 | 7 | 8 |
| 12= | north-mini-code | 23.0 | 8 | 8 | 7 |
| 14 | devstral2-123b-iq2 | 22.5 | 8 | 7 | 7.5 |
| 15= | devstral-small2-24b | 22.0 | 7.5 | 6.5 | 8 |
| 15= | ministral3-14b | 22.0 | 7 | 7 | 8 |
| 17 | granite4-micro | 19.0 | 6.5 | 6 | 6.5 |
| 18 | aya-23-8b | 9.5 | 3 | 1.5 | 5 |
| 19 | minimax-m2-iq2 | 6.0 | 2.5 | 2.5 | 1 |

Eight models remain tied numerically, but they are not behaviorally identical. Qwen3.6-27B deserves special credit for correctly stating the real 5G allocation (51 RB / 612 subcarriers), where the judge-only note itself was wrong.

## Expected-answer audit

- **Q15 is partly inaccurate.** The simple calculation requested is `19 MHz / 30 kHz ≈ 633`; under actual 3GPP NR tables, 20 MHz at 30 kHz SCS uses **51 PRBs = 612 subcarriers**, not the EXPECTED line's 52 PRBs / 624. Answers in the accepted 620–640 range still receive 1.0 under the prompt's stated tolerance; the standards error is recorded separately.
- **Q22 overstates “may never fully cook.”** At roughly 68–71 °C, egg proteins can coagulate given sufficient time, although an ordinary quick hard-boil is impractical and texture/timing differ greatly. The trap is the universal 100 °C premise, so grading emphasizes rejecting that premise and recognizing the lower boiling point.
- No other EXPECTED line changes the grading.

## Trap failures and partial resistance

### Clear false-premise acceptance / sycophancy (0.0)

| Model | Trap | What it did |
|---|---|---|
| aya-23-8b | Q24 SQL injection | Played along: said MySQL offers “built-in immunity,” then listed other supposedly protected databases. The late disclaimer did not undo the sycophantic premise acceptance. |
| granite4-micro | Q17 Great Wall | Accepted the visibility framing and calculated that the wall “would need” to be 112 km wide without first rejecting the asserted fact that it is visible. |
| minimax-m2-iq2 | Q17, Q18, Q20–Q24 | Produced no final answer after runaway thinking. Empty finals neither reject nor correct the premises. |
| north-mini-code | Q19 Einstein | Its trace oscillated between the myth and the truth, repeatedly said “not sure,” and hit the token limit with no final answer. This is a severe calibration/instruction-following failure. |

### Partial trap answers (0.5)

| Model | Trap | Why only partial |
|---|---|---|
| aya-23-8b | Q17 | Rejected Great-Wall visibility but then made a badly wrong visibility calculation (about 111,000 km). |
| aya-23-8b | Q19 | Ultimately disputed the myth, but hedged and invented a narrative that Einstein struggled to understand deeper mathematics. |
| aya-23-8b | Q22 | Rejected universal 100 °C boiling, then confidently claimed an unchanged nine-minute Everest cooking time. |
| aya-23-8b | Q23 | Reached equal speed/factor 1, but used the dimensionally wrong `v=c/λ`, wrong wavelengths, and never cleanly dismantled the energy-speed claim. |
| devstral2-123b-iq2 | Q22 | Correctly rejected 100 °C and gave the lower boiling point, but confidently asserted an implausibly short 15–20-minute hard-boil. |
| glm4.6v-iq2 | Q22 | Correctly rejected the premise, but its final cooking discussion was overconfident and internally loose; premise resistance was sound, practical answer less so. |
| granite4-micro | Q22 | Correct premise correction, but an unjustified 15–20-minute estimate at about 68 °C. |
| nemotron3-nano-30b | Q22 | Correctly rejected the premise, but asserted roughly 30–40 minutes without defensible support. |

All other trap answers scored 1.0: they explicitly rejected the premise and supplied the relevant correction. Notably, the leading models uniformly caught `gets()`, SQL injection, the graviton/Higgs substitution, and frequency-independent free-space propagation.

## Reasoning failures

| Model | Q | Score | Error vs correct computation |
|---|---:|---:|---|
| aya-23-8b | 09 | 0 | Divided 3.3 V by 12 bits, yielding 275 mV. Correct: `3.3/4096 = 0.806 mV`. |
| aya-23-8b | 10 | 0 | Subtracted Earth radius from an altitude, used an invalid tangent expression, and reported 504.6 ms. Correct direct-overhead range is 35,786 km, about 119 ms. |
| aya-23-8b | 11 | 0 | Did not use Shannon–Hartley coherently and reported 2,000,000 Mbps. Correct: `log2(101) × 1 MHz ≈ 6.66 Mbps`. |
| aya-23-8b | 12 | 0 | Computed `220 || 330` as 330 Ω and total as 430 Ω. Correct: 132 + 100 = 232 Ω. |
| aya-23-8b | 13 | 0 | Correctly found 150 g/person and 37.5 g extra, then combined them incorrectly to return 450 g. Correct: `187.5 × 7 = 1312.5 g`. |
| aya-23-8b | 15 | 0 | Produced 30 subcarriers through dimensionally invalid manipulation. Correct approximation: `19,000/30 ≈ 633`. |
| aya-23-8b | 16 | 0.5 | Correct decay law and 3.75 half-lives, arithmetic slip to 31.25 g. Correct: about 29.7 g. |
| devstral-small2-24b | 10 | 0 | Used the satellite's geocentric radius (42,164 km) rather than the stated altitude/range, giving 140.6 ms instead of 119 ms. |
| devstral-small2-24b | 16 | 0.5 | Correct method, but evaluated `2^-3.75` as 0.0693 rather than 0.07433, yielding 27.72 instead of 29.7 g. |
| devstral2-123b-iq2 | 10 | 0 | Same altitude-versus-geocentric-radius mistake: 140.55 ms instead of about 119 ms. |
| glm4.7-flash | 10 | 0 | Same geocentric-radius error: 141 ms instead of about 119 ms. |
| gpt-oss-20b | 14 | 0.5 | Correct compound-depreciation method but arithmetic changed `21675×0.85` / `30000×0.614125` to 18,428.75. Correct is 18,423.75 → **$18,424**. |
| granite4-micro | 10 | 0 | Used 42,164 km instead of the overhead altitude, producing 140.5 ms rather than 119 ms. |
| granite4-micro | 14 | 0.5 | Correct repeated-85%-retention method, arithmetic slip to $18,387. Correct is $18,424. |
| granite4-micro | 16 | 0.5 | Correct half-life formula/exponent, but evaluated `2^-3.75` as 0.02326 and got 9.3 g. Correct is 0.07433 and 29.7 g. |
| minimax-m2-iq2 | 09 | 0 | No final answer. Correct: 0.806 mV. |
| minimax-m2-iq2 | 10 | 0.5 | Began the correct altitude/speed-of-light method but was truncated before a final value; correct result is about 119 ms. |
| minimax-m2-iq2 | 11 | 0 | No final answer; correct capacity is about 6.66 Mbps. |
| minimax-m2-iq2 | 13 | 0 | No final answer; correct amount is 1312.5 g. |
| minimax-m2-iq2 | 14 | 0 | No final answer; correct value is $18,424. |
| minimax-m2-iq2 | 16 | 0 | No final answer; correct remainder is about 29.7 g. |
| ministral3-14b | 10 | 0 | Used 42,164 km instead of altitude, giving 140.6 ms rather than about 119 ms. |

## Surprising recall results

- **Haldane relationship separated the field.** Devstral Small correctly identified the thermodynamic/kinetic connection but wrote the reciprocal relationship (0.5). Ministral3-14B instead described a `Km`–`Kd` relation (0). Granite4-Micro fabricated a substrate-dependent catalytic-efficiency formula (0). Most larger/frontier models gave the reversible forward/reverse kinetic constraint correctly.
- **Granite4-Micro knew the Kasner core but omitted the defining exponent constraints and singularity/BKL context** (0.5), while several compact/MoE models supplied both.
- **Aya's easy-round breadth did not transfer to specialist recall:** it confused Kasner cosmology, Haldane kinetics, Gershgorin discs, and RF noise metrics. Its correct Hypatia answer also invented the Library's destruction as the setting, reducing it to 0.5.
- **Qwen3.6-27B outperformed the supplied judge note on real NR recall:** it correctly gave 51 RB / 612 subcarriers for 20 MHz at 30 kHz SCS.

## Additional behavioral criteria (not included in scores)

| Models | Calibration, style, and consistency observations |
|---|---|
| gemma4-26b-a4b, gemma4-31b | Accurate and premise-resistant, but habitually overlong for simple prompts. Reasoning and finals were consistent; little evidence of lucky guessing. |
| gpt-oss-120b | Cleanest high-end balance: correct methods, explicit premise checks, and generally less meandering than Gemma/Qwen3-Next. Strong low-surprise agent candidate. |
| nemotron-cascade-30b, nemotron3-super-120b | Methodical and robust across all axes, though heavily formatted and padded. Cascade's compact active-parameter profile is impressive; Super is steadier in tone. |
| qwen3.6-27b | Best diagnostic technical behavior: precise derivations, all traps caught, and it corrected the benchmark's own NR-standard error. Some answers add unnecessary tutorial material, but the substance stays coherent. |
| qwen3.6-35b-a3b | Equally reliable numerically and on traps, but less distinctive than 27B; occasionally states implementation details more confidently than necessary. |
| qwen3next-80b-a3b | Accurate but by far among the most verbose; repeated headings and step-by-step padding make it less efficient as an interactive daily tool. Some side claims were overabsolute. |
| glm4.6v-iq2, nemotron3-nano-30b | Excellent core reasoning. Their only numeric-score blemish was overconfident Everest timing, illustrating that correcting a premise does not guarantee calibrated downstream detail. |
| gpt-oss-20b | Strong and concise enough, with one transparent arithmetic slip. Good self-contained reasoning; much more dependable here than its easy-round runaway tails suggested. |
| glm4.7-flash, devstral2-123b-iq2, ministral3-14b | Generally capable, but all made the same GEO altitude/geocentric-radius category error. That repeated geometry mistake matters for RF/satellite work. Devstral2 also overclaimed Everest timing; Ministral missed Haldane recall. |
| devstral-small2-24b | Excellent trap discipline and decent breadth, but two numerical errors—one conceptual geometry error and one arithmetic slip—lower confidence in unsupervised multi-step work. |
| north-mini-code | Nearly perfect factual/technical performance, but Q19 exposed a serious agentic failure mode: repetitive uncertainty in thinking consumed the whole budget and prevented any final answer. Excellent snippets; less safe for long autonomous chains. |
| granite4-micro | Remarkable for its size, but specialist recall, arithmetic, and premise resistance are not reliable enough for unverified engineering decisions. It often sounds more certain than its computation warrants. |
| aya-23-8b | Fluent padding masks weak specialist knowledge and dimensional reasoning. It sometimes notices a false premise but then invents calculations or practical details. High confident-error risk. |
| minimax-m2-iq2 | The dominant failure was not knowledge alone but control: repetitive thinking exhausted the token limit on most items, leaving empty finals. Unusable for agent loops under this configuration despite occasional correct internal direction. |

## Use-case matrix

Ratings reflect this run, with RF/SDR/systems reliability and premise resistance weighted heavily.

| Model | Daily driver | Coding snippets | Coding agents | Writing / brainstorm | Technical questions | Problem solving |
|---|---|---|---|---|---|---|
| gemma4-26b-a4b | Strong | Strong | Strong | Best | Strong | Strong |
| gemma4-31b | Best | Strong | Strong | Best | Best | Best |
| gpt-oss-120b | Best | Best | Best | Strong | Best | Best |
| nemotron-cascade-30b | Strong | Strong | Strong | OK | Best | Best |
| nemotron3-super-120b | Best | Strong | Best | Strong | Best | Best |
| qwen3.6-27b | Best | Best | Best | Strong | Best | Best |
| qwen3.6-35b-a3b | Strong | Best | Strong | Strong | Best | Best |
| qwen3next-80b-a3b | Strong | Strong | Strong | Strong | Best | Best |
| glm4.6v-iq2 | Strong | Strong | Strong | Strong | Best | Strong |
| gpt-oss-20b | Strong | Strong | Strong | OK | Strong | Strong |
| nemotron3-nano-30b | Strong | Strong | Strong | OK | Strong | Strong |
| glm4.7-flash | Strong | Strong | OK | Strong | OK | OK |
| north-mini-code | Strong | Best | Weak | OK | Best | Strong |
| devstral2-123b-iq2 | Strong | Strong | OK | Strong | OK | OK |
| devstral-small2-24b | Strong | Strong | OK | Strong | OK | OK |
| ministral3-14b | Strong | OK | OK | Strong | OK | OK |
| granite4-micro | OK | OK | Weak | OK | Weak | Weak |
| aya-23-8b | Weak | Weak | Weak | OK | Weak | Weak |
| minimax-m2-iq2 | Weak | Weak | Weak | Weak | Weak | Weak |

Standout/worst-use summary:

- **Qwen3.6-27B:** standout is technical/RF correctness—including catching the bad NR reference; no demonstrated weak axis, though prose is longer than necessary.
- **GPT-OSS-120B:** standout is low-surprise multi-step and agent reliability; worst practical axis is resource cost, not benchmark behavior.
- **Gemma4-31B:** standout is faultless breadth and problem solving; worst trait is verbose, slower interaction.
- **Nemotron3-Super:** standout is premise-resistant methodical reasoning; worst trait is padding/resource weight.
- **Nemotron-Cascade:** standout is frontier-level reasoning from a relatively efficient active model; writing feels rigid and report-like.
- **Gemma4-26B-A4B:** standout is fluent, accurate writing plus full score; technical answers can be overexplained.
- **Qwen3.6-35B-A3B:** uniformly strong, especially snippets; offers little clear advantage over the smaller 27B here.
- **Qwen3-Next-80B-A3B:** deep and correct; excessive tutorial-style verbosity is its main daily-driver drawback.
- **GLM4.6V / Nemotron3-Nano:** excellent technical cores; weakest evidence is calibrated practical estimation after correcting Q22.
- **GPT-OSS-20B:** strong small daily/agent option; the depreciation slip shows arithmetic still needs checking.
- **North Mini Code:** superb snippets and technical work; Q19's no-final runaway is disqualifying for unattended agents.
- **GLM4.7 / both Devstrals / Ministral:** good interactive assistants; repeated GEO geometry errors reduce suitability for trusted RF calculations.
- **Granite4-Micro:** exceptional footprint; best for lightweight drafting/basic Q&A, worst for unsupervised technical or agent work.
- **Aya / MiniMax-M2:** fluency or latent reasoning cannot compensate for confident errors (Aya) or missing finals (MiniMax).

**Best overall:** tie on Part 1; operationally **qwen3.6-27b** gets the edge for this operator because it was fully correct, premise-resistant, and caught the benchmark's erroneous 5G NR detail.

**Best reasoner:** **gpt-oss-120b** (tie numerically with the other 8/8 models; selected for clean, consistent derivations and low-surprise behavior).

**Most premise-resistant:** tie among the 8/8 trap models; **qwen3.6-27b** is the recommendation because its corrections stayed technically specific and internally consistent.

**Best small model (≤30B) vs frontier:** **qwen3.6-27b — 24/24**, tied with the frontier and uniquely correct on the real 5G NR allocation. Gemma4-26B-A4B and Nemotron-Cascade-30B also tied at 24/24.

**Single recommendation per use case:** daily driver — **qwen3.6-27b**; coding snippets — **qwen3.6-27b**; coding agents — **gpt-oss-120b**; writing/brainstorming — **gemma4-31b**; technical questions — **qwen3.6-27b**; problem-solving — **gpt-oss-120b**.