qoherent/experiments-07-07-12-local-llm-knowledge-2
23
0
Fork 0
Code Issues Pull Requests Workflows Packages Projects Releases Wiki Activity
Push Tracker
experiments-07-07-12-local-.../SCORES.md
G gael 5cb7da3855 readme report
2026-07-13 01:35:10 +00:00

14 KiB
Raw Blame History

Local-LLM Knowledge Benchmark Scores

Scoring follows the requested 1.0 / 0.5 / 0.0 rubric. No model log contained FAILED TO LOAD, so there are no DNFs. “Said” below is a concise faithful summary of the emitted answer; runaway reasoning tails were graded on the factual answer actually present in the log.

Ranking

Rank Model Score /40 Verdict
1 gemma4-31b 40.0 Flawless
2= gpt-oss-120b 39.5 Excellent
2= nemotron3-super-120b 39.5 Excellent
2= north-mini-code 39.5 Excellent
2= qwen3next-80b-a3b 39.5 Excellent
6= gemma4-26b-a4b 39.0 Excellent
6= glm4.6v-iq2 39.0 Excellent
8 devstral-small2-24b 38.5 Strong
9= devstral2-123b-iq2 38.0 Strong
9= ministral3-14b 38.0 Strong
9= qwen3.6-27b 38.0 Strong
12 qwen3.6-35b-a3b 37.5 Strong
13= gpt-oss-20b 36.5 Mixed
13= minimax-m2-iq2 36.5 Mixed
13= nemotron-cascade-30b 36.5 Mixed
16 nemotron3-nano-30b 36.0 Mixed
17 granite4-micro 35.5 Efficient
18 glm4.7-flash 33.5 Uneven
19 aya-23-8b 26.5 Weak

Per-subject breakdown

Model physics chemistry biology history geography literature music rf_engineering math econ_law
gemma4-31b 4 4 4 4 4 4 4 4 4 4
gpt-oss-120b 3.5 4 4 4 4 4 4 4 4 4
nemotron3-super-120b 3.5 4 4 4 4 4 4 4 4 4
north-mini-code 3.5 4 4 4 4 4 4 4 4 4
qwen3next-80b-a3b 3.5 4 4 4 4 4 4 4 4 4
gemma4-26b-a4b 3.5 4 4 4 4 4 4 4 4 3.5
glm4.6v-iq2 3.5 4 4 4 4 4 4 4 4 3.5
devstral-small2-24b 3.5 4 4 4 4 4 4 3 4 4
devstral2-123b-iq2 3.5 3.5 4 3.5 4 4 3.5 4 4 4
ministral3-14b 3.5 4 4 4 4 4 4 3 3.5 4
qwen3.6-27b 3.5 4 3.5 4 4 4 4 3 4 4
qwen3.6-35b-a3b 3.5 4 4 4 3.5 4 4 3 4 3.5
gpt-oss-20b 3.5 4 4 4 4 3.5 4 3 4 2.5
minimax-m2-iq2 3.5 3 4 4 4 4 3 4 3 4
nemotron-cascade-30b 3.5 4 4 4 4 4 3 2.5 4 3.5
nemotron3-nano-30b 4 4 3.5 4 4 3 3 2.5 4 4
granite4-micro 4 4 3.5 4 4 3.5 3 2.5 4 3
glm4.7-flash 4 3 4 4 3 3 2.5 3 4 3
aya-23-8b 3 3.5 2 4 3 3 3 0 1 4

Every answer scored below 1.0

aya-23-8b

  • Q01 — 0.0. Said about 1.4 solar masses, but described a generic stellar core/fusion limit and neutron-star/black-hole remnant. Correct: about 1.4–1.44 solar masses, the maximum electron-degeneracy-supported white-dwarf mass; the classic accreting-white-dwarf outcome is a Type Ia supernova.
  • Q07 — 0.5. Correctly named Haber–Bosch and iron, but called it an “iron and aluminum” catalyst. Correct: an iron-based catalyst, commonly with oxide promoters such as K₂O and Al₂O₃; aluminum is not the co-catalyst metal.
  • Q09 — 0.0. Said the phrenic nerve was cranial and arose at T1–T5. Correct: phrenic nerve from C3–C5.
  • Q10 — 0.0. Said its primary role was reabsorbing glucose and water. Correct: countercurrent multiplication establishing the medullary osmotic gradient, enabling urine concentration; glucose is reabsorbed chiefly in the proximal tubule.
  • Q17 — 0.0. Named Baikal but gave 1,642 feet / 500 m. Correct: Lake Baikal, about 1,642 metres (5,387 ft).
  • Q22 — 0.0. Said Virgil. Correct: Beatrice (with St Bernard taking over at the very end).
  • Q27 — 0.0. Said “Not sure.” Correct: Rick Rubin produced; Daron Malakian co-produced.
  • Q29 — 0.0. Said 35 dB. Correct: about 100 dB.
  • Q30 — 0.0. Said a Smith chart plots two-port S-parameters. Correct: it plots complex reflection coefficient / normalized impedance or admittance and is used for transmission-line analysis and impedance matching.
  • Q31 — 0.0. Omitted bandwidth and gave C = log2(1+S/N). Correct: C = B log2(1+S/N), with C capacity, B bandwidth, S signal power, and N noise power.
  • Q32 — 0.0. Claimed a 150-chip PN code and “pulse-code modulation.” Correct: BPSK with DSSS using the 1023-chip C/A Gold code at 1.023 Mcps.
  • Q33 — 0.0. Claimed ζ(2) lacked a simple value and supplied a purported divergent expansion. Correct: ζ(2) = π²/6.
  • Q35 — 0.0. Gave an incorrect (1+i)^n equation. Correct: e^(iπ) + 1 = 0.
  • Q36 — 0.0. Said two groups. Correct: five groups of order 8 up to isomorphism.

devstral-small2-24b

  • Q01 — 0.5. Correct limit and white-dwarf meaning, but added neutron-star formation as the classic over-limit outcome. Correct: about 1.4–1.44 M☉; for the classic accreting white dwarf, exceeding it triggers a Type Ia supernova.
  • Q29 — 0.0. Said 92.45 dB even though its stated km/MHz formula evaluates to about 100.05 dB. Correct: about 100 dB.

devstral2-123b-iq2

  • Q01 — 0.5. Correctly gave 1.44 M☉ and the white-dwarf support limit, but presented neutron-star/black-hole formation as an alternative classic outcome. Correct classic case: Type Ia supernova.
  • Q06 — 0.5. Guessed oxygen correctly but explicitly said “Not sure.” Under the requested rule, a correct uncertain guess receives 0.5.
  • Q16 — 0.5. Correctly said Ming, but attributed the 1405–1433 expeditions to Yongle and Hongxi while omitting Xuande; the final voyage was under Xuande, while Hongxi suspended the voyages.
  • Q27 — 0.5. Included required producer Rick Rubin, but incorrectly said David Bottrill co-produced. Correct: Rubin produced and Daron Malakian co-produced.

gemma4-26b-a4b

  • Q01 — 0.5. Correct white-dwarf limit and Type Ia outcome, but also added neutron-star formation as the standard alternative. Correct classic case: Type Ia supernova.
  • Q39 — 0.5. Correctly named Daniel Kahneman but falsely said he shared the prize with Amos Tversky. Correct: Kahneman received it (with Vernon Smith receiving the other half for unrelated work); Tversky was deceased and ineligible.

glm4.6v-iq2

  • Q01 — 0.5. Correct limit/support mechanism, but said exceeding it leads to a neutron star or black hole and omitted the classic Type Ia result.
  • Q39 — 0.5. Correctly named Kahneman but said the work/prize was “shared with Amos Tversky” before noting Tversky was ineligible. Correct: only Kahneman received the Nobel recognition for this work.

glm4.7-flash

  • Q06 — 0.0. Said chlorine. Correct: oxygen.
  • Q18 — 0.0. Said Russia and Kazakhstan. Correct: Canada and the United States (including Alaska–Canada).
  • Q22 — 0.0. Initially and directly said Virgil guides Dante through Paradise, then contradicted itself by mentioning Beatrice. Correct: Beatrice; Virgil cannot enter Paradise.
  • Q27 — 0.0. Said the band self-produced and Rick Rubin was merely executive producer. Correct: Rick Rubin produced, with Daron Malakian co-producing.
  • Q28 — 0.5. Described the sixth/relative-minor relationship but did not supply the asked-for mode name. Correct: Aeolian.
  • Q29 — 0.0. Said 40.1 dB while stating the correct km/MHz formula, whose terms total about 100.05 dB. Correct: about 100 dB.
  • Q39 — 0.0. Said Kahneman and Tversky won in 2002. Correct: Daniel Kahneman; Tversky was ineligible posthumously.

gpt-oss-120b

  • Q01 — 0.5. Correctly defined the 1.44 M☉ white-dwarf support limit, but gave neutron-star/black-hole collapse instead of the classic Type Ia-supernova outcome.

gpt-oss-20b

  • Q01 — 0.5. Correctly defined the 1.44 M☉ white-dwarf limit, but gave neutron-star/black-hole collapse instead of the classic Type Ia result.
  • Q23 — 0.5. Gave only “All happy families are alike; each unhappy family is unhappy,” omitting “in its own way.” Correct full line includes that final clause.
  • Q32 — 0.0. Claimed BPSK with a 50 Hz PN/chipping code. Correct: BPSK/DSSS, 1023-chip C/A Gold code at 1.023 Mcps; 50 bps is the navigation-data rate.
  • Q39 — 0.5. Its runaway tail repeatedly paired Daniel Kahneman with Vernon Smith without cleanly distinguishing why each won. Correct answer to prospect theory is Kahneman alone; Smith’s half recognized experimental economics.
  • Q40 — 0.0. The runaway answer said it was unsure and speculated about light speed in a medium. Correct: under a fixed legal exchange value, overvalued “bad” money circulates while undervalued “good” money is hoarded/exported.

granite4-micro

  • Q12 — 0.5. Correctly named Trypanosoma cruzi and triatomine bugs, but incorrectly added mosquitoes as occasional vectors. Mosquitoes do not transmit Chagas disease.
  • Q23 — 0.5. Gave the essentially correct opening line but explicitly prefaced it with “Not sure”; per the requested uncertainty rule this is 0.5.
  • Q27 — 0.0. Said Scott Litt. Correct: Rick Rubin produced; Daron Malakian co-produced.
  • Q29 — 0.0. Said 116 dB and mixed incompatible units in the constant-based formula. Correct: about 100 dB.
  • Q32 — 0.5. Correctly identified DSSS and the 1.023 MHz PN code, but omitted the asked-for carrier modulation. Correct: BPSK plus DSSS/Gold-code spreading.
  • Q40 — 0.0. Reversed/garbled the consequence, saying the more intrinsically valuable currency is devalued and replaced by “better quality” currency. Correct: bad money circulates and good money is hoarded/exported when law fixes them at the same face value.

minimax-m2-iq2

  • Q01 — 0.5. Correctly defined the 1.44 M☉ white-dwarf limit but added neutron-star collapse rather than identifying the classic Type Ia outcome cleanly.
  • Q06 — 0.0. The truncated reasoning compared electronegativities but emitted no answer. Correct: oxygen.
  • Q27 — 0.0. Said only Daron Malakian. Full credit requires Rick Rubin: Rubin produced and Malakian co-produced.
  • Q36 — 0.0. The truncated reasoning never supplied the number and ended while incorrectly wondering whether there were three nonabelian groups. Correct: five total (three abelian, two nonabelian).

ministral3-14b

  • Q01 — 0.5. Correct limit and white-dwarf subject, but stated neutron-star/black-hole collapse instead of the classic Type Ia outcome.
  • Q29 — 0.0. Said approximately 90 dB. Correct: approximately 100 dB (the 10 dB miss is material here).
  • Q36 — 0.5. Correctly said five, which answers the question, but its supporting list substituted the order-4 Klein group and a dicyclic group for the actual C4×C2 and C2×C2×C2 entries.

nemotron-cascade-30b

  • Q01 — 0.5. Correct limit and white-dwarf subject, but gave neutron-star/black-hole formation and omitted the classic Type Ia outcome.
  • Q27 — 0.0. Said the band/Daron Malakian/Shavo Odadjian produced it and omitted Rick Rubin. Correct: Rick Rubin produced; Daron Malakian co-produced.
  • Q29 — 0.0. Said 92 dB while its own km/MHz formula evaluates to about 100 dB.
  • Q32 — 0.5. Correctly gave DSSS, Gold code, and chip rate, but omitted the asked-for BPSK modulation.
  • Q39 — 0.5. Guessed Daniel Kahneman correctly but began “not sure”; per the requested rule, a correct uncertain guess earns 0.5.

nemotron3-nano-30b

  • Q09 — 0.5. Correctly gave phrenic nerve C3–C5 but added an “occasional C6” contribution, a minor anatomical inaccuracy for the standard answer.
  • Q23 — 0.0. Gave Dickens’s “It was the best of times...” and said “not sure.” Correct: “All happy families are alike; each unhappy family is unhappy in its own way.”
  • Q25 — 0.0. Correct album title but said 1982. Correct: Iron Maiden, 1980.
  • Q29 — 0.0. Said 112 dB and mangled units in the constant form. Correct: about 100 dB.
  • Q32 — 0.5. Correctly gave BPSK, the 1023-chip code, and 1.023 MHz, but incorrectly claimed 1023 chips per navigation-data bit; at 50 bps there are 20,460 chips per bit.

nemotron3-super-120b

  • Q01 — 0.5. Correctly defined the white-dwarf limit but added neutron-star formation rather than cleanly stating the classic Type Ia outcome.

north-mini-code

  • Q01 — 0.5. Correctly gave the limit and Type Ia possibility, but also presented neutron-star collapse as the alternative classic white-dwarf outcome.

qwen3.6-27b

  • Q01 — 0.5. Correctly defined the limit and mentioned Type Ia, but also added neutron-star formation for the classic white-dwarf case.
  • Q10 — 0.5. Correct core function (medullary gradient/urine concentration), but incorrectly said the descending limb prevents water reabsorption; it is water-permeable, while the ascending limb is water-impermeable.
  • Q29 — 0.0. Said 92.45 dB even though the supplied km/MHz formula evaluates to about 100.05 dB.

qwen3.6-35b-a3b

  • Q01 — 0.5. Correctly defined the limit but added neutron-star formation instead of identifying the classic Type Ia outcome cleanly.
  • Q19 — 0.5. Correctly named the Bering Strait, but reversed its adjoining seas: the Bering Sea is south and the Chukchi Sea north.
  • Q29 — 0.0. Said about 92 dB despite using metre/Hz variables in the full formula; correct substitution gives about 100 dB.
  • Q40 — 0.5. Correctly explained hoarding higher-intrinsic-value money, then reversed the valuation labels in its conclusion. Correct: overvalued bad money circulates; undervalued good money disappears from circulation.

qwen3next-80b-a3b

  • Q01 — 0.5. Correctly defined the white-dwarf limit but added neutron-star formation rather than cleanly giving the classic Type Ia outcome.

Best overall: gemma4-31b — 40.0/40.

Best knowledge-per-GB: granite4-micro — 35.5/40 from an approximately 2.0 GB model file, by far the strongest score per stored GB; for a higher absolute-score efficiency option, gpt-oss-20b scores 36.5/40 at about 10.9 GiB.