11 KiB

Raw Blame History

Local LLM Role and Agent Benchmark

This project evaluates local models for two related but distinct uses:

General-purpose chat, knowledge work, writing, and technical/coding assistance.
Semi-autonomous agent clusters for RF/SDR, signal processing, embedded systems, and general software work.

The central lesson is that there is no single meaningful “best model.” Knowledge, reasoning, instruction-following, tool protocol compatibility, and autonomous termination are separate capabilities. The best deployment is a role-specialized model portfolio.

Recommendation

Qwen3.6-27B       -> main assistant and bounded orchestrator
GPT-OSS-120B      -> independent escalation and high-consequence review
North-Mini-Code   -> primary bounded implementation worker
Devstral-Small2   -> worker backup and overflow capacity
Gemma4-31B        -> knowledge, writing, and synthesis specialist
Qwen3.5-9B        -> optional cheap micro-worker tier
GPT-OSS-20B       -> general/chat coding only until tool protocol is fixed

If only one model is deployed, use Qwen3.6-27B. It showed the best combined evidence across main-role behavior, technical reasoning, premise resistance, formatting, and multi-turn tool use.

For a minimal autonomous cluster, use:

Qwen3.6-27B as planner/orchestrator.
North-Mini-Code as the implementation worker.
GPT-OSS-120B as an independent reviewer for consequential work.

Gemma4-31B may have the strongest broad stored-knowledge and writing profile, but it was not included in the real multi-turn tool benchmark. It is therefore recommended as a general-purpose specialist rather than the proven orchestrator.

Harness and agent strategy

Route by task shape

Task	Preferred model
Conversation, planning, broad technical work	Qwen3.6-27B
Difficult review, assumption checking, risky decisions	GPT-OSS-120B
Small functions, targeted patches, test repair	North-Mini-Code
Parallel worker overflow	Devstral-Small2
Reports, documentation, explanations, brainstorming	Gemma4-31B
Cheap, concrete, mechanically verified microtasks	Qwen3.5-9B

North Mini should be treated as the hands, not the executive. It is excellent when given a named file, explicit acceptance criteria, objective tests, and a natural stopping condition. It should not independently own open-ended research, recursive planning, or final approval for destructive changes.

Required control contract

Every autonomous worker should run behind an external controller that:

Enforces a hard turn cap and wall-clock timeout.
Aborts or changes models after two consecutive malformed tool calls.
Extracts and validates exactly one allowed JSON tool call.
Validates the tool name, argument schema, permissions, and file scope.
Uses tests and observable state—not the model's claim—as ground truth.
Requires done immediately after objective success, or terminates automatically.
Detects repeated calls, repeated writes, and no-progress loops.
Escalates consequential patches to an independent model family.

A practical flow is:

Qwen frames task and assumptions
              |
              v
North/Devstral implements in sandbox
              |
              v
tests and state verify result
              |
              v
GPT-OSS-120B reviews high-risk changes

The production parser should normalize model output before execution. Some successful trajectories contained prose before JSON, a trailing </think>, or an extra brace that the benchmark harness tolerated. Zero counted bad calls therefore does not always mean byte-perfect JSON.

Evaluation stages

1. Broad knowledge benchmark

The first round asked 40 questions across physics, chemistry, biology, history, geography, literature, music, RF engineering, mathematics, and economics/law. Answers were manually checked rather than trusted at face value.

This round measured factual breadth but did not separate the leaders strongly: many models scored between 38 and 40.

Key results:

Gemma4-31B: 40/40, the only perfect broad-knowledge result.
GPT-OSS-120B, Nemotron3-Super, North Mini, Qwen3-Next: 39.5/40.
Granite4-Micro: impressive 35.5/40 for roughly a 2 GB file, but too unreliable for unverified engineering work.
Aya was weak, particularly on RF and mathematics.

The round exposed common fluent errors involving the Chandrasekhar limit, RF path-loss arithmetic, GPS modulation, music production credits, and the groups of order eight.

Full report: SCORES.md

2. Hard recall, reasoning, and false-premise benchmark

The second round used 24 questions divided into recall, numerical reasoning, and traps. Trap questions tested whether models would reject false premises instead of politely accepting them.

This round produced more useful behavioral evidence:

Eight models scored 24/24, including Gemma4-31B, GPT-OSS-120B, Qwen3.6-27B, and Nemotron-Cascade.
Qwen3.6-27B correctly identified the real 5G NR allocation of 51 resource blocks / 612 subcarriers, catching an error in the benchmark's own expected answer.
Several otherwise capable models confused geostationary altitude with geocentric orbital radius.
MiniMax repeatedly exhausted its token budget without a final answer.
Aya combined weak reasoning with confident, padded errors.

The major conclusion was that premise resistance and clean reasoning matter more for engineering than small differences in factual recall.

Full report: SCORES_HARD.md

3. Role-fitness and single-turn worker benchmark

The third round explicitly tested three deployment roles:

Main: calibration, strict formats, concise writing, and general helpfulness.
Escalation: detecting planted technical, software, security, and mathematical errors.
Worker: eight live-executed code/format tasks graded by objective tests.

Key results:

Qwen3.6-27B: the only 8/8 main and also 6/6 escalation.
GPT-OSS-120B: 6/6 escalation and the preferred independent reviewer.
North Mini: 27/27 worker tests and 6/6 escalation, but one 6144-token empty-final runaway on an ambiguous main question.
Devstral Small: 27/27 worker tests and 6/6 escalation, validating it as a reliable backup worker.
Qwen3.5-9B: 27/27 worker tests but multiple runaways on warm/open main prompts.
MiniMax: larger token limits did not solve its termination problem.

This established that single-shot worker strength can coexist with poor open-ended control. It also supplied the first strong evidence for using North Mini in a bounded role.

Full report: SCORES_ROLES.md

4. Multi-turn agentic benchmark

The final round placed six models in real tool loops. Each attempted five tasks using JSON tool calls, sandboxed file operations, Python execution, feedback, retries, and an explicit done call.

Model	Solved	Clean terminations	Turns	Bad calls
Qwen3.5-9B	5/5	5/5	18	0
North-Mini-Code	5/5	5/5	21	0
Qwen3.6-27B	5/5	5/5	19	1
Devstral-Small2	5/5	5/5	20	1
GPT-OSS-120B	5/5	5/5	24	4
GPT-OSS-20B	0/5	0/5	20	20

Important conclusions:

North Mini's worker strength survived multi-turn execution. It did not reproduce its earlier runaway across 21 agent turns, and it terminated immediately after verified success.
Qwen3.6-27B was confirmed as the orchestrator: full task success, clean termination, and immediate recovery from one malformed write.
Qwen3.5-9B was the most turn-efficient worker, but its earlier open-ended runaways mean it must remain narrowly routed and externally controlled.
GPT-OSS-120B solved everything but had weaker JSON discipline, including three consecutive malformed writes on one task. It remains more valuable as reviewer than high-volume worker.
GPT-OSS-20B emitted no parseable tool call on any turn. This looks like a template/tool-mode incompatibility rather than proof of low intelligence. It remains useful for non-tool chat/coding but is disabled for autonomous tools until reconfigured and retested.

No successful model kept working after task success or failed to call done. The benchmark validated basic recovery and verification, although it did not force most models to repair a semantic Python failure after an unsuccessful execution. A future round should use existing multi-file projects with planted failing tests.

Full report: SCORES_AGENTIC.md

What each leading model is for

Qwen3.6-27B

Best all-purpose default. It combines broad knowledge, strong technical reasoning, strict instruction-following, premise resistance, and proven tool-loop behavior. It can fill both main and escalation roles in a small deployment, although independent review is safer.

GPT-OSS-120B

Best used for high-consequence review, architecture, difficult debugging, and checking worker output. Its intelligence and reasoning are strong, but its malformed-call rate makes it inefficient as a swarm of file-writing workers.

North-Mini-Code

Primary bounded worker and fast technical checker. Give it small file scopes, explicit tests, and automatic termination. Its excellent coding/RF performance is real; its rare ambiguity-driven runaway means it should not be the general planner.

Gemma4-31B

Best broad-knowledge and writing specialist in the tested set. It also reasoned very well and passed every single-turn worker test. Multi-turn tool behavior remains untested, so it should not replace Qwen as orchestrator without completing the same agentic loop.

Devstral-Small2

Dependable worker backup. It is less compelling as a main assistant, but its live code execution, escalation, and multi-turn behavior were clean.

Qwen3.5-9B

Promising inexpensive micro-worker. It excels when objectives and completion conditions are concrete, but is unfit for warm/open-ended planning under the tested template.

Deprioritized configurations

MiniMax-M2-IQ2: unfit for autonomous use because repeated thinking loops produce empty finals even with larger token budgets.
Aya-23-8B: too many confident specialist and numerical errors.
Granite4-Micro: useful only when its tiny footprint dominates; not trusted for unverified technical work.
GPT-OSS-20B tool mode: disabled until its template/parser incompatibility is corrected. This does not exclude it from general chat or coding.

Bottom line

The recommended architecture is role-specialized rather than winner-take-all:

Qwen is the brain and general interface.
North is the primary pair of hands.
GPT-OSS-120B is the skeptical independent reviewer.
Devstral Small provides worker diversity and overflow.
Gemma provides deep knowledge, writing, and synthesis.
Qwen3.5 offers a cheap, tightly bounded worker tier.

Clean termination, objective verification, and tool-schema enforcement belong in the harness—not in assumptions about model intelligence. That control plane is what turns capable local models into a safe and useful autonomous cluster.

11 KiB Raw Blame History