experiments-07-07-12-local-llm-knowledge-2/SCORES_AGENTIC.md

Multi-Turn Agentic Benchmark

Objective outcomes below come directly from the harness. Interpretation emphasizes clean termination and tool-call validity over raw capability.

Outcome table

Sorted by tasks solved, clean terminations, then tool discipline/turn efficiency.

Rank	Model	Solved /5	Clean terminations /5	Total turns	Bad calls	Bad-call rate
1	qwen3.5-9b	5	5	18	0	0%
2	north-mini-code	5	5	21	0	0%
3	qwen3.6-27b	5	5	19	1	5.3%
4	devstral-small2-24b	5	5	20	1	5.0%
5	gpt-oss-120b	5	5	24	4	16.7%
6	gpt-oss-20b	0	0	20	20	100%

All five successful models called done on the turn immediately following the final successful verification. No model solved a task and then failed to terminate cleanly in this run. GPT-OSS-20B never reached task execution at all.

Termination behavior

Model	How tasks ended	Swarm interpretation
qwen3.5-9b	done on turns 3/4/5/3/3, always immediately after a successful run. No caps or execution errors.	Excellent bounded-loop termination. This sharply contrasts with its warm/open-ended single-shot runaways.
north-mini-code	done on turns 3/5/7/3/3, immediately after verification. It used two extra inspection turns in t3 (list_dir, then read_file) before continuing, but did not work after success.	Clean and cautious, not runaway. Its prior ambiguity-driven main-role failure did not recur across tool trajectories.
qwen3.6-27b	done on turns 3/4/5/4/3. Recovered from one malformed t4 call, then verified output and stopped.	Strong termination with a small schema-risk blemish.
devstral-small2-24b	done on turns 4/4/6/3/3. One initial malformed t1 call; the rest ended promptly. An extra read_file(nums.txt) in t3 was useful verification, not post-success churn.	Reliable bounded-loop behavior with one recoverable serialization error.
gpt-oss-120b	done on turns 4/5/5/7/3. It eventually stopped correctly every time, but t4 burned turns 1–3 on malformed writes before recovering.	Correct termination, weaker protocol efficiency. Needs strict bad-call limits despite high intelligence.
gpt-oss-20b	Every task emitted four empty raw responses, hit the harness bad-call threshold at turn 4, and terminated as error, never done.	Hard unfit for this agent template. The harness—not the model—terminated the loops safely.

Specific failure-turn evidence:

• GPT-OSS-20B, every task turns 1–4: raw output was empty, followed by BAD CALL: no valid tool JSON found; turn 4 ended with too many bad calls, aborting task.
• GPT-OSS-120B t1 turn 1: an incomplete/truncated write_file JSON was rejected; turn 2 retried successfully.
• GPT-OSS-120B t4 turns 1–3: three successive incomplete write_file calls were rejected. On turn 4 it explicitly reset—We need to respond with a valid tool call JSON—then listed the directory, emitted a shorter valid write on turn 5, ran it on turn 6, and called done on turn 7.
• Qwen3.6-27B t4 turn 1: incomplete/truncated write_file JSON; turn 2 used a compact JSON form successfully.
• Devstral-Small2 t1 turn 1: incomplete/truncated write_file JSON; turn 2 retried successfully.

There were no turn-cap terminations among the five working agents and no case where an already-solved task continued consuming turns after success.

Recovery behavior

t2: inspect, diagnose, fix, verify

Model	Behavior
qwen3.5-9b	Read buggy.py, correctly inverted both returns, ran tests successfully, then called done in 4 turns. Best direct recovery path.
qwen3.6-27b	Same clean read → fix → test → done path in 4 turns. Diagnosis in done accurately described the inversion.
devstral-small2-24b	Same direct 4-turn recovery; no wasted actions.
north-mini-code	Added a harmless list_dir before reading, then correctly fixed, tested, and stopped in 5 turns.
gpt-oss-120b	Also listed before reading, then fixed and verified in 5 turns. Correct but slightly less efficient.
gpt-oss-20b	Never emitted a tool call; did not read the file or attempt diagnosis.

t4: debug/verification loop

The task allowed a fix-and-rerun loop, but the five capable models all wrote correct population-standard-deviation code on their first executable attempt. The differentiator was therefore call recovery rather than algorithm repair:

• North Mini, Qwen3.5, and Devstral Small: write → run (5.0, 2.0) → done in 3 turns.
• Qwen3.6: first write was malformed; it compacted/re-emitted the call, ran successfully, and stopped in 4 turns.
• GPT-OSS-120B: repeated the same oversized/incomplete write three times, then changed strategy, inspected state, emitted a shorter valid call, ran, and stopped. It recovered correctly but flailed before doing so.
• GPT-OSS-20B: four empty calls; no recovery attempt.

This benchmark did not force the successful models through an actual failing Python execution followed by a semantic code patch. It establishes tool-call recovery and verification discipline, but a future benchmark should plant a runtime/test failure after the first patch to separate genuine debug-loop skill.

Tool-use discipline

Model	Counted discipline	Raw-protocol observations
qwen3.5-9b	0/18 bad	Harness accepted every call. One t3 raw response had trailing </think> after the JSON, so the emitter was not perfectly pure even though extraction succeeded.
north-mini-code	0/21 bad	All calls executed. t5 turn 2 showed an extra closing brace (..."parse.py"}} }), tolerated by the harness parser. Very good practical discipline, but not byte-strict JSON.
qwen3.6-27b	1/19 bad	One truncated JSON write; immediate successful compact retry. Otherwise clean.
devstral-small2-24b	1/20 bad	One truncated first write; immediate recovery and no repeats.
gpt-oss-120b	4/24 bad	Repeated truncated/oversized JSON, especially t4. Several accepted raw turns also prefixed prose such as We need to output JSON tool call. The harness extracted the object, but a strict protocol endpoint might reject it.
gpt-oss-20b	20/20 bad	Empty raw output, not prose, wrong schema, or hallucinated tools: it emitted nothing parseable at all.

No model hallucinated a nonexistent tool name in the logged calls. The dominant malformed-call mode was truncated JSON during long write_file payloads, suggesting generation length/template/stop handling rather than misunderstanding of the tool catalog.

The harness appears permissive—it can extract a JSON object despite prose prefixes, trailing </think>, or an extra brace. For Hermes/Claw, decide whether production parsing is equally tolerant. If not, GPT-OSS-120B, Qwen3.5, and North need stricter serialization prompting or a repair layer even though their counted outcomes were successful.

Per-model autonomous-cluster verdicts

Model	Autonomous verdict	Evidence and required contract
qwen3.5-9b	Yes-with-contract	Best trajectory numbers: 5/5, 5/5 clean, 18 turns, 0 bad. However, prior warm/open prompts produced multiple no-final runaways. Restrict it to concrete tool tasks, cap turns/time, and reject non-JSON residue.
north-mini-code	Yes (bounded loops)	5/5, 5/5 clean, 0 bad; correct recovery and prompt stopping. Use as the primary worker with turn cap, objective test gate, JSON normalization, and fallback on missing done.
qwen3.6-27b	Yes-with-contract	5/5 and all clean, with one malformed write. Safe as orchestrator for bounded loops when bad calls are retried once and then escalated.
devstral-small2-24b	Yes-with-contract	5/5 and all clean, one immediately recovered bad call. Good overflow worker under the same one-retry/schema contract.
gpt-oss-120b	Yes-with-contract	Solved and terminated all tasks, but 4 bad calls/24 and three repeated failures in t4 are too noisy for unconstrained swarm fan-out. Abort/escalate after two consecutive bad calls; shorten file payloads.
gpt-oss-20b	No	0/5, 0 clean, 20/20 bad. It cannot drive this tool protocol regardless of its general reasoning ability. Do not deploy until the chat template/tool-call configuration is fixed and re-benchmarked.

Containment contracts

Autonomous cluster (Hermes/Claw)

Model	Recommended contract
North Mini	12-turn task cap; one wall-clock timeout; validate/extract one JSON object; force done immediately after objective success; abort on two consecutive bad calls; tests are authoritative.
Qwen3.5-9B	10-turn cap; narrow task schema only; repetition/no-output watchdog; strict final JSON cleanup; never promote it to open-ended planner based on worker success.
Qwen3.6-27B	12-turn cap; retry one malformed call with a compact-json reminder; orchestrator must check task success before accepting done; escalate repeated serialization failure.
Devstral Small	12-turn cap; one compact retry for truncated writes; test gate; use as overflow/parallel worker.
GPT-OSS-120B	Reviewer tasks preferred; two-consecutive-bad-call abort; prohibit prose outside JSON; encourage smaller patches/tool payloads; 16-turn hard cap and wall-clock timeout.
GPT-OSS-20B	Disabled for tools. A model-specific template/configuration fix is prerequisite, followed by a fresh protocol smoke test.

Cluster-wide safeguards:

1. Enforce a hard turn and wall-clock cap per task.
2. Abort or switch model after two consecutive malformed calls.
3. Parse exactly one allowed tool call and reject unknown tools/arguments.
4. Treat objective tests/state as success, not the model's claim.
5. Once success is observed, require the next call to be done; synthesize termination if necessary.
6. Detect repeated identical writes/calls and force a read/inspect or escalation.
7. Keep tool sandboxes, file scopes, and destructive permissions bounded.

General-purpose role (separate from autonomous safety)

Model	General/chat/coding assessment from combined rounds
qwen3.6-27b	Still the best general default: perfect role-main score, excellent knowledge/reasoning, premise resistance, and now strong multi-turn execution.
gpt-oss-120b	Still the strongest escalation/reviewer and a very capable general/coding model. Its JSON serialization inefficiency is an agent-protocol issue, not a reason to discard it from chat/review.
north-mini-code	Excellent coding and technical model, now validated in multi-turn bounded work. Still not the preferred general chat model because ambiguity/open-ended prompts have triggered a prior no-final runaway.
devstral-small2-24b	Better agentic worker than its main-role score suggests. Useful coding/worker backup; merely average as a broad daily assistant due to format/recall misses.
qwen3.5-9b	Poor main/chat under the tested warm template but excellent narrow agent worker. This is the clearest role inversion: cheap and effective for concrete tool loops, unsafe for broad conversational orchestration.
gpt-oss-20b	Potentially useful general/chat/business-coding model, but not an autonomous agent with this template. It scored 7.5/8 as a role-main and performed well in prior knowledge/reasoning, while emitting zero tool JSON here. Keep it for non-tool applications pending configuration repair.

Do not conflate protocol compatibility with intelligence. GPT-OSS-20B's total failure is likely template/tool-mode incompatibility; Qwen3.5's success is highly task-shape dependent. Both facts matter operationally.

Topology confirmation and revision

Working topology

Qwen3.6-27B       -> main/general driver and bounded orchestrator
GPT-OSS-120B      -> escalation/reviewer; tool use behind stricter call guard
North-Mini-Code   -> primary bounded worker
Devstral-Small2   -> worker backup / overflow pool
Qwen3.5-9B        -> optional cheap narrow worker tier
GPT-OSS-20B       -> general/chat only; tools disabled pending fix

Findings by candidate

• North Mini survives multi-turn. Its single-shot worker strength did not merely survive—it translated into 5/5 solved, 5/5 clean termination, and zero counted bad calls. The prior runaway tendency did not appear over 21 agent turns. Keep it as the primary bounded worker and fast technical checker. Do not infer that it is safe as an open-ended main.
• Qwen3.6-27B is confirmed as main. It combines the strongest prior main/general evidence with 5/5 agentic success. One malformed write justifies a retry contract, not a topology change.
• GPT-OSS-120B remains escalation, with a caveat. It solved everything, but its 16.7% bad-call rate and repeated t4 serialization failures make it less attractive as a high-volume worker. Its value is judgment/review, not cheap tool throughput.
• Devstral Small is confirmed as backup worker. It recovered immediately from its one malformed call and otherwise followed efficient trajectories.
• Qwen3.5-9B earns a new narrow role. It is unsafe as main/general under warm open prompts but was the most turn-efficient perfect agent here. Add it only as a cheap, tightly bounded worker with watchdogs.
• GPT-OSS-20B does not hold up in the loop. It remains useful for chat and ordinary coding, but autonomous tool use must be disabled until its template/config is corrected.

Final synthesis

For autonomous Hermes/Claw clusters, the preferred order is:

1. North-Mini-Code — primary bounded implementer.
2. Qwen3.6-27B — orchestrator/general technical agent.
3. Devstral-Small2-24B — worker backup.
4. Qwen3.5-9B — inexpensive narrow worker tier, never open-ended planner.
5. GPT-OSS-120B — independent reviewer/escalation behind strict JSON guards.
6. GPT-OSS-20B — disabled for tools.

For general-purpose chat, Pi, knowledge work, and business/general coding, keep the prior order conceptually separate: Qwen3.6-27B remains the default, GPT-OSS-120B the deep reviewer, North Mini the coding specialist, and GPT-OSS-20B remains viable as a smaller conversational/coding model despite being agentically unfit here.

The original four-model topology is therefore confirmed, not overturned. The only revisions are to add Qwen3.5-9B as an optional cheap worker and to put a stricter serialization guard around GPT-OSS-120B. The benchmark's strongest practical result is that North Mini's opportunity is real: bounded, verified, multi-turn implementation is exactly where it should be deployed.