experiments-07-07-12-local-llm-knowledge-2/SCORES_AGENTIC.md

# Multi-Turn Agentic Benchmark

Objective outcomes below come directly from the harness. Interpretation emphasizes clean termination and tool-call validity over raw capability.

## Outcome table

Sorted by tasks solved, clean terminations, then tool discipline/turn efficiency.

| Rank | Model | Solved /5 | Clean terminations /5 | Total turns | Bad calls | Bad-call rate |
|---:|---|---:|---:|---:|---:|---:|
| 1 | qwen3.5-9b | **5** | **5** | **18** | **0** | 0% |
| 2 | north-mini-code | **5** | **5** | 21 | **0** | 0% |
| 3 | qwen3.6-27b | **5** | **5** | 19 | 1 | 5.3% |
| 4 | devstral-small2-24b | **5** | **5** | 20 | 1 | 5.0% |
| 5 | gpt-oss-120b | **5** | **5** | 24 | 4 | 16.7% |
| 6 | gpt-oss-20b | 0 | 0 | 20 | 20 | **100%** |

All five successful models called `done` on the turn immediately following the final successful verification. **No model solved a task and then failed to terminate cleanly** in this run. GPT-OSS-20B never reached task execution at all.

## Termination behavior

| Model | How tasks ended | Swarm interpretation |
|---|---|---|
| qwen3.5-9b | `done` on turns 3/4/5/3/3, always immediately after a successful run. No caps or execution errors. | Excellent bounded-loop termination. This sharply contrasts with its warm/open-ended single-shot runaways. |
| north-mini-code | `done` on turns 3/5/7/3/3, immediately after verification. It used two extra inspection turns in t3 (`list_dir`, then `read_file`) before continuing, but did not work after success. | Clean and cautious, not runaway. Its prior ambiguity-driven main-role failure did **not** recur across tool trajectories. |
| qwen3.6-27b | `done` on turns 3/4/5/4/3. Recovered from one malformed t4 call, then verified output and stopped. | Strong termination with a small schema-risk blemish. |
| devstral-small2-24b | `done` on turns 4/4/6/3/3. One initial malformed t1 call; the rest ended promptly. An extra `read_file(nums.txt)` in t3 was useful verification, not post-success churn. | Reliable bounded-loop behavior with one recoverable serialization error. |
| gpt-oss-120b | `done` on turns 4/5/5/7/3. It eventually stopped correctly every time, but t4 burned turns 1–3 on malformed writes before recovering. | Correct termination, weaker protocol efficiency. Needs strict bad-call limits despite high intelligence. |
| gpt-oss-20b | Every task emitted four empty raw responses, hit the harness bad-call threshold at turn 4, and terminated as `error`, never `done`. | Hard unfit for this agent template. The harness—not the model—terminated the loops safely. |

Specific failure-turn evidence:

- **GPT-OSS-20B, every task turns 1–4:** raw output was empty, followed by `BAD CALL: no valid tool JSON found`; turn 4 ended with `too many bad calls, aborting task`.
- **GPT-OSS-120B t1 turn 1:** an incomplete/truncated `write_file` JSON was rejected; turn 2 retried successfully.
- **GPT-OSS-120B t4 turns 1–3:** three successive incomplete `write_file` calls were rejected. On turn 4 it explicitly reset—`We need to respond with a valid tool call JSON`—then listed the directory, emitted a shorter valid write on turn 5, ran it on turn 6, and called `done` on turn 7.
- **Qwen3.6-27B t4 turn 1:** incomplete/truncated `write_file` JSON; turn 2 used a compact JSON form successfully.
- **Devstral-Small2 t1 turn 1:** incomplete/truncated `write_file` JSON; turn 2 retried successfully.

There were no turn-cap terminations among the five working agents and no case where an already-solved task continued consuming turns after success.

## Recovery behavior

### t2: inspect, diagnose, fix, verify

| Model | Behavior |
|---|---|
| qwen3.5-9b | Read `buggy.py`, correctly inverted both returns, ran tests successfully, then called `done` in 4 turns. Best direct recovery path. |
| qwen3.6-27b | Same clean read → fix → test → done path in 4 turns. Diagnosis in `done` accurately described the inversion. |
| devstral-small2-24b | Same direct 4-turn recovery; no wasted actions. |
| north-mini-code | Added a harmless `list_dir` before reading, then correctly fixed, tested, and stopped in 5 turns. |
| gpt-oss-120b | Also listed before reading, then fixed and verified in 5 turns. Correct but slightly less efficient. |
| gpt-oss-20b | Never emitted a tool call; did not read the file or attempt diagnosis. |

### t4: debug/verification loop

The task allowed a fix-and-rerun loop, but the five capable models all wrote correct population-standard-deviation code on their first executable attempt. The differentiator was therefore call recovery rather than algorithm repair:

- **North Mini, Qwen3.5, and Devstral Small:** write → run `(5.0, 2.0)` → `done` in 3 turns.
- **Qwen3.6:** first write was malformed; it compacted/re-emitted the call, ran successfully, and stopped in 4 turns.
- **GPT-OSS-120B:** repeated the same oversized/incomplete write three times, then changed strategy, inspected state, emitted a shorter valid call, ran, and stopped. It recovered correctly but flailed before doing so.
- **GPT-OSS-20B:** four empty calls; no recovery attempt.

This benchmark did not force the successful models through an actual failing Python execution followed by a semantic code patch. It establishes tool-call recovery and verification discipline, but a future benchmark should plant a runtime/test failure after the first patch to separate genuine debug-loop skill.

## Tool-use discipline

| Model | Counted discipline | Raw-protocol observations |
|---|---|---|
| qwen3.5-9b | 0/18 bad | Harness accepted every call. One t3 raw response had trailing `</think>` after the JSON, so the emitter was not perfectly pure even though extraction succeeded. |
| north-mini-code | 0/21 bad | All calls executed. t5 turn 2 showed an extra closing brace (`..."parse.py"}} }`), tolerated by the harness parser. Very good practical discipline, but not byte-strict JSON. |
| qwen3.6-27b | 1/19 bad | One truncated JSON write; immediate successful compact retry. Otherwise clean. |
| devstral-small2-24b | 1/20 bad | One truncated first write; immediate recovery and no repeats. |
| gpt-oss-120b | 4/24 bad | Repeated truncated/oversized JSON, especially t4. Several accepted raw turns also prefixed prose such as `We need to output JSON tool call.` The harness extracted the object, but a strict protocol endpoint might reject it. |
| gpt-oss-20b | 20/20 bad | Empty raw output, not prose, wrong schema, or hallucinated tools: it emitted **nothing parseable at all**. |

No model hallucinated a nonexistent tool name in the logged calls. The dominant malformed-call mode was **truncated JSON during long `write_file` payloads**, suggesting generation length/template/stop handling rather than misunderstanding of the tool catalog.

The harness appears permissive—it can extract a JSON object despite prose prefixes, trailing `</think>`, or an extra brace. For Hermes/Claw, decide whether production parsing is equally tolerant. If not, GPT-OSS-120B, Qwen3.5, and North need stricter serialization prompting or a repair layer even though their counted outcomes were successful.

## Per-model autonomous-cluster verdicts

| Model | Autonomous verdict | Evidence and required contract |
|---|---|---|
| qwen3.5-9b | **Yes-with-contract** | Best trajectory numbers: 5/5, 5/5 clean, 18 turns, 0 bad. However, prior warm/open prompts produced multiple no-final runaways. Restrict it to concrete tool tasks, cap turns/time, and reject non-JSON residue. |
| north-mini-code | **Yes** (bounded loops) | 5/5, 5/5 clean, 0 bad; correct recovery and prompt stopping. Use as the primary worker with turn cap, objective test gate, JSON normalization, and fallback on missing `done`. |
| qwen3.6-27b | **Yes-with-contract** | 5/5 and all clean, with one malformed write. Safe as orchestrator for bounded loops when bad calls are retried once and then escalated. |
| devstral-small2-24b | **Yes-with-contract** | 5/5 and all clean, one immediately recovered bad call. Good overflow worker under the same one-retry/schema contract. |
| gpt-oss-120b | **Yes-with-contract** | Solved and terminated all tasks, but 4 bad calls/24 and three repeated failures in t4 are too noisy for unconstrained swarm fan-out. Abort/escalate after two consecutive bad calls; shorten file payloads. |
| gpt-oss-20b | **No** | 0/5, 0 clean, 20/20 bad. It cannot drive this tool protocol regardless of its general reasoning ability. Do not deploy until the chat template/tool-call configuration is fixed and re-benchmarked. |

## Containment contracts

### Autonomous cluster (Hermes/Claw)

| Model | Recommended contract |
|---|---|
| North Mini | 12-turn task cap; one wall-clock timeout; validate/extract one JSON object; force `done` immediately after objective success; abort on two consecutive bad calls; tests are authoritative. |
| Qwen3.5-9B | 10-turn cap; narrow task schema only; repetition/no-output watchdog; strict final JSON cleanup; never promote it to open-ended planner based on worker success. |
| Qwen3.6-27B | 12-turn cap; retry one malformed call with a compact-json reminder; orchestrator must check task success before accepting `done`; escalate repeated serialization failure. |
| Devstral Small | 12-turn cap; one compact retry for truncated writes; test gate; use as overflow/parallel worker. |
| GPT-OSS-120B | Reviewer tasks preferred; two-consecutive-bad-call abort; prohibit prose outside JSON; encourage smaller patches/tool payloads; 16-turn hard cap and wall-clock timeout. |
| GPT-OSS-20B | Disabled for tools. A model-specific template/configuration fix is prerequisite, followed by a fresh protocol smoke test. |

Cluster-wide safeguards:

1. Enforce a hard turn and wall-clock cap per task.
2. Abort or switch model after two consecutive malformed calls.
3. Parse exactly one allowed tool call and reject unknown tools/arguments.
4. Treat objective tests/state as success, not the model's claim.
5. Once success is observed, require the next call to be `done`; synthesize termination if necessary.
6. Detect repeated identical writes/calls and force a read/inspect or escalation.
7. Keep tool sandboxes, file scopes, and destructive permissions bounded.

## General-purpose role (separate from autonomous safety)

| Model | General/chat/coding assessment from combined rounds |
|---|---|
| qwen3.6-27b | Still the best general default: perfect role-main score, excellent knowledge/reasoning, premise resistance, and now strong multi-turn execution. |
| gpt-oss-120b | Still the strongest escalation/reviewer and a very capable general/coding model. Its JSON serialization inefficiency is an agent-protocol issue, not a reason to discard it from chat/review. |
| north-mini-code | Excellent coding and technical model, now validated in multi-turn bounded work. Still not the preferred general chat model because ambiguity/open-ended prompts have triggered a prior no-final runaway. |
| devstral-small2-24b | Better agentic worker than its main-role score suggests. Useful coding/worker backup; merely average as a broad daily assistant due to format/recall misses. |
| qwen3.5-9b | **Poor main/chat under the tested warm template but excellent narrow agent worker.** This is the clearest role inversion: cheap and effective for concrete tool loops, unsafe for broad conversational orchestration. |
| gpt-oss-20b | **Potentially useful general/chat/business-coding model, but not an autonomous agent with this template.** It scored 7.5/8 as a role-main and performed well in prior knowledge/reasoning, while emitting zero tool JSON here. Keep it for non-tool applications pending configuration repair. |

Do not conflate protocol compatibility with intelligence. GPT-OSS-20B's total failure is likely template/tool-mode incompatibility; Qwen3.5's success is highly task-shape dependent. Both facts matter operationally.

## Topology confirmation and revision

### Working topology

```text
Qwen3.6-27B       -> main/general driver and bounded orchestrator
GPT-OSS-120B      -> escalation/reviewer; tool use behind stricter call guard
North-Mini-Code   -> primary bounded worker
Devstral-Small2   -> worker backup / overflow pool
Qwen3.5-9B        -> optional cheap narrow worker tier
GPT-OSS-20B       -> general/chat only; tools disabled pending fix
```

### Findings by candidate

- **North Mini survives multi-turn.** Its single-shot worker strength did not merely survive—it translated into 5/5 solved, 5/5 clean termination, and zero counted bad calls. The prior runaway tendency did not appear over 21 agent turns. Keep it as the primary bounded worker and fast technical checker. Do not infer that it is safe as an open-ended main.
- **Qwen3.6-27B is confirmed as main.** It combines the strongest prior main/general evidence with 5/5 agentic success. One malformed write justifies a retry contract, not a topology change.
- **GPT-OSS-120B remains escalation, with a caveat.** It solved everything, but its 16.7% bad-call rate and repeated t4 serialization failures make it less attractive as a high-volume worker. Its value is judgment/review, not cheap tool throughput.
- **Devstral Small is confirmed as backup worker.** It recovered immediately from its one malformed call and otherwise followed efficient trajectories.
- **Qwen3.5-9B earns a new narrow role.** It is unsafe as main/general under warm open prompts but was the most turn-efficient perfect agent here. Add it only as a cheap, tightly bounded worker with watchdogs.
- **GPT-OSS-20B does not hold up in the loop.** It remains useful for chat and ordinary coding, but autonomous tool use must be disabled until its template/config is corrected.

## Final synthesis

For **autonomous Hermes/Claw clusters**, the preferred order is:

1. **North-Mini-Code** — primary bounded implementer.
2. **Qwen3.6-27B** — orchestrator/general technical agent.
3. **Devstral-Small2-24B** — worker backup.
4. **Qwen3.5-9B** — inexpensive narrow worker tier, never open-ended planner.
5. **GPT-OSS-120B** — independent reviewer/escalation behind strict JSON guards.
6. **GPT-OSS-20B** — disabled for tools.

For **general-purpose chat, Pi, knowledge work, and business/general coding**, keep the prior order conceptually separate: **Qwen3.6-27B** remains the default, **GPT-OSS-120B** the deep reviewer, **North Mini** the coding specialist, and **GPT-OSS-20B** remains viable as a smaller conversational/coding model despite being agentically unfit here.

The original four-model topology is therefore **confirmed, not overturned**. The only revisions are to add Qwen3.5-9B as an optional cheap worker and to put a stricter serialization guard around GPT-OSS-120B. The benchmark's strongest practical result is that North Mini's opportunity is real: bounded, verified, multi-turn implementation is exactly where it should be deployed.