157 lines
12 KiB
Markdown
157 lines
12 KiB
Markdown
# Role-Fitness Benchmark
|
||
|
||
Scores are deliberately separate: **main /8**, **escalation /6**, and **worker live tests /27**. Worker correctness comes directly from `EXEC`; code style and control are secondary. No aggregate score is reported.
|
||
|
||
## Expected-answer audit
|
||
|
||
- **e04 is operationally ambiguous.** Noise figure alone does not determine a literal output-referenced noise floor without receiver gain. The intended `-174 + 3 = -171 dBm/Hz` is an input-referred/equivalent noise density (or assumes unity gain). The equivalent added noise temperature is correctly `Te = 290(10^(3/10)-1) ≈ 288.6 K`. I graded against that intended convention while noting the terminology issue.
|
||
- **m05:** WAV is a container/file format, not itself a codec; it can contain lossless PCM but does not satisfy “genuinely-lossless codec.” FLAC, ALAC, WavPack, WMA Lossless, Monkey's Audio, and TTA do.
|
||
- The other EXPECTED lines are accurate enough for grading.
|
||
|
||
## Main-role ranking
|
||
|
||
| Rank | Model | Main /8 | Behavioral note |
|
||
|---:|---|---:|---|
|
||
| 1 | qwen3.6-27b | **8.0** | Only fully compliant all-round main: calibrated, factually complete, exact warm formats, and clean termination. |
|
||
| 2= | gemma4-31b | 7.5 | Excellent calibration/writing; lost half for calling WAV a codec. |
|
||
| 2= | gpt-oss-20b | 7.5 | Concise and format-aware; “Tripoli” was right but omitted the historical/no-current-country qualification. |
|
||
| 2= | nemotron-cascade-30b | 7.5 | Strong calibration and exact four-word naming rationales; m08 was underqualified. |
|
||
| 5= | gpt-oss-120b | 7.0 | Reliable and complete, but WAV failed m05 and m07 rationales were not exactly four words. |
|
||
| 5= | nemotron3-super-120b | 7.0 | Clean and well formatted; omitted the real swallow estimate and historical flag qualification. |
|
||
| 5= | north-mini-code | 7.0 | Strong on every completed item, including exact formats, but m08 ran to 6144 tokens with no final—serious default-driver risk. |
|
||
| 8 | devstral2-123b-iq2 | 6.5 | Good daily behavior except verbose m07 format drift and a confident Saudi/Riyadh error on m08. |
|
||
| 9= | minimax-m2-iq2 | 6.0 | Good completed warm answers, but m03 and m08 both exhausted 8192 tokens with empty finals. |
|
||
| 9= | ministral3-14b | 6.0 | Generally usable; incomplete swallow answer, WAV-as-codec, m07 format drift, and unqualified historical flag answer. |
|
||
| 11 | devstral-small2-24b | 5.5 | Sound basics, but omitted swallow estimate, used WAV, missed exact m07 rationale length, and declined m08. |
|
||
| 12 | qwen3.5-9b | 4.0 | Four of eight main prompts ended at 6144 tokens with no final; unfit as a default despite good completed answers. |
|
||
|
||
Main scoring notes: m03 required both the Monty Python response and a real estimate; m05 received 0.5 when the exact numbered format held but one item was WAV; m07 received 0.5 when it had three usable ideas but violated the exact four-word rationale constraint. Empty finals received 0.
|
||
|
||
## Escalation-role ranking
|
||
|
||
| Rank | Model | Escalation /6 | Behavioral note |
|
||
|---:|---|---:|---|
|
||
| 1= | devstral-small2-24b | **6.0** | Detected all six planted errors and supplied the right corrections; verbose but dependable. |
|
||
| 1= | devstral2-123b-iq2 | **6.0** | Complete premise resistance and correct RF/security/math corrections. |
|
||
| 1= | gemma4-31b | **6.0** | Precise reviewer; explicitly handled e04 as input-referred and computed 288.6 K correctly. |
|
||
| 1= | gpt-oss-120b | **6.0** | Strongest low-surprise reviewer: all verdicts correct, well qualified, and cleanly terminated. |
|
||
| 1= | nemotron3-super-120b | **6.0** | Correct and methodical across RF, code, security, and math. |
|
||
| 1= | north-mini-code | **6.0** | Flawless reviewer here, including correct distinction between added 290 K and total 580 K. |
|
||
| 1= | qwen3.6-27b | **6.0** | Flawless error detection and technically careful corrections; sometimes more expansive than needed. |
|
||
| 8= | ministral3-14b | 5.5 | Found every planted premise, but e04 confidently finalized `-141 dBm/Hz` after noticing its own inconsistency. |
|
||
| 8= | nemotron-cascade-30b | 5.5 | Strong reviewer except e04 mislabeled total 580 K as equivalent added noise temperature. |
|
||
| 10 | qwen3.5-9b | 5.0 | Five correct reviews, but e04 ran to 6144 tokens with no final. |
|
||
| 11 | gpt-oss-20b | 4.5 | e02 ran away with no final; e04 confused total 580 K with added equivalent temperature. |
|
||
| 12 | minimax-m2-iq2 | 3.0 | Correct on e02/e03/e05, but e01/e04/e06 each consumed 8192 tokens and emitted no final. |
|
||
|
||
No completed escalation answer sycophantically agreed with a planted false premise. The major reviewer failure mode was **non-termination**, followed by e04's added-versus-total noise-temperature confusion.
|
||
|
||
## Worker-role ranking
|
||
|
||
All ties below are genuine on live correctness; behavioral notes distinguish deployment preference.
|
||
|
||
| Rank | Model | Worker tests | Tasks | Behavioral note |
|
||
|---:|---|---:|---:|---|
|
||
| 1= | north-mini-code | **27/27** | 8/8 | Every task passed, every worker final terminated, and all requested code/JSON formats held. Excellent bounded implementer. |
|
||
| 1= | devstral-small2-24b | **27/27** | 8/8 | Perfect live execution with concise, clean finals; strongest low-control-risk runner-up. |
|
||
| 1= | devstral2-123b-iq2 | **27/27** | 8/8 | Perfect implementation correctness and clean task completion, though oversized for this role. |
|
||
| 1= | gemma4-31b | **27/27** | 8/8 | Perfect tests and formats; capable but unnecessarily expensive/verbose as a worker. |
|
||
| 1= | gpt-oss-120b | **27/27** | 8/8 | Perfect but grossly overqualified for bounded snippets. Better reserved for escalation. |
|
||
| 1= | gpt-oss-20b | **27/27** | 8/8 | Perfect live worker performance; useful compact alternative. |
|
||
| 1= | nemotron-cascade-30b | **27/27** | 8/8 | Perfect finals; hidden traces were heavily padded, but control remained intact. |
|
||
| 1= | nemotron3-super-120b | **27/27** | 8/8 | Perfect but resource-inefficient for bounded work. |
|
||
| 1= | qwen3.5-9b | **27/27** | 8/8 | Surprisingly perfect small worker; verbose traces, but every worker final arrived and passed. |
|
||
| 1= | qwen3.6-27b | **27/27** | 8/8 | Perfect, though better used as main/orchestrator than commodity worker. |
|
||
| 11 | ministral3-14b | 24/27 | 7/8 | w02 failed all 3 dBm-to-watts tests; other seven tasks passed. |
|
||
| 12 | minimax-m2-iq2 | 19/27 | 6/8 | w04 and w07 ran to 8192 tokens without defining the functions, causing eight NameError failures. |
|
||
|
||
Worker format compliance was otherwise excellent: Python tasks returned a single code block and w06 returned raw JSON as requested. No worker process hung beyond the benchmark's token-limited completion; MiniMax's w04/w07 were token-limit runaways rather than executable hangs.
|
||
|
||
## Role-fit matrix
|
||
|
||
| Model | Main | Escalation | Worker |
|
||
|---|---|---|---|
|
||
| qwen3.6-27b | **Best** | **Best** | Strong |
|
||
| gpt-oss-120b | Strong | **Best** | Strong |
|
||
| gemma4-31b | **Best** | **Best** | Strong |
|
||
| nemotron-cascade-30b | Strong | Strong | Strong |
|
||
| nemotron3-super-120b | Strong | **Best** | Strong |
|
||
| north-mini-code | OK | **Best** | **Best** |
|
||
| devstral-small2-24b | OK | **Best** | **Best** |
|
||
| devstral2-123b-iq2 | OK | **Best** | Strong |
|
||
| gpt-oss-20b | Strong | OK | **Best** |
|
||
| qwen3.5-9b | **Unfit** | OK | **Best** |
|
||
| ministral3-14b | OK | Strong | OK |
|
||
| minimax-m2-iq2 | **Unfit** | **Unfit** | **Unfit** |
|
||
|
||
“Best” does not mean cost-optimal in every cell. GPT-OSS-120B and Gemma can execute worker tasks perfectly, but North Mini, Devstral Small, GPT-OSS-20B, and Qwen3.5-9B are more sensible worker deployments.
|
||
|
||
## Termination and control report
|
||
|
||
This is the decisive swarm-risk section.
|
||
|
||
| Model | Item | Failure |
|
||
|---|---|---|
|
||
| minimax-m2-iq2 | m03 | 8192-token runaway; empty final. |
|
||
| minimax-m2-iq2 | m08 | 8192-token runaway; empty final. |
|
||
| minimax-m2-iq2 | e01 | 8192-token runaway; empty final. |
|
||
| minimax-m2-iq2 | e04 | 8192-token runaway; empty final. |
|
||
| minimax-m2-iq2 | e06 | 8192-token runaway; empty final. |
|
||
| minimax-m2-iq2 | w04 | 8192-token runaway; no `parse_freq` definition; 0/4 tests. |
|
||
| minimax-m2-iq2 | w07 | 8192-token runaway; no `roman_to_int` definition; 0/4 tests. |
|
||
| qwen3.5-9b | m01 | 6144-token runaway; empty final. |
|
||
| qwen3.5-9b | m03 | 6144-token runaway; empty final. |
|
||
| qwen3.5-9b | m07 | 6144-token runaway; empty final. |
|
||
| qwen3.5-9b | m08 | 6144-token runaway; empty final. |
|
||
| qwen3.5-9b | e04 | 6144-token runaway; empty final. |
|
||
| north-mini-code | m08 | 6144-token runaway; empty final. Its worker and escalation blocks all terminated normally. |
|
||
| gpt-oss-20b | e02 | 6144-token runaway; empty final despite the trace circling the simple inverted-return bug. |
|
||
|
||
Exact-format violations that did terminate:
|
||
|
||
- **m05 invalid codec item:** devstral-small2-24b, gemma4-31b, gpt-oss-120b, and ministral3-14b each listed WAV.
|
||
- **m07 four-word rationale violation:** devstral-small2-24b, devstral2-123b-iq2, gpt-oss-120b, and ministral3-14b produced three ideas but not exact four-word rationales.
|
||
- **m02:** all twelve models that reached the final complied exactly with `8`.
|
||
- **m06:** all models produced exactly two on-topic sentences without the banned hype words.
|
||
- **Worker formats:** every non-runaway worker final complied; w06 correctly required raw JSON rather than a code fence.
|
||
|
||
For a swarm, qwen3.5-9B's perfect worker score should not be generalized to orchestration: its role-conditioned control is excellent on narrow code prompts and unacceptable on warm/open main prompts. North Mini shows the same pattern much more mildly—one main runaway, zero escalation/worker control failures.
|
||
|
||
## MiniMax-M2 verdict
|
||
|
||
**No—the larger token budget plus warm sampling did not fix the empty-final problem.**
|
||
|
||
Evidence:
|
||
|
||
- Seven of 22 blocks hit the 8192-token ceiling without a usable requested final: m03, m08, e01, e04, e06, w04, and w07.
|
||
- The two worker runaways directly caused **8 failed tests** through missing function definitions.
|
||
- Total worker result was only **19/27**, versus 27/27 for ten competitors.
|
||
- Completed answers were often good—m01/m02/m04–m07, e02/e03/e05, and six worker tasks—but increasing the budget merely allowed repetitive thought to continue for roughly 435 seconds per failed item.
|
||
|
||
This configuration remains **unfit for autonomous clusters**. The problem is termination/control, not insufficient token budget.
|
||
|
||
## Final recommendations
|
||
|
||
| Role | Winner | Runner-up | Deployment rationale |
|
||
|---|---|---|---|
|
||
| **Main** | **qwen3.6-27b** | **gemma4-31b** | Qwen was the sole 8/8 main, with exact formatting, calibrated corrections, good RF writing, and no control failure. Gemma was polished and reliable, missing only the WAV distinction. |
|
||
| **Escalation** | **gpt-oss-120b** | **qwen3.6-27b** | Both scored 6/6; GPT-OSS gets the edge for low-surprise reviewer behavior, while Qwen is the more economical technically precise alternative. |
|
||
| **Worker** | **north-mini-code** | **devstral-small2-24b** | Both passed 27/27 with clean worker termination. North's prior opportunity is validated specifically for bounded implementation; Devstral Small is the clean, concise fallback. |
|
||
|
||
### Can one model fill two roles?
|
||
|
||
**Yes: Qwen3.6-27B is good enough to serve both main and escalation.** It is the only perfect main and tied for perfect escalation, with no termination failure in this run. For a small cluster, using Qwen as both default and reviewer is defensible, although independent-model review is safer; GPT-OSS-120B adds useful diversity.
|
||
|
||
**North Mini should fill worker plus bounded escalation, not main plus worker.** Its 27/27 worker and 6/6 escalation results show real value, but the m08 runaway confirms that warm/open-ended main traffic can trigger its known control weakness. Route concrete implementation, code review, RF calculations, and planted-error checks to North; enforce final-output/time limits and fall back to Qwen on missing finals.
|
||
|
||
Recommended cluster topology:
|
||
|
||
```text
|
||
Qwen3.6-27B -> main/orchestrator
|
||
GPT-OSS-120B -> independent escalation/review
|
||
North-Mini-Code -> bounded worker and fast technical checker
|
||
Devstral-Small2 -> worker runner-up / overflow pool
|
||
```
|
||
|
||
For autonomous swarms, this role specialization is preferable to choosing models by a single aggregate benchmark score. North Mini's opportunity is now evidence-backed: **do not trust it to converse indefinitely, but do trust it with bounded code whose completion is mechanically verified.**
|