# Local Model Portfolio Analysis ## Executive recommendation For this operator—RF/SDR and signal processing, embedded/systems work, AI/ML for radio, local coding agents, general knowledge work, and writing—the best outcome is not one universal model. The benchmark supports a small portfolio: | Role | Recommended model | Why | |---|---|---| | Default daily driver | **Qwen3.6-27B** | Perfect hard-round score, all traps rejected, all calculations correct, and it caught the benchmark's own incorrect 5G NR reference. Best balance of breadth, technical judgment, and manageable size. | | High-reliability reasoning / coding agent | **GPT-OSS-120B** | Perfect hard round with clean derivations and strong premise resistance. Its answers were more controlled than several equally accurate but more verbose models. | | Writing and broad knowledge | **Gemma4-31B** | Only perfect model in the easy knowledge round and perfect in the hard round. Fluent, broad, and reliable, though slower and inclined to overexplain. | | Efficient specialist / alternative daily model | **Nemotron-Cascade-30B** | Perfect hard round, strong reasoning and trap handling, with an efficient active-parameter architecture. Tone is more rigid and report-like. | | Fast technical snippet model | **North-Mini-Code** | Near-perfect knowledge and reasoning, full RF/math performance, excellent software-trap awareness, and fast generation. Its isolated catastrophic no-final failure makes it inappropriate as an unconstrained agent, but highly valuable in bounded roles. | | Smaller economical fallback | **GPT-OSS-20B** | Strong hard-round performance with one arithmetic slip. A credible lower-cost general/coding assistant when the 120B model is unnecessary. | If only one model is kept, choose **Qwen3.6-27B**. If two are kept, add **GPT-OSS-120B** for difficult agentic and reasoning work. If three are kept, add **North-Mini-Code** as a fast bounded coding and technical worker—or Gemma4-31B if writing is more important than throughput. ## What the benchmark actually revealed The easy round mostly measured factual breadth. Many models clustered between 38 and 40 out of 40, so those totals alone overstated equivalence. The hard round exposed three more useful traits: 1. **Can the model execute a numerical chain without category or arithmetic errors?** 2. **Will it reject a plausible but false premise rather than comply with it?** 3. **Will its reasoning terminate in a usable final answer?** These are directly relevant to engineering and agentic work. A model that recalls a formula but uses orbital radius instead of altitude, accepts a security misconception, or consumes its token budget without producing a final result is expensive in practice even if its prose sounds capable. The strongest evidence came from Q15. The supplied judge note claimed that real 5G NR uses 52 resource blocks (624 subcarriers) for 20 MHz at 30 kHz SCS. The real standardized allocation is 51 RBs (612 subcarriers). **Qwen3.6-27B correctly identified 612**, demonstrating domain-aware recall and a willingness to go beyond a flawed reference answer. That is unusually relevant for this operator. ## Recommended shortlist ### Qwen3.6-27B — primary daily driver Qwen3.6-27B is the best overall fit rather than merely a leaderboard winner. It scored 38/40 in the easy round and 24/24 in the hard round. Its easy-round misses were narrow: an overbroad Chandrasekhar-limit consequence, an incorrect detail about loop-of-Henle permeability, and bad FSPL arithmetic. In the harder run it corrected all of those behavioral concerns: every reasoning question was solved, every false premise was rejected, and its 5G standards knowledge exceeded the reference key. Best applications: - RF/SDR and systems questions where the answer must combine recall with calculation. - Interactive debugging and design review. - Coding snippets and bounded coding tasks. - Primary local conversational assistant. - First-pass technical analysis before escalation to a larger model. Residual risks: - It can overexplain. - The easy-round FSPL miss shows that even this model should expose calculations for review. - A 24-question hard round cannot establish long-horizon agent reliability by itself. Operational recommendation: make it the default model, require explicit assumptions and unit checks for engineering calculations, and route only difficult or high-consequence tasks to GPT-OSS-120B. ### GPT-OSS-120B — high-confidence reasoner and agent supervisor GPT-OSS-120B scored 39.5/40 in the easy round and 24/24 in the hard round. It demonstrated correct numerical methods, strong premise resistance, and relatively controlled presentation. Its only easy-round deduction came from repeating the common overbroad white-dwarf-collapse answer. Best applications: - Planning and supervising multi-step coding tasks. - Reviewing patches produced by smaller models. - Difficult debugging, architecture, and systems reasoning. - Independent verification of RF/math calculations. - Tasks where a wrong confident answer costs more than inference time. Residual risks: - Large storage/memory footprint and higher total inference cost. - Perfect benchmark performance does not guarantee correct tool use or repository-scale persistence. Operational recommendation: use it as the escalation and review model, not necessarily for every short interaction. A productive pattern is Qwen3.6-27B or North Mini producing work, then GPT-OSS-120B reviewing assumptions, edge cases, and final changes. ### Gemma4-31B — writing, synthesis, and broad knowledge Gemma4-31B was the only 40/40 model in the easy round and also scored 24/24 in the hard round. It is the strongest evidence-backed choice for broad factual questions and polished explanatory writing. It consistently produced coherent, well-structured answers but often used much more text than required. Best applications: - Reports, documentation, explanations, and rewriting. - Brainstorming where breadth and fluency matter. - General research synthesis from supplied material. - Teaching-style explanations of technical concepts. Residual risks: - Slower observed generation than several alternatives. - Verbosity can obscure the decision or action item. - The benchmark did not directly test creative writing quality, voice matching, or long-document editing. Operational recommendation: use a system instruction such as “lead with the answer; use at most five bullets unless asked for detail.” It is a strong writing partner, but Qwen3.6-27B remains the better engineering default. ### Nemotron-Cascade-30B — efficient technical alternative Nemotron-Cascade scored 36.5/40 in the easy round but 24/24 in the hard round. Its easy misses were factual/discriminator items—especially *Toxicity*, FSPL, and omitted BPSK—not failures of the hard reasoning chain. In the hard run it was methodical, correct, and premise-resistant. Best applications: - Structured technical analysis. - Calculation-heavy engineering assistance. - A local fallback when the larger GPT-OSS model is excessive. - Reviewer or second-opinion model for Qwen output. Residual risks: - It can be padded and rigid. - Its weaker easy-round RF recall means it should not be assumed superior to Qwen3.6-27B merely because it aced the hard calculations. Operational recommendation: keep it if its runtime characteristics are favorable on the MI50 setup. It is more valuable as an independent second opinion than as another conversational model duplicating Qwen. ### GPT-OSS-20B — economical generalist GPT-OSS-20B scored 36.5/40 in the easy round and 23.5/24 in the hard round. The hard-round loss was a small arithmetic error in compound depreciation, not a conceptual failure. It resisted all traps. The earlier easy run did show runaway answers on two late questions, so output control still matters. Best applications: - Lower-cost daily Q&A. - Straightforward coding and review. - Bounded reasoning tasks with calculator/test verification. - Parallel second opinions. Residual risks: - Arithmetic slips. - Occasional runaway generation under the tested prompting/template configuration. - Weaker easy-round RF and econ/law reliability than the leading group. Operational recommendation: a good utility model, but it does not displace Qwen3.6-27B as the main driver or GPT-OSS-120B as the high-confidence agent. ## North-Mini-Code: where the opportunity is North Mini should be retained. Its aggregate hard score of 23/24 hides a very asymmetric profile: - **Recall: 8/8** - **Reasoning: 8/8** - **Traps: 7/8** - Easy-round score: **39.5/40** - Easy-round RF, math, chemistry, biology, history, geography, literature, music, and econ/law: **all 4/4** Its only hard failure was not a wrong final answer. On the Einstein trap, its thinking trace became repetitive, oscillated between the myth and the correction, and exhausted 4096 tokens without emitting a final answer. That is a severe failure for an autonomous agent, but it does not erase what the other 63 benchmark answers show: excellent technical knowledge, correct calculations, strong software-security awareness, and high throughput. The opportunity is to use North Mini where tasks are **short, verifiable, and structurally bounded**, while preventing it from owning long open-ended loops. ### High-value North Mini roles 1. **Snippet generation** Give it one function, one interface, explicit constraints, and a test or expected behavior. It correctly rejected `gets()` and SQL-injection premises and recommended bounded input and parameterized queries. That is exactly the behavior wanted from a fast snippet model. 2. **Patch drafting within a narrow file scope** Ask it to modify one or two named files with explicit acceptance criteria. Have a stronger model or tests review the patch before merge. 3. **Compiler/test-error repair** Feed it a concrete error plus the relevant code window. These tasks have a natural stopping condition and objective verification. 4. **RF/math calculation worker** It solved every hard reasoning question, including ADC resolution, Shannon capacity, resistor reduction, exponential depreciation, NR subcarrier approximation, and radioactive decay. It is suitable for calculations when required to show units and when results are checked by a calculator or test harness. 5. **Code explanation and API lookup from supplied context** It has broad recall, but grounding it in headers, documentation, or repository code reduces the risk of invented API details. 6. **Candidate generator in a model ensemble** Let North Mini quickly propose two or three implementations; let Qwen3.6-27B select/refine them or GPT-OSS-120B review the chosen result. Its speed becomes an asset without making its rare control failure the system's single point of failure. 7. **Static-review checklist worker** Use bounded prompts for unsafe C functions, SQL construction, unchecked lengths, resource leaks, error handling, and suspicious concurrency patterns. The software traps suggest good security instincts. ### Roles North Mini should not own - Unattended multi-hour coding agents. - Open-ended research where it decides when it has enough evidence. - Tasks whose success depends on recovering from uncertainty across many steps. - Final approval for security-critical or destructive changes. - Long conversations with large hidden thinking budgets and no enforced output contract. ### Recommended containment Use an execution contract rather than relying on prompting style alone: - Cap internal/output tokens more aggressively for short tasks. - Require a final answer schema: `assumptions`, `answer/patch`, `tests`, `uncertainties`. - Treat missing final output as an automatic retry/fallback condition. - Detect repetition (repeated n-grams or paragraphs) and terminate early. - Retry once with thinking disabled or a terse prompt; then route to Qwen3.6-27B. - Give it explicit maximum steps and named files. - Require commands/tests to validate calculations and code. - Do not let it recursively re-plan without external progress. A useful routing policy: ```text short + concrete + testable -> North Mini broad interactive technical work -> Qwen3.6-27B long-horizon or high-consequence agent -> GPT-OSS-120B writing / synthesis -> Gemma4-31B independent technical second opinion -> Nemotron-Cascade-30B ``` North Mini's best role is therefore **fast technical implementer**, not autonomous technical authority. Used this way, its 23/24 hard result is more informative than the one failure: it can do nearly all the local work cheaply, provided another component owns termination, verification, and escalation. ## Models to deprioritize | Model | Recommendation | Reason | |---|---|---| | MiniMax-M2-IQ2 | Drop under this configuration | Repetitive thinking produced empty finals on most hard questions. This is a control failure, not a small quality gap. | | Aya-23-8B | Drop for technical use | Very weak numerical reasoning, specialist recall errors, and confident fabrication despite fluent prose. | | Granite4-Micro | Keep only for footprint-constrained utility | Exceptional for ~2 GB, but insufficient premise resistance and arithmetic reliability for trusted RF/systems work. | | Ministral3-14B | Deprioritize | Respectable, but Qwen3.6-27B and GPT-OSS-20B provide stronger evidence for the intended roles. | | Devstral-Small2-24B | Deprioritize | Good traps and broad competence, but weaker numerical reliability than the preferred small/medium options. | | Devstral2-123B-IQ2 | Deprioritize | Large footprint without matching the smaller leaders; made the GEO category error and overconfident practical estimates. | | GLM4.7-Flash | Deprioritize for RF | Good general behavior, but the GEO geometry error is an unfavorable signal for this operator's domain. | | GLM4.6V-IQ2 | Optional only if vision matters | Excellent hard core, but this text benchmark provides no evidence for its vision advantage and Qwen is a cleaner default. | | Nemotron3-Nano-30B | Optional utility model | Strong hard reasoning but no clear portfolio role not already covered better by Qwen, Cascade, or GPT-OSS-20B. | ## Deployment plan ### Minimal two-model setup - **Qwen3.6-27B** for daily work, RF/technical questions, snippets, and interactive problem solving. - **GPT-OSS-120B** for agentic coding, difficult reviews, and high-consequence verification. ### Recommended three-model setup - Add **North-Mini-Code** as the fast bounded worker. - Route all North output through tests or review for consequential changes. ### Writing-focused extension - Add **Gemma4-31B** for reports, documentation, brainstorming, and broad explanatory work. ### Ensemble workflow for engineering 1. Qwen3.6-27B frames the problem and assumptions. 2. North Mini drafts code or performs bounded calculations. 3. Automated tests/calculators verify objective outputs. 4. GPT-OSS-120B reviews difficult or high-risk results. 5. Gemma4-31B turns approved material into documentation when needed. This division exploits model diversity rather than asking every model to imitate the same general assistant. The main strategic opportunity is North Mini: it has frontier-like bounded technical performance and high speed, but needs an external control plane. With termination detection, schema enforcement, tests, and escalation, its failure mode is containable and its strengths become economically useful. ## Bottom line The evidence supports **Qwen3.6-27B as the primary model**, **GPT-OSS-120B as the trusted reasoner/agent**, and **North-Mini-Code as a fast bounded implementer**. Gemma4-31B is the best addition for writing and broad synthesis. Nemotron-Cascade-30B is the strongest independent technical alternative. Do not discard North Mini because of one catastrophic trace. Do not ignore that trace either. Its correct deployment is not “let it run until done”; it is “give it a small, testable job, enforce a final-output contract, detect loops, and escalate failures.” In that role, it may offer one of the best practical throughput-to-quality ratios in the entire tested set.