experiments-07-07-12-local-llm-knowledge-2/POWER_ANALYSIS.md

# Dual-MI50 Power × Context Sweep

**Recommendation:** do **not** adopt 180 W or 160 W for large-context service; their 32K prefill retention falls to **71.1%** and **20.1%**. On the present data, use **225 W as the single safe cap**. If accepting a provisional split, use **225 W for interactive/chat** and **200 W for prefill-heavy long-context/RAG**, because 200 W retains **102.7%** of 32K prefill—but this split must be revalidated: decode measurements are highly non-monotonic and every `draw_w` value is missing.

Source: `mi50_pxc_sweep_20260712_191205.csv` (most recent sweep).

## Raw pivot tables

### Prefill throughput (`pp_tok_s`)

| Power | 512 | 4,096 | 16,384 | 32,768 |
|---:|---:|---:|---:|---:|
| 225 W | 3.42 | 2.52 | 2.79 | 5.51 |
| 200 W | 3.14 | 2.68 | 3.52 | 5.66 |
| 180 W | 3.62 | 3.34 | 1.88 | 3.92 |
| 160 W | 2.96 | 2.99 | 1.41 | 1.11 |

### Decode throughput (`tg_tok_s`)

| Power | 512 | 4,096 | 16,384 | 32,768 |
|---:|---:|---:|---:|---:|
| 225 W | 0.32 | 0.60 | 0.22 | 0.72 |
| 200 W | 0.16 | 0.34 | 0.22 | 0.41 |
| 180 W | 0.21 | 0.29 | 0.27 | 0.28 |
| 160 W | 0.24 | 0.16 | 0.23 | 0.53 |

The absolute throughput shape is suspicious: decode changes non-monotonically with both context and power, and 32K prefill is faster than several much smaller contexts. Some batching/overhead effects can make long prefills more efficient, but the size and inconsistency here require repeated measurements and variance reporting.

## Retention versus 225 W

Legend: **⚠ <95%**, **⛔ <90%**. Values above 100% indicate measurement variance or a confound, not a physical efficiency gain that should be assumed reproducible.

### Prefill retention

| Power | 512 | 4,096 | 16,384 | 32,768 |
|---:|---:|---:|---:|---:|
| 225 W | 100.0% | 100.0% | 100.0% | 100.0% |
| 200 W | **⚠ 91.8%** | 106.3% | 126.2% | 102.7% |
| 180 W | 105.8% | 132.5% | **⛔ 67.4%** | **⛔ 71.1%** |
| 160 W | **⛔ 86.5%** | 118.7% | **⛔ 50.5%** | **⛔ 20.1%** |

### Decode retention

| Power | 512 | 4,096 | 16,384 | 32,768 |
|---:|---:|---:|---:|---:|
| 225 W | 100.0% | 100.0% | 100.0% | 100.0% |
| 200 W | **⛔ 50.0%** | **⛔ 56.7%** | 100.0% | **⛔ 56.9%** |
| 180 W | **⛔ 65.6%** | **⛔ 48.3%** | 122.7% | **⛔ 38.9%** |
| 160 W | **⛔ 75.0%** | **⛔ 26.7%** | 104.5% | **⛔ 73.6%** |

No lower cap is consistently within 95% of baseline. For prefill, 200 W is effectively free at 4K–32K in this sample, while 180/160 W collapse at 16K–32K. Decode does **not** show the expected flat memory-bound behavior; almost every lower-power cell is below 90%, interspersed with implausible reversals at 16K.

## Decode versus prefill divergence

### Hypothesis verdict

**Partly confirmed for prefill; refuted by the recorded decode data, but the decode result is not yet trustworthy.**

- At small contexts, prefill appears insensitive/noisy: 180 W reports 105.8% retention at 512 and 132.5% at 4K; 160 W reports 86.5% and 118.7%.
- At large contexts, the compute penalty becomes decisive below 200 W: at 16K/32K, 180 W retains only **67.4%/71.1%**, and 160 W only **50.5%/20.1%**.
- 200 W does **not** starve large-context prefill in this run: it reports **126.2% at 16K** and **102.7% at 32K**. Those >100% values should be interpreted as “no detected loss,” not as a real speedup.
- Decode should be comparatively power-insensitive if purely HBM-bandwidth-bound, but the recorded results are not flat. At 200 W, decode retention is **50.0%, 56.7%, 100.0%, and 56.9%** across increasing contexts. At 160 W it rebounds from 26.7% at 4K to 104.5% at 16K. This pattern is too inconsistent for a clean architectural conclusion.

Direct answer to “is the optimal low-power config useless at the largest context?”: **160 W is clearly useless for 32K prefill, and 180 W imposes a meaningful ~29% loss. 200 W is the only lower cap that looks viable for large prefill.**

## Thermal and draw check

### Edge temperature (`edge_temp_c`)

| Power | 512 | 4,096 | 16,384 | 32,768 |
|---:|---:|---:|---:|---:|
| 225 W | 32°C | 42°C | 54°C | 55°C |
| 200 W | 37°C | 43°C | 54°C | 55°C |
| 180 W | 38°C | 43°C | 54°C | 57°C |
| 160 W | 40°C | 44°C | 52°C | 53°C |

- Maximum observed edge temperature is **57°C**, which does not suggest thermal throttling for an MI50. The 225 W cells top out at 55°C, so there is no evidence that a thermally throttled baseline artificially favored 200 W.
- Temperature rises as the sweep progresses through context sizes and, at 512 tokens, lower caps are paradoxically hotter than 225 W. This strongly suggests run-order/warm-up history affects edge temperature; randomized power/context order or steady-state soak is needed.
- **Actual draw cannot be checked:** all 16 `draw_w` entries are `NA`. Therefore the sweep cannot establish whether the cards reached their caps, whether both GPUs drew similarly, or how many real watts a lower cap saved.

## Efficiency and knee

The requested actual `tg_tok_s / draw_w` metric is **not computable** because `draw_w` is missing everywhere. Substituting the configured cap would falsely assume actual draw equals the cap, so it is not used as the efficiency result.

For diagnosis only, the proxy `tg_tok_s / configured-power-W` is:

| Power | 512 | 4,096 | 16,384 | 32,768 |
|---:|---:|---:|---:|---:|
| 225 W | 0.001422 | 0.002667 | 0.000978 | 0.003200 |
| 200 W | 0.000800 | 0.001700 | 0.001100 | 0.002050 |
| 180 W | 0.001167 | 0.001611 | 0.001500 | 0.001556 |
| 160 W | 0.001500 | 0.001000 | 0.001437 | 0.003313 |

This proxy is internally inconsistent because the underlying decode measurements are. It cannot identify a defensible efficiency knee. The only repeatable-looking knee in the present sweep is the **prefill knee at 200 W**: dropping from 200 to 180 W changes 32K retention from 102.7% to 71.1%, and 16K from 126.2% to 67.4%.

Per-context provisional knees:

| Context | Provisional knee | Basis |
|---:|---:|---|
| 512 | Undetermined / 225 W safe | Lower-power pp and tg are non-monotonic; 200 W loses 8.2% pp and 50% tg. |
| 4,096 | Undetermined / 225 W safe | Prefill looks free at all caps, but decode falls 43–73%. |
| 16,384 | 200 W | Prefill is retained at 200 W, then falls to 67.4% at 180 W; decode readings are too noisy. |
| 32,768 | 200 W for prefill, 225 W overall | Prefill is retained at 200 W, but measured decode falls to 56.9%; 180/160 W prefill is unacceptable. |

## Recommendation

### If setting one cap now

Set **225 W**. This is the only cap supported across all measured decode cells, and the data does not demonstrate that any lower setting is “free.” Temperatures at 225 W were modest (≤55°C), with no evidence of throttling.

### If splitting boxes provisionally

- **Interactive chat / Pi / decode-sensitive box: 225 W.** The recorded decode penalty at 200 W is 43–50% at 512, 4K, and 32K. Those values may be measurement artifacts, but they make a lower recommendation unsafe until rerun.
- **Long-context prefill-heavy agent/RAG box: 200 W.** It shows no prefill loss at 16K/32K and avoids the severe large-context collapse at 180/160 W. If long generations follow the prefill, 225 W remains safer because 200 W's 32K decode was only 56.9% of baseline.

Thus the data does **not** support the hoped-for simple conclusion that low power is free for chat decode. It supports a narrower result: **200 W may be free for large prefills, while 180 W and especially 160 W are not.**

### Required rerun before cabinet deployment

1. Fix `draw_w` collection and record per-GPU draw plus total board draw.
2. Run at least 3–5 repetitions per cell and retain standard deviation/min/max, not only an average.
3. Randomize or alternate power order to remove warm-up/run-order bias.
4. Soak each cap to steady-state temperature before timing.
5. Record core clock, HBM clock, throttling flags, and GPU utilization.
6. Verify llama.cpp timing extraction; the decode reversals suggest parsing or run variance.
7. Add end-to-end latency: `prefill_time + generation_time`, weighted for real chat versus RAG request distributions.

Until that rerun, **225 W is the defensible production setting; 200 W is the promising cabinet-friendly candidate for long-context prefill, not yet a proven universal optimum.**