experiments-07-07-12-local-llm-knowledge-2/POWER_ANALYSIS.md

Dual-MI50 Power × Context Sweep

Recommendation: do not adopt 180 W or 160 W for large-context service; their 32K prefill retention falls to 71.1% and 20.1%. On the present data, use 225 W as the single safe cap. If accepting a provisional split, use 225 W for interactive/chat and 200 W for prefill-heavy long-context/RAG, because 200 W retains 102.7% of 32K prefill—but this split must be revalidated: decode measurements are highly non-monotonic and every draw_w value is missing.

Source: mi50_pxc_sweep_20260712_191205.csv (most recent sweep).

Raw pivot tables

Prefill throughput (pp_tok_s)

Power	512	4,096	16,384	32,768
225 W	3.42	2.52	2.79	5.51
200 W	3.14	2.68	3.52	5.66
180 W	3.62	3.34	1.88	3.92
160 W	2.96	2.99	1.41	1.11

Decode throughput (tg_tok_s)

Power	512	4,096	16,384	32,768
225 W	0.32	0.60	0.22	0.72
200 W	0.16	0.34	0.22	0.41
180 W	0.21	0.29	0.27	0.28
160 W	0.24	0.16	0.23	0.53

The absolute throughput shape is suspicious: decode changes non-monotonically with both context and power, and 32K prefill is faster than several much smaller contexts. Some batching/overhead effects can make long prefills more efficient, but the size and inconsistency here require repeated measurements and variance reporting.

Retention versus 225 W

Legend: ⚠ <95%, ⛔ <90%. Values above 100% indicate measurement variance or a confound, not a physical efficiency gain that should be assumed reproducible.

Prefill retention

Power	512	4,096	16,384	32,768
225 W	100.0%	100.0%	100.0%	100.0%
200 W	⚠ 91.8%	106.3%	126.2%	102.7%
180 W	105.8%	132.5%	⛔ 67.4%	⛔ 71.1%
160 W	⛔ 86.5%	118.7%	⛔ 50.5%	⛔ 20.1%

Decode retention

Power	512	4,096	16,384	32,768
225 W	100.0%	100.0%	100.0%	100.0%
200 W	⛔ 50.0%	⛔ 56.7%	100.0%	⛔ 56.9%
180 W	⛔ 65.6%	⛔ 48.3%	122.7%	⛔ 38.9%
160 W	⛔ 75.0%	⛔ 26.7%	104.5%	⛔ 73.6%

No lower cap is consistently within 95% of baseline. For prefill, 200 W is effectively free at 4K–32K in this sample, while 180/160 W collapse at 16K–32K. Decode does not show the expected flat memory-bound behavior; almost every lower-power cell is below 90%, interspersed with implausible reversals at 16K.

Decode versus prefill divergence

Hypothesis verdict

Partly confirmed for prefill; refuted by the recorded decode data, but the decode result is not yet trustworthy.

• At small contexts, prefill appears insensitive/noisy: 180 W reports 105.8% retention at 512 and 132.5% at 4K; 160 W reports 86.5% and 118.7%.
• At large contexts, the compute penalty becomes decisive below 200 W: at 16K/32K, 180 W retains only 67.4%/71.1%, and 160 W only 50.5%/20.1%.
• 200 W does not starve large-context prefill in this run: it reports 126.2% at 16K and 102.7% at 32K. Those >100% values should be interpreted as “no detected loss,” not as a real speedup.
• Decode should be comparatively power-insensitive if purely HBM-bandwidth-bound, but the recorded results are not flat. At 200 W, decode retention is 50.0%, 56.7%, 100.0%, and 56.9% across increasing contexts. At 160 W it rebounds from 26.7% at 4K to 104.5% at 16K. This pattern is too inconsistent for a clean architectural conclusion.

Direct answer to “is the optimal low-power config useless at the largest context?”: 160 W is clearly useless for 32K prefill, and 180 W imposes a meaningful ~29% loss. 200 W is the only lower cap that looks viable for large prefill.

Thermal and draw check

Edge temperature (edge_temp_c)

Power	512	4,096	16,384	32,768
225 W	32°C	42°C	54°C	55°C
200 W	37°C	43°C	54°C	55°C
180 W	38°C	43°C	54°C	57°C
160 W	40°C	44°C	52°C	53°C

• Maximum observed edge temperature is 57°C, which does not suggest thermal throttling for an MI50. The 225 W cells top out at 55°C, so there is no evidence that a thermally throttled baseline artificially favored 200 W.
• Temperature rises as the sweep progresses through context sizes and, at 512 tokens, lower caps are paradoxically hotter than 225 W. This strongly suggests run-order/warm-up history affects edge temperature; randomized power/context order or steady-state soak is needed.
• Actual draw cannot be checked: all 16 draw_w entries are NA. Therefore the sweep cannot establish whether the cards reached their caps, whether both GPUs drew similarly, or how many real watts a lower cap saved.

Efficiency and knee

The requested actual tg_tok_s / draw_w metric is not computable because draw_w is missing everywhere. Substituting the configured cap would falsely assume actual draw equals the cap, so it is not used as the efficiency result.

For diagnosis only, the proxy tg_tok_s / configured-power-W is:

Power	512	4,096	16,384	32,768
225 W	0.001422	0.002667	0.000978	0.003200
200 W	0.000800	0.001700	0.001100	0.002050
180 W	0.001167	0.001611	0.001500	0.001556
160 W	0.001500	0.001000	0.001437	0.003313

This proxy is internally inconsistent because the underlying decode measurements are. It cannot identify a defensible efficiency knee. The only repeatable-looking knee in the present sweep is the prefill knee at 200 W: dropping from 200 to 180 W changes 32K retention from 102.7% to 71.1%, and 16K from 126.2% to 67.4%.

Per-context provisional knees:

Context	Provisional knee	Basis
512	Undetermined / 225 W safe	Lower-power pp and tg are non-monotonic; 200 W loses 8.2% pp and 50% tg.
4,096	Undetermined / 225 W safe	Prefill looks free at all caps, but decode falls 43–73%.
16,384	200 W	Prefill is retained at 200 W, then falls to 67.4% at 180 W; decode readings are too noisy.
32,768	200 W for prefill, 225 W overall	Prefill is retained at 200 W, but measured decode falls to 56.9%; 180/160 W prefill is unacceptable.

Recommendation

If setting one cap now

Set 225 W. This is the only cap supported across all measured decode cells, and the data does not demonstrate that any lower setting is “free.” Temperatures at 225 W were modest (≤55°C), with no evidence of throttling.

If splitting boxes provisionally

• Interactive chat / Pi / decode-sensitive box: 225 W. The recorded decode penalty at 200 W is 43–50% at 512, 4K, and 32K. Those values may be measurement artifacts, but they make a lower recommendation unsafe until rerun.
• Long-context prefill-heavy agent/RAG box: 200 W. It shows no prefill loss at 16K/32K and avoids the severe large-context collapse at 180/160 W. If long generations follow the prefill, 225 W remains safer because 200 W's 32K decode was only 56.9% of baseline.

Thus the data does not support the hoped-for simple conclusion that low power is free for chat decode. It supports a narrower result: 200 W may be free for large prefills, while 180 W and especially 160 W are not.

Required rerun before cabinet deployment

1. Fix draw_w collection and record per-GPU draw plus total board draw.
2. Run at least 3–5 repetitions per cell and retain standard deviation/min/max, not only an average.
3. Randomize or alternate power order to remove warm-up/run-order bias.
4. Soak each cap to steady-state temperature before timing.
5. Record core clock, HBM clock, throttling flags, and GPU utilization.
6. Verify llama.cpp timing extraction; the decode reversals suggest parsing or run variance.
7. Add end-to-end latency: prefill_time + generation_time, weighted for real chat versus RAG request distributions.

Until that rerun, 225 W is the defensible production setting; 200 W is the promising cabinet-friendly candidate for long-context prefill, not yet a proven universal optimum.