Property KvCacheBytesPerElement
KvCacheBytesPerElement
Bytes per KV cache element. Default 2 (fp16).
This is the one field the cross-check cannot validate. genai_config.json
states layers, KV heads and head size but says nothing about the KV dtype, so a
disagreement here is undetectable at load and shows up only as a budget that is wrong by a
factor of two. 2 is the calibrated default, not a guess: for
Llama32_1BInstructInt4 it puts the arithmetic within 1.2% of a measured 1280 MiB
peak, where 4 would put it 11% over. Spec section 19 item 4 is the nightly measurement that
answers it per preset, and MeasuredPeakBytes is what absorbs the error
meanwhile.
public int KvCacheBytesPerElement { get; init; }