Class ChunkOptions
The unresolved chunk budget. Resolve(ChunkModelProfile, IChunkTokenizer) freezes it against a tokenizer.
public sealed class ChunkOptions- Inheritance
-
ChunkOptions
- Inherited Members
Properties
- HeadingPathTokenBudget
Reserved out of the model ceiling for the rendered breadcrumb.
- IncludePreamble
Content before the first heading becomes a chunk. The prior art dropped it.
- MaxTokens
Null resolves per spec 8.1. Setting it explicitly opts out of the derivation.
- MergeShortSections
A section under MinTokens merges forward (event 911).
- MinTokens
Null resolves to
max(16, MaxTokens / 8).
- Overflow
What to do with an INPUT unit bigger than the budget.
- OverlapTokens
Null resolves to
max(8, (MaxTokens * 15 / 100) / 8 * 8).
- PrependHeadingPath
Prepend the breadcrumb to the EMBED text. Never to the stored text (spec 8.3).
- RepeatTableHeaderRow
Re-emit a table's header row on every piece of a row-wise split.
- SentenceAware
Nudge every cut backwards to a sentence boundary inside a 15% look-back window.
- SplitHeadingLevels
H1-H3 by default, so H4+ stay inside their parent section.
Methods
- Resolve(ChunkModelProfile, IChunkTokenizer)
Needs the tokenizer, so it runs in the order-400 startup task and not at builder time (spec 8.1). Throws IngestionChunkBudgetInvalid (6003) with the arithmetic in the message, or ChunkTokenizerCeilingExceeded (6153) when the tokenizer and the profile disagree about the ceiling.