Class PlainChunker

Namespace

Qavren.Edge.Ingestion

Assembly

Qavren.Edge.Ingestion.dll

Spec 8.2's block-accumulating chunker: blocks are accumulated while they fit, a block over budget goes to the token window, and a run under MinTokens merges forward. Offsets are the accumulated blocks' [first.Start, last.End), so the stored text is a verbatim slice of Text — the separating newlines included.

public sealed class PlainChunker : IChunker

Inheritance

object

PlainChunker

Implements

IChunker

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

Spec 8.2 words the accumulation over Paragraph blocks, which is every block PlainTextExtractor emits. This implementation accumulates blocks of EVERY kind, because auto resolves here for any non-Markdown media type and a chunker that silently dropped a TableRow would lose the document rather than chunk it. The emitted chunk takes the kind of the first block in the run.

Constructors

PlainChunker()

PlainChunker(ILogger?)

Properties

Id

One of ChunkerIds.

Version

Bumped when the emitted boundaries change; part of the recipe hash.

Methods

Chunk(ExtractedDocument, ResolvedChunkOptions, IChunkTokenizer)

Chunk one extracted document against a frozen budget.