Class TokenWindowChunker

Namespace

Qavren.Edge.Ingestion

Assembly

Qavren.Edge.Ingestion.dll

The terminal fallback every other chunker delegates to (spec 8.2). Forward cut via IndexByTokenCount(string, int, out int), each candidate nudged backwards through a 15% sentence look-back, every cut snapped to a grapheme-cluster boundary, and the next window seeded by counting OverlapTokens back from the emitted chunk's end.

public sealed class TokenWindowChunker : IChunker

Inheritance

object

TokenWindowChunker

Implements

IChunker

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Constructors

TokenWindowChunker()

TokenWindowChunker(ILogger?)

Properties

Id

One of ChunkerIds.

Version

Bumped when the emitted boundaries change; part of the recipe hash.

Methods

Chunk(ExtractedDocument, ResolvedChunkOptions, IChunkTokenizer)

Chunk one extracted document against a frozen budget.