Interface IChunkTokenizer

Namespace

Qavren.Edge.Ingestion

Assembly

Qavren.Edge.Ingestion.dll

The token seam every chunker is budgeted against (spec 8.1). Declared in the core so no core type names Qavren.Edge.Embeddings.Onnx; the ONNX satellite supplies its own implementation.

public interface IChunkTokenizer

Properties

Id

A stable identity for the vocabulary and model shape, e.g. wordpiece:30522:minilm-l6-v2-int8.

MaxSequenceLength

The model ceiling, special tokens INCLUDED.

SpecialTokenOverhead

Tokens the encoder adds that CountTokens(ReadOnlySpan<char>) does not report.

Methods

CountTokens(ReadOnlySpan<char>)

The WordPiece (or equivalent) cost of text, specials EXCLUDED.

IndexByTokenCount(string, int, out int)

The largest index i such that text[0..i) costs at most maxTokens tokens. CONTRACT: i indexes into text AS PASSED — never into a normalised copy of it (spec 8.1, "The offset contract"). This is load-bearing: every committed golden offset is an index into Text.