Class EdgeChunkTokenizer

Namespace

Qavren.Edge.Ingestion.Onnx

Assembly

Qavren.Edge.Ingestion.Onnx.dll

IChunkTokenizer over SP2's IEdgeTokenizer (spec 8.1, 11.2). It holds ONLY the IEdgeTokenizer: CountTokens(ReadOnlySpan<char>) forwards to CountTokens(ReadOnlySpan<char>), which SP2 forwards verbatim to Tokenizer.CountTokens at its defaults and which takes a span rather than returning an index, so no normalised string can leak through it.

public sealed class EdgeChunkTokenizer : IChunkTokenizer

Inheritance

object

EdgeChunkTokenizer

Implements

IChunkTokenizer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

IndexByTokenCount(string, int, out int) does not forward to IndexByTokenCount(string, int, out int). That member forwards Tokenizer.GetIndexByTokenCount with normalisation left on and discards the normalised string, so its index is into a copy SP2 does not need and SP3 cannot use - the exact bug spec 8.1's offset contract exists to prevent. The index is derived here from CountTokens(ReadOnlySpan<char>) through Internal.TokenIndexSearch, the ONE implementation both tokenizers share (plan adjustment 1, ADR 0013), reached through the core's InternalsVisibleTo grant (plan adjustment 24).

SpecialTokenOverhead is 2, measured on 2026-09-11 against the shipped Microsoft.ML.Tokenizers 2.0.0 BertTokenizer (plan "Environment ground truth", spec 17 item 2): "hello world" counts 2 and encodes to 4 ids, "the quick brown fox" counts 4 and encodes to 6, "a" counts 1 and encodes to 3 - [CLS] and [SEP] on every sequence, which CountTokens does not report.

Id is spelled exactly as EdgeTokenCounter.CreateWordPiece spells its own - wordpiece:{MaxSequenceLength}:{uncased|cased} - on purpose. The id is a recipe input, and the ONNX-free path and this bridge are the same vocabulary at the same settings; a consumer who swaps one wiring for the other must not re-index a corpus for a spelling difference.

Constructors

EdgeChunkTokenizer(IEdgeTokenizer, bool)

Wraps a built SP2 tokenizer.

Properties

Id

A stable identity for the vocabulary and model shape, e.g. wordpiece:30522:minilm-l6-v2-int8.

MaxSequenceLength

The model ceiling, special tokens INCLUDED.

SpecialTokenOverhead

Tokens the encoder adds that CountTokens(ReadOnlySpan<char>) does not report.

Methods

CountTokens(ReadOnlySpan<char>)

The WordPiece (or equivalent) cost of text, specials EXCLUDED.

IndexByTokenCount(string, int, out int)

The largest index i such that text[0..i) costs at most maxTokens tokens. CONTRACT: i indexes into text AS PASSED — never into a normalised copy of it (spec 8.1, "The offset contract"). This is load-bearing: every committed golden offset is an index into Text.