Class MlChunkTokenizer

Namespace

Qavren.Edge.Ingestion

Assembly

Qavren.Edge.Ingestion.dll

IChunkTokenizer over a Microsoft.ML.Tokenizers Tokenizer. IndexByTokenCount(string, int, out int) returns an index into the string AS PASSED — the offset contract spec 8.1 turns on — and it derives that index from CountTokens(ReadOnlySpan<char>) through Internal.TokenIndexSearch rather than forwarding to Tokenizer.GetIndexByTokenCount.

public sealed class MlChunkTokenizer : IChunkTokenizer

Inheritance

object

MlChunkTokenizer

Implements

IChunkTokenizer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.ReferenceEquals(object, object)

object.ToString()

Remarks

Spec 8.1's bullet and ADR 0013's context paragraph still describe the forwarding call (considerNormalization: false); both were written before the real 30,522-entry vocabulary was measured. Neither overload of that call satisfies the offset contract: at considerNormalization: true the index is into the NORMALISED string, which mislocates every boundary after the first non-ASCII difference, and at false the shipped BertTokenizer does not search at all — it hands back text.Length with tokenCount 1 for every budget. ADR 0013's decision (one shared prefix search) is unchanged; only its stated reason moves.

Constructors

MlChunkTokenizer(Tokenizer, int, int, string?)

Properties

Id

A stable identity for the vocabulary and model shape, e.g. wordpiece:30522:minilm-l6-v2-int8.

MaxSequenceLength

The model ceiling, special tokens INCLUDED.

SpecialTokenOverhead

Tokens the encoder adds that CountTokens(ReadOnlySpan<char>) does not report.

Methods

CountTokens(ReadOnlySpan<char>)

The WordPiece (or equivalent) cost of text, specials EXCLUDED.

IndexByTokenCount(string, int, out int)

The largest index i such that text[0..i) costs at most maxTokens tokens. CONTRACT: i indexes into text AS PASSED — never into a normalised copy of it (spec 8.1, "The offset contract"). This is load-bearing: every committed golden offset is an index into Text.