Interface IEdgeTokenizer

Namespace

Qavren.Edge.Embeddings.Onnx

Assembly

Qavren.Edge.Embeddings.Onnx.dll

The one abstraction over Microsoft.ML.Tokenizers SP2 exposes. Implementations hold the CONCRETE tokenizer type, never the Tokenizer base: BertTokenizer.EncodeToIds is declared new, so a base-typed field silently drops [CLS] and [SEP].

public interface IEdgeTokenizer : IDisposable

Inherited Members

IDisposable.Dispose()

Properties

Kind

Which tokenizer family this instance is.

MaxSequenceLength

The longest sequence this tokenizer will emit, special tokens included.

PadTokenId

The id padded positions carry.

VocabularySize

How many entries the vocabulary holds.

Methods

CountTokens(ReadOnlySpan<char>)

Counts the tokens in text.

Encode(ReadOnlySpan<char>, int, Span<int>, out int)

Single encode into a caller-owned buffer. Writes ids into destination, adding the special tokens and truncating so the total never exceeds maxTokens. Returns the id count; throws ArgumentException when destination is shorter than maxTokens.

This is NOT allocation-free on the pinned tokenizer version, and the contract must not claim otherwise. Every EncodeToIds overload in Microsoft.ML.Tokenizers 2.0.0 returns a list; the only span-destination members in the library - BuildInputsWithSpecialTokens, GetSpecialTokensMask, CreateTokenTypeIdsFromSequences - all take ids that already exist. The implementation therefore calls the truncating EncodeToIds overload and copies the result into destination: one short-lived list per input, and no allocation in the batch assembler, which is where the buffers that actually matter live. The span signature is kept because it is the shape the assembler wants, because it keeps the per-input allocation an implementation detail rather than a public one, and because Tokenizers 3.x may add a span overload this method can then adopt without a surface change.

EncodeBatch(IReadOnlyList<string>, int, IReadOnlyList<int>)

Encodes, truncates, right-pads to the smallest configured bucket that fits the batch maximum, and synthesises the attention mask.

The returned batch's three long[] are rented from ArrayPool<T>.Shared and are valid over their first Qavren.Edge.Embeddings.Onnx.TokenizedBatch.TensorLength elements only. The generator hands them back once the OrtValues built over them are disposed; a caller outside this package simply drops the batch and the GC reclaims the buffers without them ever re-entering the pool.

IndexByTokenCount(string, int, out int)

Index into text at which maxTokens tokens are consumed. Exists so SP3's token-window chunker never writes a second token counter.