Method Encode

Namespace

Qavren.Edge.Embeddings.Onnx

Assembly

Qavren.Edge.Embeddings.Onnx.dll

Encode(ReadOnlySpan<char>, int, Span<int>, out int)

Single encode into a caller-owned buffer. Writes ids into destination, adding the special tokens and truncating so the total never exceeds maxTokens. Returns the id count; throws ArgumentException when destination is shorter than maxTokens.

This is NOT allocation-free on the pinned tokenizer version, and the contract must not claim otherwise. Every EncodeToIds overload in Microsoft.ML.Tokenizers 2.0.0 returns a list; the only span-destination members in the library - BuildInputsWithSpecialTokens, GetSpecialTokensMask, CreateTokenTypeIdsFromSequences - all take ids that already exist. The implementation therefore calls the truncating EncodeToIds overload and copies the result into destination: one short-lived list per input, and no allocation in the batch assembler, which is where the buffers that actually matter live. The span signature is kept because it is the shape the assembler wants, because it keeps the per-input allocation an implementation detail rather than a public one, and because Tokenizers 3.x may add a span overload this method can then adopt without a surface change.

int Encode(ReadOnlySpan<char> text, int maxTokens, Span<int> destination, out int charsConsumed)

Parameters

text ReadOnlySpan<char>

The text to encode.

maxTokens int

The id ceiling, special tokens included.

destination Span<int>

Where the ids are written. At least maxTokens long.

charsConsumed int

How many characters of the normalized text were consumed.

Returns

int

The number of ids written.