Interface IEdgeTokenizerProvider

Namespace

Qavren.Edge.Embeddings.Onnx

Assembly

Qavren.Edge.Embeddings.Onnx.dll

Builds the tokenizer on first use and caches it for the process. A provider rather than a directly-injected IEdgeTokenizer, because the vocab file is OnnxModelFileRole.Vocabulary inside the same manifest as the graph, and spec 10 keeps provisioning LAZY: on a first launch there is no vocab path at the moment DI constructs the generator, and DI factories cannot await a download. So the generator takes this, and the first GenerateAsync awaits it - the same call that already awaits IOnnxSessionHost.AcquireAsync, which provisions the same manifest. Construction parses the whole vocab, so it happens exactly once PER PRESET, behind a SemaphoreSlim(1), and each built instance is reused for the process lifetime. Per preset, not once overall: the keyed AddOnnxEmbeddings(name, ...) overload exists so two presets can coexist in one process, and they differ in LowerCase and MaxSequenceLength - handing one preset's tokenizer to the other is a silent quality bug, never an exception.

public interface IEdgeTokenizerProvider

Properties

Current

Null until the first GetAsync(EmbeddingPreset, CancellationToken) completes, and the MOST RECENTLY BUILT tokenizer once more than one preset is registered. Diagnostics read a tokenizer rather than forcing provisioning to report a vocab size.

Methods

GetAsync(EmbeddingPreset, CancellationToken)

Provisions the vocabulary if needed and returns the cached tokenizer.