Enum EdgeTokenizerKind

Namespace

Qavren.Edge.Embeddings.Onnx

Assembly

Qavren.Edge.Embeddings.Onnx.dll

Which tokenizer family a preset's tokenizer asset belongs to.

public enum EdgeTokenizerKind

Fields

WordPieceVocabTxt = 0

A WordPiece vocab.txt, which is every preset SP2 ships.

UnigramTokenizerJson = 1

A Unigram tokenizer.json. Present so the multilingual shape is ready when the library is; raises TokenizerKindUnsupported today (ADR 0006).