Method CreateWordPiece

Namespace

Qavren.Edge.Embeddings.Onnx

Assembly

Qavren.Edge.Embeddings.Onnx.dll

CreateWordPiece(string, WordPieceTokenizerOptions?)

Builds a WordPiece tokenizer from a vocab.txt on disk.

public static IEdgeTokenizer CreateWordPiece(string vocabFilePath, WordPieceTokenizerOptions? options = null)

Parameters

vocabFilePath string

The absolute path of the vocabulary.

options WordPieceTokenizerOptions

Construction settings, or null for the defaults.

Returns

IEdgeTokenizer

The tokenizer.

CreateWordPiece(Stream, WordPieceTokenizerOptions?)

Builds a WordPiece tokenizer from a vocab.txt stream.

public static IEdgeTokenizer CreateWordPiece(Stream vocabTxt, WordPieceTokenizerOptions? options = null)

Parameters

vocabTxt Stream

The vocabulary, one token per line.

options WordPieceTokenizerOptions

Construction settings, or null for the defaults.

Returns

IEdgeTokenizer

The tokenizer.