Method CreateWordPieceAsync

Namespace

Qavren.Edge.Embeddings.Onnx

Assembly

Qavren.Edge.Embeddings.Onnx.dll

CreateWordPieceAsync(string, WordPieceTokenizerOptions?, CancellationToken)

Builds a WordPiece tokenizer from a vocab.txt on disk, asynchronously.

public static Task<IEdgeTokenizer> CreateWordPieceAsync(string vocabFilePath, WordPieceTokenizerOptions? options = null, CancellationToken cancellationToken = default)

Parameters

vocabFilePath string

The absolute path of the vocabulary.

options WordPieceTokenizerOptions

Construction settings, or null for the defaults.

cancellationToken CancellationToken

Cancellation.

Returns

Task<IEdgeTokenizer>

The tokenizer.