Class EdgeTokenizer

Namespace

Qavren.Edge.Embeddings.Onnx

Assembly

Qavren.Edge.Embeddings.Onnx.dll

Builds WordPiece tokenizers over a vocab.txt.

public static class EdgeTokenizer

Inheritance

object

EdgeTokenizer

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Methods

CreateWordPiece(Stream, WordPieceTokenizerOptions?)

Builds a WordPiece tokenizer from a vocab.txt stream.

CreateWordPiece(string, WordPieceTokenizerOptions?)

Builds a WordPiece tokenizer from a vocab.txt on disk.

CreateWordPieceAsync(string, WordPieceTokenizerOptions?, CancellationToken)

Builds a WordPiece tokenizer from a vocab.txt on disk, asynchronously.