Class EdgeTokenCounter

Namespace

Qavren.Edge.Ingestion

Assembly

Qavren.Edge.Ingestion.dll

The ONNX-free construction path for IChunkTokenizer (spec 8.1).

public static class EdgeTokenCounter

Inheritance

object

EdgeTokenCounter

Inherited Members

object.Equals(object)

object.Equals(object, object)

object.GetHashCode()

object.GetType()

object.MemberwiseClone()

object.ReferenceEquals(object, object)

object.ToString()

Methods

CreateWordPiece(Stream, int, bool)

WordPiece over a vocab.txt stream, for an embedded or asset-packed vocabulary.

CreateWordPiece(string, int, bool)

WordPiece over a vocab.txt. bert-base-uncased shape; overhead 2.

FromTokenizer(Tokenizer, int, int)

Any Microsoft.ML.Tokenizers tokenizer, with the overhead stated by the caller.