Method CreateWordPiece

Namespace

Qavren.Edge.Ingestion

Assembly

Qavren.Edge.Ingestion.dll

CreateWordPiece(string, int, bool)

WordPiece over a vocab.txt. bert-base-uncased shape; overhead 2.

public static IChunkTokenizer CreateWordPiece(string vocabFilePath, int maxSequenceLength, bool lowerCase = true)

Parameters

vocabFilePath string

maxSequenceLength int

lowerCase bool

Returns

IChunkTokenizer

CreateWordPiece(Stream, int, bool)

WordPiece over a vocab.txt stream, for an embedded or asset-packed vocabulary.

public static IChunkTokenizer CreateWordPiece(Stream vocabTxt, int maxSequenceLength, bool lowerCase = true)

Parameters

vocabTxt Stream

maxSequenceLength int

lowerCase bool

Returns

IChunkTokenizer