Method FromTokenizer

Namespace

Qavren.Edge.Ingestion

Assembly

Qavren.Edge.Ingestion.dll

FromTokenizer(Tokenizer, int, int)

Any Microsoft.ML.Tokenizers tokenizer, with the overhead stated by the caller.

public static IChunkTokenizer FromTokenizer(Tokenizer tokenizer, int maxSequenceLength, int specialTokenOverhead)

Parameters

tokenizer Tokenizer

maxSequenceLength int

specialTokenOverhead int

Returns

IChunkTokenizer