Constructor MlChunkTokenizer

Namespace

Qavren.Edge.Ingestion

Assembly

Qavren.Edge.Ingestion.dll

MlChunkTokenizer(Tokenizer, int, int, string?)

Wraps tokenizer as an IChunkTokenizer.

public MlChunkTokenizer(Tokenizer tokenizer, int maxSequenceLength, int specialTokenOverhead, string? id = null)

Parameters

tokenizer Tokenizer

The underlying Microsoft.ML.Tokenizers tokenizer.

maxSequenceLength int

The model ceiling, special tokens included.

specialTokenOverhead int

Tokens the encoder adds that tokenizer's own count does not report.

id string

A stable identity, or null to derive one from the tokenizer's type and maxSequenceLength.

Exceptions

ArgumentNullException

tokenizer is null.

ArgumentOutOfRangeException

maxSequenceLength is not positive, or specialTokenOverhead is negative.