Method UseChunkTokenizer

Namespace

Qavren.Edge.Ingestion

Assembly

Qavren.Edge.Ingestion.dll

UseChunkTokenizer(EdgeBuilder, Func<IServiceProvider, IChunkTokenizer>)

Supplies the IChunkTokenizer without the ONNX satellite — EdgeTokenCounter.CreateWordPiece builds one from a vocab.txt.

public static EdgeBuilder UseChunkTokenizer(this EdgeBuilder builder, Func<IServiceProvider, IChunkTokenizer> factory)

Parameters

builder EdgeBuilder

The Qavren.Edge builder.

factory Func<IServiceProvider, IChunkTokenizer>

Builds the tokenizer.

Returns

EdgeBuilder

The builder, for chaining.