Constructor EdgeChunkTokenizer

Namespace

Qavren.Edge.Ingestion.Onnx

Assembly

Qavren.Edge.Ingestion.Onnx.dll

EdgeChunkTokenizer(IEdgeTokenizer, bool)

Wraps a built SP2 tokenizer.

public EdgeChunkTokenizer(IEdgeTokenizer tokenizer, bool lowerCase = true)

Parameters

tokenizer IEdgeTokenizer

The tokenizer, already provisioned.

lowerCase bool

EmbeddingPreset.LowerCase for the preset this tokenizer was built for. It is part of the Id only; IEdgeTokenizer does not expose it.