Method IndexByTokenCount

Namespace

Qavren.Edge.Embeddings.Onnx

Assembly

Qavren.Edge.Embeddings.Onnx.dll

IndexByTokenCount(string, int, out int)

Index into text at which maxTokens tokens are consumed. Exists so SP3's token-window chunker never writes a second token counter.

int IndexByTokenCount(string text, int maxTokens, out int tokenCount)

Parameters

text string

The text to measure.

maxTokens int

The token ceiling.

tokenCount int

How many tokens were actually consumed.

Returns

int

The character index.