Method IndexByTokenCount

Namespace

Qavren.Edge.Ingestion

Assembly

Qavren.Edge.Ingestion.dll

IndexByTokenCount(string, int, out int)

The largest index i such that text[0..i) costs at most maxTokens tokens. CONTRACT: i indexes into text AS PASSED — never into a normalised copy of it (spec 8.1, "The offset contract"). This is load-bearing: every committed golden offset is an index into Text.

public int IndexByTokenCount(string text, int maxTokens, out int tokenCount)

Parameters

text string

maxTokens int

tokenCount int

Returns

int