Table of Contents

Class MarkdownHeadingChunker

Namespace
Qavren.Edge.Ingestion
Assembly
Qavren.Edge.Ingestion.dll

Spec 8.2's structure-aware chunker: a string?[7] heading stack, cleared below the current level on each new heading, splitting only at SplitHeadingLevels (H1-H3 by default) so H4+ stay inside their parent section - which is how documents are actually written.

public sealed class MarkdownHeadingChunker : IChunker
Inheritance
MarkdownHeadingChunker
Implements
Inherited Members

Remarks

The seven rules, each of which the prior art got wrong:

  1. ChunkDraft.HeadingPath is the structural array in memory; what is stored is the rendered breadcrumb, sanitised through SanitizeHeading(string).
  2. A splitting heading is EXCLUDED from the chunk body and prepended to the EMBED text, never duplicated into the stored text.
  3. Content before the first heading becomes a preamble chunk when IncludePreamble. The prior art dropped it and lost every document's lede.
  4. A section over budget falls to the token window with overlap preserved and the SAME heading path on every piece.
  5. Sections under MinTokens merge into the next sibling, event 911.
  6. Tables split row-wise with the header row re-emitted on every piece when RepeatTableHeaderRow.
  7. A breadcrumb over HeadingPathTokenBudget is truncated from the LEFT and logged as event 910 - in ChunkAssembly, which owns that rule for every chunker.

Two behaviours the spec leaves implicit and this implementation fixes, recorded here because a reader of the prose alone would guess differently. FIRST: a heading BELOW the split levels - an H4 under the default H1-H3 - stays in the body as content, because "H4+ stay inside their parent section" and a heading excluded from the body while also not splitting would simply vanish. SECOND: a merged run's stored text stays the verbatim slice [first.Start, last.End), which means the intervening splitting headings ARE inside a merged body. Joining the bodies instead would make Text stop being Text[CharStart..CharEnd), and every offset in every golden depends on that identity. A merged chunk takes the FIRST section's heading path, because that is the path at its CharStart.

Constructors

MarkdownHeadingChunker()
MarkdownHeadingChunker(ILogger?)

Properties

Id

One of ChunkerIds.

Version

Bumped when the emitted boundaries change; part of the recipe hash.

Methods

Chunk(ExtractedDocument, ResolvedChunkOptions, IChunkTokenizer)

Chunk one extracted document against a frozen budget.

Edit this page