Table of Contents

Namespace Qavren.Edge.Ingestion.DataIngestion

Classes

EdgeChunkerMediAdapter

Wraps an SP3 IChunker behind MEDI's IngestionChunker<T> contract (spec 11, plan task 6.3 step 2).

It carries no constant. MEDI's shipped chunkers default to 2000 tokens with a 500-token overlap, four to eight times what a 384-dimension device model can encode; this adapter takes the ResolvedChunkOptions the caller already froze — normally through Resolve(ChunkModelProfile, IChunkTokenizer) against the real IChunkTokenizer — and hands that budget to the chunker verbatim.

Each IngestionChunk<T> carries the chunk's stored text as Content, its breadcrumb as Context, and the rest of the ChunkDraft under the metadata keys declared here, so EdgeVectorStoreMediWriter can write the same row SP3's own runner would.

EdgeDocumentConverter

Converts between SP3's ExtractedDocument and MEDI's IngestionDocument, in both directions (spec 11, plan task 6.3 step 1).

Lossless in both directions, except for two things, both by design.

What MEDI's vocabulary actually is. Spec 6 describes DocumentBlockKind as "MEDI's vocabulary". The shipped abstractions carry five concrete element types — paragraph, header, footer, table and image — so Heading, Footer and Paragraph map to a native element, a run of TableRow blocks becomes one IngestionDocumentTable (one row per block, one column), and the four kinds MEDI has no element for — ListItem, Code, Caption, Quote — ride on an IngestionDocumentParagraph carrying BlockKindKey in its metadata. The mapping is one switch over all eight members with no default bucket, so a ninth kind fails to compile here rather than silently becoming a paragraph.

Document-level facts (extractor id and version, media type, page count, text-layer flag, warnings and Metadata) have no home on IngestionDocument either, so they travel on the root IngestionDocumentSection's metadata under the qedge. keys declared below. A MEDI document that did not come from SP3 simply has none of them and gets the documented defaults.

EdgeVectorStoreMediWriter

Writes MEDI chunks into an SP2 collection shaped by BuildDefinition(int, string, bool) (spec 11, plan task 6.3 step 4), so a MEDI pipeline and SP3's own IIngestionPipeline can share one collection and one search.

Content-hash incremental re-index is NOT available on this path. MEDI's model has nowhere to put a document hash, a recipe hash or a state row, so this writer cannot know whether a document changed. Every WriteAsync(IAsyncEnumerable<IngestionChunk<string>>, CancellationToken) is therefore a full rewrite of each document it sees: every chunk is embedded again and upserted, and any previously stored chunk of that document that the new set does not contain is deleted afterwards. On a phone that is the difference between a run that skips ninety-nine unchanged files and one that re-embeds all hundred — IIngestionPipeline is what a device should use; this type exists so the MEDI ecosystem can reach an SP2 collection at all.

Rows are written through ToRecord(ReadOnlyMemory<float>) with the identity SP3 derives (spec 9.3: source id, document id, the embed-text hash and a duplicate ordinal), so a document written here and later re-indexed by IIngestionPipeline diffs cleanly instead of duplicating. Chunks that came through EdgeChunkerMediAdapter carry their offsets, token count, kind and page; chunks from any other MEDI chunker get -1 offsets, a token count from the optional tokenizer (or 0), Paragraph and page -1. The embedded text is EmbedTextKey when present and the chunk's Content otherwise — MEDI's own writer embeds Content too.

Additions land before deletions, per document, so a failure mid-write leaves the old chunks in place rather than none. Nothing here opens a transaction around a collection call (spec 9.5's invariant): the stored-key read is one connection, each upsert is SP2's own transaction, and the stale-key delete is SP2's DeleteAsync.

MediReaderAdapter

Turns any MEDI IngestionDocumentReader into an SP3 IDocumentExtractor (spec 11, plan task 6.3 step 3). This is how a consumer gets a format SP3 does not ship — HTML, say — without SP3 shipping an extractor for it (spec 18): register the adapter through AddDocumentExtractor or IngestionOptions.Extractors and the registry resolves it like any other extractor, ahead of the built-ins.

The source is opened through OpenAsync, as every SP3 extractor must be, and handed to the reader's stream overload. When NormalizeText is on, each element's text is normalised (CRLF and lone CR to LF, NFC, BOM stripped) before the text buffer is built, so the offsets FromMedi(IngestionDocument, ILogger?) assigns already index the normalised text. The same caveat as the core's normaliser applies: under InvariantGlobalization Normalize(NormalizationForm) is a no-op, so there the line-ending and BOM rules are the whole of it.