Class EdgeVectorStoreMediWriter
- Namespace
- Qavren.Edge.Ingestion.DataIngestion
- Assembly
- Qavren.Edge.Ingestion.DataIngestion.dll
Writes MEDI chunks into an SP2 collection shaped by BuildDefinition(int, string, bool) (spec 11, plan task 6.3 step 4), so a MEDI pipeline and SP3's own IIngestionPipeline can share one collection and one search.
Content-hash incremental re-index is NOT available on this path. MEDI's model has nowhere to put a document hash, a recipe hash or a state row, so this writer cannot know whether a document changed. Every WriteAsync(IAsyncEnumerable<IngestionChunk<string>>, CancellationToken) is therefore a full rewrite of each document it sees: every chunk is embedded again and upserted, and any previously stored chunk of that document that the new set does not contain is deleted afterwards. On a phone that is the difference between a run that skips ninety-nine unchanged files and one that re-embeds all hundred — IIngestionPipeline is what a device should use; this type exists so the MEDI ecosystem can reach an SP2 collection at all.
Rows are written through ToRecord(ReadOnlyMemory<float>) with the identity SP3 derives
(spec 9.3: source id, document id, the embed-text hash and a duplicate ordinal), so a document
written here and later re-indexed by IIngestionPipeline diffs cleanly instead of
duplicating. Chunks that came through EdgeChunkerMediAdapter carry their offsets,
token count, kind and page; chunks from any other MEDI chunker get -1 offsets, a token
count from the optional tokenizer (or 0), Paragraph and page
-1. The embedded text is EmbedTextKey when present
and the chunk's Content otherwise — MEDI's own writer embeds Content too.
Additions land before deletions, per document, so a failure mid-write leaves the old chunks in
place rather than none. Nothing here opens a transaction around a collection call (spec 9.5's
invariant): the stored-key read is one connection, each upsert is SP2's own transaction, and the
stale-key delete is SP2's DeleteAsync.
public sealed class EdgeVectorStoreMediWriter : IngestionChunkWriter<string>, IDisposable- Inheritance
-
EdgeVectorStoreMediWriter
- Implements
- Inherited Members
Constructors
- EdgeVectorStoreMediWriter(VectorStoreCollection<object, Dictionary<string, object?>>, IEmbeddingGenerator<string, Embedding<float>>, string, IChunkTokenizer?, int, TimeProvider?)
Creates the writer.
Properties
- SourceId
The source id every written row carries.
Methods
- Dispose(bool)
Disposes the writer.
- WriteAsync(IAsyncEnumerable<IngestionChunk<string>>, CancellationToken)
Writes chunks asynchronously.