Namespace Qavren.Edge.Ingestion
Namespaces
Classes
- ChunkDraft
One chunk before identity and embedding. HeadingPath is the STRUCTURAL array; the stored column is the joined breadcrumb (spec 8.3).
- ChunkModelProfile
The six facts about an embedding model that chunking and the recipe need (spec 11). Plain data, declared HERE in the core, so no core type names
Qavren.Edge.Embeddings.Onnx.
- ChunkOptions
The unresolved chunk budget. Resolve(ChunkModelProfile, IChunkTokenizer) freezes it against a tokenizer.
- ChunkerIds
The four chunker ids SP3 ships.
- DocumentSourceItem
One item a source yields. SP3 never interprets Path.
- EdgeChunkingException
A chunker violated or could not satisfy the frozen budget.
- EdgeExtractionException
An extractor failed on one document.
- EdgeIngestionEventIds
Every log event Qavren.Edge.Ingestion emits, at spec section 11's numbers.
- EdgeIngestionException
Base type for every fault Qavren.Edge.Ingestion raises deliberately (spec 13.2). Configuration faults reuse SP1's EdgeConfigurationException with a 60xx code, so
AddIngestionfails the way every other builder call in the suite fails.
- EdgeIngestionStartupOrder
The order SP3's single startup task runs at. It opens no database; it resolves the tokenizer and freezes the chunk budget (spec 8.1, 11.1 step 5).
- EdgeIngestionStateException
The persisted ingestion state is missing, corrupt, or of an unsupported version.
- EdgeTokenCounter
The ONNX-free construction path for IChunkTokenizer (spec 8.1).
- ExtractedDocument
One normalised text buffer plus blocks indexing into it. CRLF and lone CR to LF, NFC, BOM stripped, when NormalizeText is on (spec 6).
- ExtractionContext
What an extractor is handed. Read-only: warnings come back on Warnings.
- ExtractionOptions
Extractor-agnostic knobs. Per-format options live on the satellites' own option types.
- FixedIngestionThrottle
The core's default: always the configured batch, no delay, never pauses.
- IngestedChunk
The typed view of one stored row. SP3 writes
Dictionary<string, object?>through the dynamic collection; this is the mapper both directions.
- IngestionBudget
A ceiling a run suspends against (spec 10.2). Every member is null by default, so a budget that sets nothing is Unlimited.
- IngestionColumns
Every storage name as a
const, plus the breadcrumb separator and the sanitiser that makes the join/split round-trip (spec 8.3). The sanitiser lives beside the separator on purpose: the round-trip is a property of the PAIR, and splitting them across two files is how the two drift.
- IngestionDocumentResult
What one document cost and produced.
- IngestionDocumentStatus
The state table's
statusvalues, whose spelling is this enum's names.
- IngestionEdgeBuilderExtensions
The four builder calls
Qavren.Edge.Ingestionpublishes.
- IngestionFailure
One reported (not thrown) failure. Carries the code so a caller can switch on it.
- IngestionMediaTypes
Extension-to-media-type mapping for the formats SP3 ships. Used by spec 11.3's sample.
- IngestionOptions
Everything
AddIngestionis configured with. Every default here is stated in the plan's "Defaults that change behaviour" heading and asserted byOptionsDefaultsTests.
- IngestionRecipe
The tuple a chunk's vector is a pure function of (spec 9.2). When a document's stored
recipe_hashdiffers from the run's, that document is dirty regardless of its content hash — which is what stops aMaxTokenschange from leaving every old vector in place.
- IngestionRunOptions
Per-run overrides. Every member is null-or-false by design; each default is stated in the plan's "Defaults that change behaviour" heading for Task 5.1 and asserted by a test.
- IngestionRunResult
What a run did. A suspension always lands on a committed boundary (spec 10.2).
- IngestionSchema
The two schemas SP3 owns: the chunk collection's
VectorStoreCollectionDefinitionand the three state tables.
- IngestionSource
Where a run's documents come from. The abstract shape is two members; the five factories below cover the filesystem shape and the mobile shape (spec 11, 11.3).
- IngestionStatus
What the collection currently holds (spec 12).
- MarkdownExtractor
Spec 7.3's Markdown extractor, over Markdig. Blocks come from the top-level Markdig.Syntax.MarkdownDocument children, and because every
MarkdownObjectcarries aSourceSpan, a block's[Start, End)is the VERBATIM source range — fences, tables and links survive intact instead of being re-rendered.
- MarkdownExtractorOptions
Knobs for MarkdownExtractor (spec 7.3).
- MarkdownHeadingChunker
Spec 8.2's structure-aware chunker: a
string?[7]heading stack, cleared below the current level on each new heading, splitting only atSplitHeadingLevels(H1-H3 by default) so H4+ stay inside their parent section - which is how documents are actually written.
- MlChunkTokenizer
IChunkTokenizer over a
Microsoft.ML.TokenizersTokenizer. IndexByTokenCount(string, int, out int) returns an index into the string AS PASSED — the offset contract spec 8.1 turns on — and it derives that index from CountTokens(ReadOnlySpan<char>) throughInternal.TokenIndexSearchrather than forwarding toTokenizer.GetIndexByTokenCount.
- PlainChunker
Spec 8.2's block-accumulating chunker: blocks are accumulated while they fit, a block over budget goes to the token window, and a run under
MinTokensmerges forward. Offsets are the accumulated blocks'[first.Start, last.End), so the stored text is a verbatim slice of Text — the separating newlines included.
- PlainTextExtractor
Spec 7.2's plain-text extractor. Blocks are Paragraphs split on blank lines; encoding resolves BOM, then a strict UTF-8 probe, then Latin-1, with no third-party dependency.
- PlainTextExtractorOptions
Knobs for PlainTextExtractor (spec 7.2).
- ResolvedChunkOptions
The frozen budget. Carried verbatim by the recipe, so every field is in the hash.
- TokenWindowChunker
The terminal fallback every other chunker delegates to (spec 8.2). Forward cut via IndexByTokenCount(string, int, out int), each candidate nudged backwards through a 15% sentence look-back, every cut snapped to a grapheme-cluster boundary, and the next window seeded by counting
OverlapTokensback from the emitted chunk's end.
Structs
- ContentHash
A 128-bit xxHash128 content hash. Persisted big-endian as
BLOB(16)(spec 9.1).
- DocumentBlock
[Start, End)half-open into Text.
- IngestionProgress
Counts, never a percentage: enumeration is streaming, so the denominator is unknown until the run ends and a fabricated percentage is worse than none (spec 10.4).
- IngestionThrottleContext
What the runner tells a throttle about the run so far.
- IngestionThrottleDecision
The tri-state spec 10.2's table produces. Pause wins over Delay wins over BatchSize; Reason is one of that table's strings and becomes
IngestionRunResult.SuspendReasonon a suspend.
Interfaces
- IChunkTokenizer
The token seam every chunker is budgeted against (spec 8.1). Declared in the core so no core type names
Qavren.Edge.Embeddings.Onnx; the ONNX satellite supplies its own implementation.
- IChunker
A chunker. Synchronous by design: CPU over an in-memory string, no I/O (spec 8.2).
- IDocumentExtractor
Turns one source item into one ExtractedDocument (spec 7.1). Every extractor opens the source through OpenAsync;
File.ReadAllBytesappears nowhere in SP3, and a test asserts it.
- IDocumentExtractorRegistry
Resolves an extractor for one source item. Consumer registrations sit ahead of the built-ins, so
.mdcan be overridden (spec 7.1).
- IIngestionPipeline
The one runtime surface. Resolved from DI after
AddIngestion.
- IIngestionThrottle
A policy consulted before each document and before each write window.
Enums
- ChunkOverflow
Policy for an INPUT unit larger than the budget. Distinct from 6151 (spec 8.2).
- DocumentBlockKind
MEDI's element vocabulary, so the shim is lossless both ways. No image kind (spec 6).
- IngestionDocumentOutcome
The per-document verdict. The same values are persisted as the state row's status.
- IngestionRunOutcome
How a run ended.
- IngestionStage
What a run is doing right now. Drives a foreground notification's text (spec 10.4).
- ThrottlePauseBehavior
What a pause means. Suspend ends the run on a committed boundary; Wait delays and re-evaluates (spec 10.2).