Class WordPieceTokenizerOptions
- Namespace
- Qavren.Edge.Embeddings.Onnx
- Assembly
- Qavren.Edge.Embeddings.Onnx.dll
WordPiece construction settings. Every default here changes behaviour.
public sealed class WordPieceTokenizerOptions- Inheritance
-
WordPieceTokenizerOptions
- Inherited Members
Remarks
The four special-token strings are the bert-base-uncased spellings and are looked up in
the vocabulary BY STRING to get their ids. All four shipped presets share the identical
30,522-entry vocab, so all four resolve to the same four ids. Getting one wrong does not throw:
an [UNK]-token name absent from the vocab resolves to no id and the tokenizer emits a
sequence with no unknown marker, which is a silent quality bug of exactly the class every
required field on EmbeddingPreset exists to prevent.
Properties
- ClassificationToken
The classification-token spelling, prepended to every sequence.
- LowerCase
Lower-case before tokenization. True by default and OVERRIDDEN PER PRESET from LowerCase, never left at the default: a cased model fed lower-cased text produces plausible, wrong vectors.
- MaxSequenceLength
The longest sequence this tokenizer will emit, special tokens included.
- PaddingToken
The padding-token spelling. Padded positions carry its id.
- RemoveNonSpacingMarks
Strip non-spacing marks. False by default, which matches
bert-base-uncased: it strips accents through its own normaliser rather than this flag.
- SeparatorToken
The separator-token spelling, appended to every sequence.
- UnknownToken
The unknown-token spelling.