Explore Library
QuizIntermediate

Subword Tokenization

BPE and WordPiece split rare words into reusable subword pieces.

How does subword tokenization (e.g. BPE or WordPiece) help with rare or unknown words?