FlashcardIntermediate
Tokenization (Subword/BPE)
Tokenization splits text into subword units so models can handle any word efficiently.
Question
What is subword tokenization (e.g., BPE) in LLMs?
Click to reveal answer
Tokenization splits text into subword units so models can handle any word efficiently.
What is subword tokenization (e.g., BPE) in LLMs?