Explore Library
Code QuizIntermediate

Tokenizing a Sentence into Words

Spot why this word tokenizer returns one giant token instead of separate words.

Codepython
text = "Natural language processing is fun"
tokens = text.split(",")
print(tokens)

Why does this word tokenizer fail to split the sentence into words?