tokenizers 0.21.0

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Documentation