tokenizers 0.20.3

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Documentation 
1
2
3
4
5
6

{
  "git": {
    "sha1": "b63262a481495453011bf7a3b27995d6611dfae7"
  },
  "path_in_vcs": "tokenizers"
}