tokenizers-0.20.3
- tokenizers 0.20.3
- Docs.rs crate page
- Apache-2.0
- Links
- Homepage
- Repository
- crates.io
- Source
- Owners
- n1t0
- Narsil
- ArthurZucker
- Dependencies
- aho-corasick ^1.1 normal
  
  derive_builder ^0.20 normal
  
  esaxx-rs ^0.1.10 normal
  
  fancy-regex ^0.13 normal optional
  
  getrandom ^0.2.10 normal
  
  hf-hub ^0.3.2 normal optional
  
  indicatif ^0.17 normal optional
  
  itertools ^0.12 normal
  
  lazy_static ^1.4 normal
  
  log ^0.4 normal
  
  macro_rules_attribute ^0.2.0 normal
  
  monostate ^0.1.12 normal
  
  onig ^6.4 normal optional
  
  paste ^1.0.14 normal
  
  rand ^0.8 normal
  
  rayon ^1.10 normal
  
  rayon-cond ^0.3 normal
  
  regex ^1.10 normal
  
  regex-syntax ^0.8 normal
  
  serde ^1.0 normal
  
  serde_json ^1.0 normal
  
  spm_precompiled ^0.1.3 normal
  
  thiserror ^1.0.49 normal
  
  unicode-normalization-alignments ^0.1 normal
  
  unicode-segmentation ^1.11 normal
  
  unicode_categories ^0.1 normal
  
  assert_approx_eq ^1.1 dev
  
  criterion ^0.5 dev
  
  tempfile ^3.10 dev
  
  tracing ^0.1 dev
  
  tracing-subscriber ^0.3.18 dev
- Versions
- 46.6% of the crate is documented
Go to latest version
Platform
- x86_64-unknown-linux-gnu
Feature flags

Rust

tokenizers0.20.3

Module pre_tokenizers

Module Items

Modules
Enums

In crate tokenizers

Module pre_tokenizers

Modules§

bert
byte_level
delimiter
digits
metaspace
punctuation
sequence
split
unicode_scripts
whitespace

Enums§

PreTokenizerWrapper