tokenizers 0.21.0

Provides an implementation of today's most used tokenizers, with a focus on performances and versatility.
Documentation
Build #1565914 2024-11-27T13:31:10.733719+00:00
# rustc version
rustc 1.85.0-nightly (dff3e7ccd 2024-11-26)# docs.rs version
docsrs 0.6.0 (36c721fb 2024-11-06)# build log
[INFO] running `Command { std: "docker" "create" "-v" "/home/cratesfyi/workspace/builds/tokenizers-0.21.0/target:/opt/rustwide/target:rw,Z" "-v" "/home/cratesfyi/workspace/builds/tokenizers-0.21.0/source:/opt/rustwide/workdir:ro,Z" "-v" "/home/cratesfyi/workspace/cargo-home:/opt/rustwide/cargo-home:ro,Z" "-v" "/home/cratesfyi/workspace/rustup-home:/opt/rustwide/rustup-home:ro,Z" "-e" "SOURCE_DIR=/opt/rustwide/workdir" "-e" "CARGO_TARGET_DIR=/opt/rustwide/target" "-e" "DOCS_RS=1" "-e" "CARGO_HOME=/opt/rustwide/cargo-home" "-e" "RUSTUP_HOME=/opt/rustwide/rustup-home" "-w" "/opt/rustwide/workdir" "-m" "6442450944" "--cpus" "6" "--user" "1001:1001" "--network" "none" "ghcr.io/rust-lang/crates-build-env/linux@sha256:4a844ea9eb2546a2d2c7022eacef16ef2e8229c7fbb2c7d4d55a9ceca922f72d" "/opt/rustwide/cargo-home/bin/cargo" "+nightly" "rustdoc" "--lib" "-Zrustdoc-map" "--config" "build.rustdocflags=[\"--cfg\", \"docsrs\", \"-Z\", \"unstable-options\", \"--emit=invocation-specific\", \"--resource-suffix\", \"-20241126-1.85.0-nightly-dff3e7ccd\", \"--static-root-path\", \"/-/rustdoc.static/\", \"--cap-lints\", \"warn\", \"--extern-html-root-takes-precedence\"]" "--offline" "-Zunstable-options" "--config=doc.extern-map.registries.crates-io=\"https://docs.rs/{pkg_name}/{version}/x86_64-unknown-linux-gnu\"" "-Zrustdoc-scrape-examples" "-j6" "--target" "x86_64-unknown-linux-gnu", kill_on_drop: false }`
[INFO] [stderr] WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap.
[INFO] [stdout] a78331beb26164f3c0737ab4b3611d1fd86d94c87629fa00dd8cc915b8fd157e
[INFO] running `Command { std: "docker" "start" "-a" "a78331beb26164f3c0737ab4b3611d1fd86d94c87629fa00dd8cc915b8fd157e", kill_on_drop: false }`
[INFO] [stderr] warning: Rustdoc did not scrape the following examples because they require dev-dependencies: encode_batch, serialization
[INFO] [stderr]     If you want Rustdoc to scrape these examples, then add `doc-scrape-examples = true`
[INFO] [stderr]     to the [[example]] target configuration of at least one example.
[INFO] [stderr] warning: target filter specified, but no targets matched; this is a no-op
[INFO] [stderr]  Documenting tokenizers v0.21.0 (/opt/rustwide/workdir)
[INFO] [stderr]     Finished `dev` profile [unoptimized + debuginfo] target(s) in 1.60s
[INFO] [stderr]    Generated /opt/rustwide/target/x86_64-unknown-linux-gnu/doc/tokenizers/index.html
[INFO] running `Command { std: "docker" "inspect" "a78331beb26164f3c0737ab4b3611d1fd86d94c87629fa00dd8cc915b8fd157e", kill_on_drop: false }`
[INFO] running `Command { std: "docker" "rm" "-f" "a78331beb26164f3c0737ab4b3611d1fd86d94c87629fa00dd8cc915b8fd157e", kill_on_drop: false }`
[INFO] [stdout] a78331beb26164f3c0737ab4b3611d1fd86d94c87629fa00dd8cc915b8fd157e