pub trait TokenSource {
type Token: Hash + Eq;
type Tokenizer: Iterator<Item = Self::Token>;
// Required methods
fn tokenize(&self) -> Self::Tokenizer;
fn estimate_tokens(&self) -> u32;
}
Available on crate feature
blob
only.Required Associated Types§
Required Methods§
Implementations on Foreign Types§
source§impl<'a> TokenSource for &'a str
impl<'a> TokenSource for &'a str
By default a line diff is produced for a string
source§impl<'a> TokenSource for &'a [u8]
impl<'a> TokenSource for &'a [u8]
By default a line diff is produced for a bytes
Implementors§
source§impl<'a, const INCLUDE_LINE_TERMINATOR: bool> TokenSource for ByteLines<'a, INCLUDE_LINE_TERMINATOR>
impl<'a, const INCLUDE_LINE_TERMINATOR: bool> TokenSource for ByteLines<'a, INCLUDE_LINE_TERMINATOR>
By default a line diff is produced for a string