Trait Tokenizer

pub trait Tokenizer {
    // Required method
    fn tokenize<'a>(&self, text: &'a str) -> Vec<&'a str>;
}

A trait that all Tokenizer should implement.

Required Methods

fn tokenize<'a>(&self, text: &'a str) -> Vec<&'a str>

Tokenize given text and return a Vec<&str> where each &str inside a Vec is a slice from given text.

Implementors

impl Tokenizer for tokenizer::en::Tokenizer

impl Tokenizer for tokenizer::th::Tokenizer