Trait tantivy_tokenizer_api::Tokenizer

pub trait Tokenizer: 'static + Clone + Send + Sync {
    type TokenStream<'a>: TokenStream;

    // Required method
    fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>;
}

Tokenizer are in charge of splitting text into a stream of token before indexing.

Required Associated Types

type TokenStream<'a>: TokenStream

The token stream returned by this Tokenizer.

Required Methods

fn token_stream<'a>(&'a mut self, text: &'a str) -> Self::TokenStream<'a>

Creates a token stream for a given str.

Object Safety

This trait is not object safe.

Implementors