Trait tantivy_tokenizer_api::TokenFilter

pub trait TokenFilter: 'static + Send + Sync {
    type Tokenizer<T: Tokenizer>: Tokenizer;

    // Required method
    fn transform<T: Tokenizer>(self, tokenizer: T) -> Self::Tokenizer<T>;
}

Trait for the pluggable components of Tokenizers.

Required Associated Types

type Tokenizer<T: Tokenizer>: Tokenizer

The Tokenizer type returned by this filter, typically parametrized by the underlying Tokenizer.

Required Methods

fn transform<T: Tokenizer>(self, tokenizer: T) -> Self::Tokenizer<T>

Wraps a Tokenizer and returns a new one.

Object Safety

This trait is not object safe.

Implementors