Struct Executor

pub struct Executor { /* private fields */ }

Executor is responsible for running the LLAMA model and managing its context.

Trait Implementations

impl Executor for Executor

Implement the ExecutorTrait for the Executor, defining methods for handling input and output.

type StepTokenizer<'a> = LLamaTokenizer<'a>

fn new_with_options(options: Options) -> Result<Executor, ExecutorCreationError>

Create a new executor with the given options. If you don’t need to set any options, you can use the new method instead.

fn execute<'life0, 'life1, 'life2, 'async_trait>( &'life0 self, options: &'life1 Options, prompt: &'life2 Prompt, ) -> Pin<Box<dyn Future<Output = Result<Output, ExecutorError>> + Send + 'async_trait>>

where Self: 'async_trait, 'life0: 'async_trait, 'life1: 'async_trait, 'life2: 'async_trait,

fn tokens_used( &self, options: &Options, prompt: &Prompt, ) -> Result<TokenCount, PromptTokensError>

Calculates the number of tokens used by the step given a set of parameters.

fn answer_prefix(&self, prompt: &Prompt) -> Option<String>

Returns a possible answer prefix inserted by the model, during a certain prompt mode

fn max_tokens_allowed(&self, _step: &Options) -> i32

Returns the maximum number of input tokens allowed by the model used.

fn get_tokenizer( &self, _step: &Options, ) -> Result<LLamaTokenizer<'_>, TokenizerError>

Creates a tokenizer, depending on the model used by step.

fn new() -> Result<Self, ExecutorCreationError>