Struct InferencingParams

#[repr(C)]
pub struct InferencingParams {
    pub max_tokens: u32,
    pub repeat_penalty: f32,
    pub repeat_penalty_last_n_token_count: u32,
    pub temperature: f32,
    pub top_k: u32,
    pub top_p: f32,
}

Inference request parameters

Fields

max_tokens: u32

The maximum tokens that should be inferred.

Note: the backing implementation may return less tokens.

repeat_penalty: f32

The amount the model should avoid repeating tokens.

repeat_penalty_last_n_token_count: u32

The number of tokens the model should apply the repeat penalty to.

temperature: f32

The randomness with which the next token is selected.

top_k: u32

The number of possible next tokens the model will choose from.

top_p: f32

The probability total of next tokens the model will choose from.

Trait Implementations

impl Clone for InferencingParams

fn clone(&self) -> InferencingParams

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for InferencingParams

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for InferencingParams

fn default() -> Self

Returns the “default value” for a type.

impl Copy for InferencingParams