Struct datafusion_common::config::ExecutionOptions

#[non_exhaustive]
pub struct ExecutionOptions {
    pub batch_size: usize,
    pub coalesce_batches: bool,
    pub collect_statistics: bool,
    pub target_partitions: usize,
    pub time_zone: Option<String>,
    pub parquet: ParquetOptions,
    pub aggregate: AggregateOptions,
    pub planning_concurrency: usize,
    pub sort_spill_reservation_bytes: usize,
    pub sort_in_place_threshold_bytes: usize,
}

Options related to query execution

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

batch_size: usize

Default batch size while creating new batches, it’s especially useful for buffer-in-memory batches since creating tiny batches would result in too much metadata memory consumption

coalesce_batches: bool

When set to true, record batches will be examined between each operator and small batches will be coalesced into larger batches. This is helpful when there are highly selective filters or joins that could produce tiny output batches. The target batch size is determined by the configuration setting

collect_statistics: bool

Should DataFusion collect statistics after listing files

target_partitions: usize

Number of partitions for query execution. Increasing partitions can increase concurrency.

Defaults to the number of CPU cores on the system

time_zone: Option<String>

The default time zone

Some functions, e.g. EXTRACT(HOUR from SOME_TIME), shift the underlying datetime according to this time zone, and then extract the hour

parquet: ParquetOptions

Parquet options

aggregate: AggregateOptions

Aggregate options

planning_concurrency: usize

Fan-out during initial physical planning.

This is mostly use to plan UNION children in parallel.

Defaults to the number of CPU cores on the system

sort_spill_reservation_bytes: usize

Specifies the reserved memory for each spillable sort operation to facilitate an in-memory merge.

When a sort operation spills to disk, the in-memory data must be sorted and merged before being written to a file. This setting reserves a specific amount of memory for that in-memory sort/merge process.

Note: This setting is irrelevant if the sort operation cannot spill (i.e., if there’s no DiskManager configured).

sort_in_place_threshold_bytes: usize

When sorting, below what size should data be concatenated and sorted in a single RecordBatch rather than sorted in batches and merged.

Trait Implementations

impl Clone for ExecutionOptions

fn clone(&self) -> ExecutionOptions

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for ExecutionOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for ExecutionOptions

fn default() -> Self

Returns the “default value” for a type.