Struct datafusion_common::config::OptimizerOptions

#[non_exhaustive]
pub struct OptimizerOptions {

    pub enable_round_robin_repartition: bool,
    pub filter_null_join_keys: bool,
    pub repartition_aggregations: bool,
    pub repartition_file_min_size: usize,
    pub repartition_joins: bool,
    pub allow_symmetric_joins_without_pruning: bool,
    pub repartition_file_scans: bool,
    pub repartition_windows: bool,
    pub repartition_sorts: bool,
    pub bounded_order_preserving_variants: bool,
    pub skip_failed_rules: bool,
    pub max_passes: usize,
    pub top_down_join_key_reordering: bool,
    pub prefer_hash_join: bool,
    pub hash_join_single_partition_threshold: usize,
}

Options related to query optimization

Fields (Non-exhaustive)

Non-exhaustive structs could have additional fields added in future. Therefore, non-exhaustive structs cannot be constructed in external crates using the traditional Struct { .. } syntax; cannot be matched against without a wildcard ..; and struct update syntax will not work.

enable_round_robin_repartition: bool

When set to true, the physical plan optimizer will try to add round robin repartitioning to increase parallelism to leverage more CPU cores

filter_null_join_keys: bool

When set to true, the optimizer will insert filters before a join between a nullable and non-nullable column to filter out nulls on the nullable side. This filter can add additional overhead when the file format does not fully support predicate push down.

repartition_aggregations: bool

Should DataFusion repartition data using the aggregate keys to execute aggregates in parallel using the provided target_partitions level

repartition_file_min_size: usize

Minimum total files size in bytes to perform file scan repartitioning.

repartition_joins: bool

Should DataFusion repartition data using the join keys to execute joins in parallel using the provided target_partitions level

allow_symmetric_joins_without_pruning: bool

Should DataFusion allow symmetric hash joins for unbounded data sources even when its inputs do not have any ordering or filtering If the flag is not enabled, the SymmetricHashJoin operator will be unable to prune its internal buffers, resulting in certain join types - such as Full, Left, LeftAnti, LeftSemi, Right, RightAnti, and RightSemi - being produced only at the end of the execution. This is not typical in stream processing. Additionally, without proper design for long runner execution, all types of joins may encounter out-of-memory errors.

repartition_file_scans: bool

When set to true, file groups will be repartitioned to achieve maximum parallelism. Currently Parquet and CSV formats are supported.

If set to true, all files will be repartitioned evenly (i.e., a single large file might be partitioned into smaller chunks) for parallel scanning. If set to false, different files will be read in parallel, but repartitioning won’t happen within a single file.

repartition_windows: bool

Should DataFusion repartition data using the partitions keys to execute window functions in parallel using the provided target_partitions level

repartition_sorts: bool

Should DataFusion execute sorts in a per-partition fashion and merge afterwards instead of coalescing first and sorting globally. With this flag is enabled, plans in the form below

     "SortExec: [a@0 ASC]",
     "  CoalescePartitionsExec",
     "    RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1",

would turn into the plan below which performs better in multithreaded environments

     "SortPreservingMergeExec: [a@0 ASC]",
     "  SortExec: [a@0 ASC]",
     "    RepartitionExec: partitioning=RoundRobinBatch(8), input_partitions=1",

bounded_order_preserving_variants: bool

When true, DataFusion will opportunistically remove sorts by replacing RepartitionExec with SortPreservingRepartitionExec, and CoalescePartitionsExec with SortPreservingMergeExec, even when the query is bounded.

skip_failed_rules: bool

When set to true, the logical plan optimizer will produce warning messages if any optimization rules produce errors and then proceed to the next rule. When set to false, any rules that produce errors will cause the query to fail

max_passes: usize

Number of times that the optimizer will attempt to optimize the plan

top_down_join_key_reordering: bool

When set to true, the physical plan optimizer will run a top down process to reorder the join keys

prefer_hash_join: bool

When set to true, the physical plan optimizer will prefer HashJoin over SortMergeJoin. HashJoin can work more efficiently than SortMergeJoin but consumes more memory

hash_join_single_partition_threshold: usize

The maximum estimated size in bytes for one input side of a HashJoin will be collected into a single partition

Trait Implementations

impl Clone for OptimizerOptions

fn clone(&self) -> OptimizerOptions

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for OptimizerOptions

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Default for OptimizerOptions

fn default() -> Self

Returns the “default value” for a type.