Crate datafusion_physical_plan

Traits for physical query plan, supporting parallel execution for partitioned relations.

Entrypoint of this crate is trait ExecutionPlan.

Re-exports

pub use crate::display::DefaultDisplay;

pub use crate::display::DisplayAs;

pub use crate::display::DisplayFormatType;

pub use crate::display::VerboseDisplay;

pub use crate::execution_plan::collect;

pub use crate::execution_plan::collect_partitioned;

pub use crate::execution_plan::displayable;

pub use crate::execution_plan::execute_input_stream;

pub use crate::execution_plan::execute_stream;

pub use crate::execution_plan::execute_stream_partitioned;

pub use crate::execution_plan::get_plan_string;

pub use crate::execution_plan::with_new_children_if_necessary;

pub use crate::execution_plan::ExecutionPlan;

pub use crate::execution_plan::ExecutionPlanProperties;

pub use crate::execution_plan::PlanProperties;

pub use crate::metrics::Metric;

pub use crate::stream::EmptyRecordBatchStream;

Modules

aggregates

Aggregates functionalities

analyze

Defines the ANALYZE operator

coalesce

coalesce_batches

CoalesceBatchesExec combines small batches into larger batches.

coalesce_partitions

Defines the merge plan for executing partitions in parallel and then merging the results into a single partition

common

Defines common code used in execution plans

display

Implementation of physical plan display. See crate::displayable for examples of how to format

empty

EmptyRelation with produce_one_row=false execution plan

execution_plan

explain

Defines the EXPLAIN operator

expressions

Defines physical expressions that can evaluated at runtime during query execution

filter

hash_utils

Functionality used both on logical and physical plans

insert

Execution plan for writing data to DataSinks

joins

DataFusion Join implementations

limit

Defines the LIMIT plan

memory

Execution plan for reading in-memory batches of data

metrics

Metrics for recording information about execution

placeholder_row

EmptyRelation produce_one_row=true execution plan

projection

Defines the projection execution plan. A projection determines which columns or expressions are returned from a query. The SQL statement SELECT a, b, a+b FROM t1 is an example of a projection on table t1 where the expressions a, b, and a+b are the projection expressions. SELECT without FROM will only evaluate expressions.

recursive_query

Defines the recursive query plan

repartition

This file implements the RepartitionExec operator, which maps N input partitions to M output partitions based on a partitioning scheme, optionally maintaining the order of the input rows in the output.

sorts

Sort functionalities

spill

Defines the spilling functions

stream

Stream wrappers for physical operators

streaming

Generic plans for deferred execution: StreamingTableExec and PartitionStream

tree_node

This module provides common traits for visiting or rewriting tree nodes easily.

udaf

udf

union

The Union operator combines multiple inputs with the same schema

unnest

Define a plan for unnesting values in columns that contain a list type.

values

Values execution plan

windows

Physical expressions for window functions

work_table

Defines the work table query plan

Macros

handle_state

The handle_state macro is designed to process the result of a state-changing operation. It operates on a StatefulStreamResult by matching its variants and executing corresponding actions. This macro is used to streamline code that deals with state transitions, reducing boilerplate and improving readability.

internal_err

Macro wraps Err($ERR) to add backtrace feature

Structs

ColumnStatistics

Statistics for a column within a relation

Statistics

Statistics for a relation Fields are optional and can be inexact because the sources sometimes provide approximate estimates for performance reasons and the transformations output are not always predictable.

TopK

Global TopK

Enums

ColumnarValue

The result of evaluating an expression.

Distribution

How data is distributed amongst partitions. See Partitioning for more details.

InputOrderMode

Specifies how the input to an aggregation or window operator is ordered relative to their GROUP BY or PARTITION BY expressions.

Partitioning

Output partitioning supported by ExecutionPlans.

Traits

Accumulator

Tracks an aggregate function’s state.

ExecutionPlanVisitor

Trait that implements the Visitor pattern for a depth first walk of ExecutionPlan nodes. pre_visit is called before any children are visited, and then post_visit is called after all children have been visited.

PhysicalExpr

PhysicalExprs represent expressions such as A + 1 or CAST(c1 AS int).

RecordBatchStream

Trait for types that stream RecordBatch

WindowExpr

Common trait for window function implementations

Functions

accept

Visit all children of this plan, according to the order defined on ExecutionPlanVisitor.

project_schema

Applies an optional projection to a SchemaRef, returning the projected schema

visit_execution_plan

Recursively calls pre_visit and post_visit for this node and all of its children, as described on ExecutionPlanVisitor

Type Aliases

SendableRecordBatchStream

Trait for a Stream of RecordBatches that can be passed between threads