Enum Partitioning

pub enum Partitioning {
    RoundRobinBatch(usize),
    Hash(Vec<Arc<dyn PhysicalExpr>>, usize),
    UnknownPartitioning(usize),
}

Output partitioning supported by ExecutionPlans.

Calling ExecutionPlan::execute produce one or more independent streams of RecordBatches in parallel, referred to as partitions. The streams are Rust async Streams (a special kind of future). The number of output partitions varies based on the input and the operation performed.

For example, an ExecutionPlan that has output partitioning of 3 will produce 3 distinct output streams as the result of calling ExecutionPlan::execute(0), ExecutionPlan::execute(1), and ExecutionPlan::execute(2), as shown below:

                                                  ...         ...        ...
              ...                                  ▲           ▲           ▲
                                                   │           │           │
               ▲                                   │           │           │
               │                                   │           │           │
               │                               ┌───┴────┐  ┌───┴────┐  ┌───┴────┐
    ┌────────────────────┐                     │ Stream │  │ Stream │  │ Stream │
    │   ExecutionPlan    │                     │  (0)   │  │  (1)   │  │  (2)   │
    └────────────────────┘                     └────────┘  └────────┘  └────────┘
               ▲                                   ▲           ▲           ▲
               │                                   │           │           │
    ┌ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─                          │           │           │
            Input        │                         │           │           │
    └ ─ ─ ─ ─ ─ ─ ─ ─ ─ ─                          │           │           │
               ▲                               ┌ ─ ─ ─ ─   ┌ ─ ─ ─ ─   ┌ ─ ─ ─ ─
               │                                 Input  │    Input  │    Input  │
               │                               │ Stream    │ Stream    │ Stream
                                                  (0)   │     (1)   │     (2)   │
              ...                              └ ─ ▲ ─ ─   └ ─ ▲ ─ ─   └ ─ ▲ ─ ─
                                                   │           │           │
                                                   │           │           │
                                                   │           │           │

ExecutionPlan with 1 input                      3 (async) streams, one for each
that has 3 partitions, which itself             output partition
has 3 output partitions

It is common (but not required) that an ExecutionPlan has the same number of input partitions as output partitions. However, some plans have different numbers such as the RepartitionExec that redistributes batches from some number of inputs to some number of outputs

              ...                                     ...         ...        ...

                                                       ▲           ▲           ▲
               ▲                                       │           │           │
               │                                       │           │           │
      ┌────────┴───────────┐                           │           │           │
      │  RepartitionExec   │                      ┌────┴───┐  ┌────┴───┐  ┌────┴───┐
      └────────────────────┘                      │ Stream │  │ Stream │  │ Stream │
               ▲                                  │  (0)   │  │  (1)   │  │  (2)   │
               │                                  └────────┘  └────────┘  └────────┘
               │                                       ▲           ▲           ▲
               ...                                     │           │           │
                                                       └──────────┐│┌──────────┘
                                                                  │││
                                                                  │││
RepartitionExec with 1 input
partition and 3 output partitions                 3 (async) streams, that internally
                                                   pull from the same input stream
                                                                 ...

Additional Examples

A simple FileScanExec might produce one output stream (partition) for each file (note the actual DataFusion file scaners can read individual files in parallel, potentially producing multiple partitions per file)

Plans such as SortPreservingMerge produce a single output stream (1 output partition) by combining some number of input streams (input partitions)

Plans such as FilterExec produce the same number of output streams (partitions) as input streams (partitions).

Variants

RoundRobinBatch(usize)

Allocate batches using a round-robin algorithm and the specified number of partitions

Hash(Vec<Arc<dyn PhysicalExpr>>, usize)

Allocate rows based on a hash of one of more expressions and the specified number of partitions

UnknownPartitioning(usize)

Unknown partitioning scheme with a known number of partitions

Implementations

impl Partitioning

pub fn partition_count(&self) -> usize

Returns the number of partitions in this partitioning scheme

pub fn satisfy( &self, required: &Distribution, eq_properties: &EquivalenceProperties, ) -> bool

Returns true when the guarantees made by this Partitioning are sufficient to satisfy the partitioning scheme mandated by the required Distribution.

pub fn project( &self, projection_mapping: &ProjectionMapping, input_eq_properties: &EquivalenceProperties, ) -> Self

Calculate the output partitioning after applying the given projection.

Trait Implementations

impl Clone for Partitioning

fn clone(&self) -> Partitioning

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for Partitioning

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for Partitioning

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for Partitioning

fn eq(&self, other: &Partitioning) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.