Struct BatchCoalescer

pub struct BatchCoalescer { /* private fields */ }

Concatenate multiple RecordBatches

BatchCoalescer concatenates multiple small RecordBatches, produced by operations such as FilterExec and RepartitionExec, into larger ones for more efficient processing by subsequent operations.

Background

Generally speaking, larger RecordBatches are more efficient to process than smaller record batches (until the CPU cache is exceeded) because there is fixed processing overhead per batch. DataFusion tries to operate on batches of target_batch_size rows to amortize this overhead

┌────────────────────┐
│    RecordBatch     │
│   num_rows = 23    │
└────────────────────┘                 ┌────────────────────┐
                                       │                    │
┌────────────────────┐     Coalesce    │                    │
│                    │      Batches    │                    │
│    RecordBatch     │                 │                    │
│   num_rows = 50    │  ─ ─ ─ ─ ─ ─ ▶  │                    │
│                    │                 │    RecordBatch     │
│                    │                 │   num_rows = 106   │
└────────────────────┘                 │                    │
                                       │                    │
┌────────────────────┐                 │                    │
│                    │                 │                    │
│    RecordBatch     │                 │                    │
│   num_rows = 33    │                 └────────────────────┘
│                    │
└────────────────────┘

Notes:

1. Output rows are produced in the same order as the input rows

2. The output is a sequence of batches, with all but the last being at least target_batch_size rows.

3. Eventually this may also be able to handle other optimizations such as a combined filter/coalesce operation.

Implementations

impl BatchCoalescer

pub fn new( schema: SchemaRef, target_batch_size: usize, fetch: Option<usize>, ) -> Self

Create a new BatchCoalescer

Arguments

• schema - the schema of the output batches
• target_batch_size - the minimum number of rows for each output batch (until limit reached)
• fetch - the maximum number of rows to fetch, None means fetch all rows

pub fn schema(&self) -> SchemaRef

Return the schema of the output batches

pub fn push_batch(&mut self, batch: RecordBatch) -> CoalescerState

Push next batch, and returns CoalescerState indicating the current state of the buffer.

pub fn is_empty(&self) -> bool

Return true if the there is no data buffered

pub fn finish_batch(&mut self) -> Result<RecordBatch>

Concatenates and returns all buffered batches, and clears the buffer.

Trait Implementations

impl Debug for BatchCoalescer

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.