Struct PipeLine

pub struct PipeLine { /* private fields */ }

A pipeline consists of:

• 1. One or more sources. Sources get pulled and their data is pushed into operators.
• 2. Zero or more operators. The operators simply pass through data, modifying it as they need. Operators can work on batches and don’t need all data in scope to succeed. Think for example on multiply a few columns, or applying a predicate. Operators can shrink the batches: filter Grow the batches: explode/ unpivot Keep them the same size: element-wise operations The probe side of join operations is also an operator.
• 3. One or more sinks A sink needs all data in scope to finalize a pipeline branch. Think of sorts, preparing a build phase of a join, group_by + aggregations.

This struct will have the SOS (source, operators, sinks) of its own pipeline branch, but also the SOS of other branches. The SOS are stored data oriented and the sinks have an offset that indicates the last operator node before that specific sink. We only store the end offset and keep track of the starting operator during execution.

Pipelines branches are shared with other pipeline branches at the join/union nodes.

JOIN

Consider this tree: out / /
1 2

And let’s consider that branch 2 runs first. It will run until the join node where it will sink into a build table. Once that is done it will replace the build-phase placeholder operator in branch 1. Branch one can then run completely until out.

Implementations

impl PipeLine

pub fn run_pipeline( &mut self, ec: &PExecutionContext, pipelines: &mut Vec<PipeLine>, ) -> PolarsResult<Option<FinalizedSink>>

Run a single pipeline branch. This pulls data from the sources and pushes it into the operators which run on a different thread and finalize in a sink.

The sink can be finished, but can also become a new source and then rinse and repeat.

Trait Implementations

impl Debug for PipeLine

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Send for PipeLine

impl Sync for PipeLine