Struct MemoryExec

pub struct MemoryExec { /* private fields */ }

Execution plan for reading in-memory batches of data

Implementations

impl MemoryExec

pub fn try_new( partitions: &[Vec<RecordBatch>], schema: SchemaRef, projection: Option<Vec<usize>>, ) -> Result<Self>

Create a new execution plan for reading in-memory record batches The provided schema should not have the projection applied.

pub fn try_new_as_values( schema: SchemaRef, data: Vec<Vec<Arc<dyn PhysicalExpr>>>, ) -> Result<Self>

Create a new execution plan from a list of constant values (ValuesExec)

pub fn try_new_from_batches( schema: SchemaRef, batches: Vec<RecordBatch>, ) -> Result<Self>

Create a new plan using the provided schema and batches.

Errors if any of the batches don’t match the provided schema, or if no batches are provided.

pub fn with_constraints(self, constraints: Constraints) -> Self

pub fn with_show_sizes(self, show_sizes: bool) -> Self

Set show_sizes to determine whether to display partition sizes

pub fn constraints(&self) -> &Constraints

Ref to constraints

pub fn partitions(&self) -> &[Vec<RecordBatch>]

Ref to partitions

pub fn projection(&self) -> &Option<Vec<usize>>

Ref to projection

pub fn show_sizes(&self) -> bool

Show sizes

pub fn sort_information(&self) -> &[LexOrdering]

Ref to sort information

pub fn try_with_sort_information( self, sort_information: Vec<LexOrdering>, ) -> Result<Self>

A memory table can be ordered by multiple expressions simultaneously. EquivalenceProperties keeps track of expressions that describe the global ordering of the schema. These columns are not necessarily same; e.g.

┌-------┐
| a | b |
|---|---|
| 1 | 9 |
| 2 | 8 |
| 3 | 7 |
| 5 | 5 |
└---┴---┘

where both a ASC and b DESC can describe the table ordering. With EquivalenceProperties, we can keep track of these equivalences and treat a ASC and b DESC as the same ordering requirement.

Note that if there is an internal projection, that projection will be also applied to the given sort_information.

pub fn original_schema(&self) -> SchemaRef

Arc clone of ref to original schema

Trait Implementations

impl Clone for MemoryExec

fn clone(&self) -> MemoryExec

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for MemoryExec

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl DisplayAs for MemoryExec

fn fmt_as(&self, t: DisplayFormatType, f: &mut Formatter<'_>) -> Result

Format according to DisplayFormatType, used when verbose representation looks different from the default one

impl ExecutionPlan for MemoryExec

fn as_any(&self) -> &dyn Any

Return a reference to Any that can be used for downcasting

fn statistics(&self) -> Result<Statistics>

We recompute the statistics dynamically from the arrow metadata as it is pretty cheap to do so

fn name(&self) -> &'static str

Short name for the ExecutionPlan, such as ‘ParquetExec’.

fn properties(&self) -> &PlanProperties

Return properties of the output of the ExecutionPlan, such as output ordering(s), partitioning information etc.

fn children(&self) -> Vec<&Arc<dyn ExecutionPlan>>

Get a list of children ExecutionPlans that act as inputs to this plan. The returned list will be empty for leaf nodes such as scans, will contain a single value for unary nodes, or two values for binary nodes (such as joins).

fn with_new_children( self: Arc<Self>, children: Vec<Arc<dyn ExecutionPlan>>, ) -> Result<Arc<dyn ExecutionPlan>>

Returns a new ExecutionPlan where all existing children were replaced by the children, in order

fn execute( &self, partition: usize, _context: Arc<TaskContext>, ) -> Result<SendableRecordBatchStream>

Begin execution of partition, returning a Stream of RecordBatches.

fn try_swapping_with_projection( &self, projection: &ProjectionExec, ) -> Result<Option<Arc<dyn ExecutionPlan>>>

Attempts to push down the given projection into the input of this ExecutionPlan.

fn static_name() -> &'static str

where Self: Sized,

Short name for the ExecutionPlan, such as ‘ParquetExec’. Like name but can be called without an instance.

fn schema(&self) -> SchemaRef

Get the schema for this execution plan

fn check_invariants(&self, _check: InvariantLevel) -> Result<()>

Returns an error if this individual node does not conform to its invariants. These invariants are typically only checked in debug mode.

fn required_input_distribution(&self) -> Vec<Distribution>

Specifies the data distribution requirements for all the children for this ExecutionPlan, By default it’s [Distribution::UnspecifiedDistribution] for each child,

fn required_input_ordering(&self) -> Vec<Option<LexRequirement>>

Specifies the ordering required for all of the children of this ExecutionPlan.

fn maintains_input_order(&self) -> Vec<bool>

Returns false if this ExecutionPlan’s implementation may reorder rows within or between partitions.

fn benefits_from_input_partitioning(&self) -> Vec<bool>

Specifies whether the ExecutionPlan benefits from increased parallelization at its input for each child.

fn repartitioned( &self, _target_partitions: usize, _config: &ConfigOptions, ) -> Result<Option<Arc<dyn ExecutionPlan>>>

If supported, attempt to increase the partitioning of this ExecutionPlan to produce target_partitions partitions.

fn metrics(&self) -> Option<MetricsSet>

Return a snapshot of the set of Metrics for this ExecutionPlan. If no Metrics are available, return None.

fn supports_limit_pushdown(&self) -> bool

Returns true if a limit can be safely pushed down through this ExecutionPlan node.

fn with_fetch(&self, _limit: Option<usize>) -> Option<Arc<dyn ExecutionPlan>>

Returns a fetching variant of this ExecutionPlan node, if it supports fetch limits. Returns None otherwise.

fn fetch(&self) -> Option<usize>

Gets the fetch count for the operator, None means there is no fetch.

fn cardinality_effect(&self) -> CardinalityEffect

Gets the effect on cardinality, if known