Module common

re-export of datafusion_common crate

Modules

alias

arrow

A complete, safe, native Rust implementation of Apache Arrow, a cross-language development platform for in-memory data.

cast

This module provides DataFusion specific casting functions that provide error handling. They are intended to “never fail” but provide an error message rather than a panic, as the corresponding kernels in arrow-rs such as as_boolean_array do.

config

Runtime configuration, via ConfigOptions

cse

Common Subexpression Elimination logic implemented in CSE can be controlled with a CSEController, that defines how to eliminate common subtrees from a particular TreeNode tree.

display

Types for plan display

error

DataFusion error types

file_options

Options related to how files should be written

format

hash_utils

Functionality used both on logical and physical plans

instant

WASM-compatible Instant wrapper.

parsers

Interval parsing logic

rounding

Floating point rounding mode utility library TODO: Remove this custom implementation and the “libc” dependency when floating-point rounding mode manipulation functions become available in Rust.

runtime

re-export of datafusion_common_runtime crate

scalar

ScalarValue: stores single values

stats

This module provides data structures to represent statistics

test_util

Utility functions to make testing DataFusion based crates easier

tree_node

TreeNode for visiting and rewriting expression and plan trees

types

utils

This module provides the bisect function, which implements binary search.

Macros

_config_datafusion_err

Macro wraps $ERR to add backtrace feature

_exec_datafusion_err

Macro wraps $ERR to add backtrace feature

_internal_datafusion_err

Macro wraps $ERR to add backtrace feature

_not_impl_datafusion_err

Macro wraps $ERR to add backtrace feature

_plan_datafusion_err

Macro wraps $ERR to add backtrace feature

_resources_datafusion_err

Macro wraps $ERR to add backtrace feature

_substrait_datafusion_err

Macro wraps $ERR to add backtrace feature

arrow_datafusion_err

arrow_err

assert_batches_eq

Compares formatted output of a record batch with an expected vector of strings, with the result of pretty formatting record batches. This is a macro so errors appear on the correct line

assert_batches_sorted_eq

Compares formatted output of a record batch with an expected vector of strings in a way that order does not matter. This is a macro so errors appear on the correct line

assert_contains

A macro to assert that one string is contained within another with a nice error message if they are not.

assert_not_contains

A macro to assert that one string is NOT contained within another with a nice error message if they are are.

config_datafusion_err

Macro wraps $ERR to add backtrace feature

config_err

Macro wraps Err($ERR) to add backtrace feature

config_field

context

create_array

downcast_value

Downcast an Arrow Array to a concrete type, return an DataFusionError::Internal if the cast is not possible. In normal usage of DataFusion the downcast should always succeed.

exec_datafusion_err

Macro wraps $ERR to add backtrace feature

exec_err

Macro wraps Err($ERR) to add backtrace feature

extensions_options

Convenience macro to create ExtensionsOptions.

internal_datafusion_err

Macro wraps $ERR to add backtrace feature

internal_err

Macro wraps Err($ERR) to add backtrace feature

not_impl_datafusion_err

Macro wraps $ERR to add backtrace feature

not_impl_err

Macro wraps Err($ERR) to add backtrace feature

plan_datafusion_err

Macro wraps $ERR to add backtrace feature

plan_err

Macro wraps Err($ERR) to add backtrace feature

record_batch

Creates a record batch from literal slice of values, suitable for rapid testing and development.

resources_datafusion_err

Macro wraps $ERR to add backtrace feature

resources_err

Macro wraps Err($ERR) to add backtrace feature

schema_datafusion_err

schema_err

sql_datafusion_err

sql_err

substrait_datafusion_err

Macro wraps $ERR to add backtrace feature

substrait_err

Macro wraps Err($ERR) to add backtrace feature

unwrap_or_internal_err

Unwrap an Option if possible. Otherwise return an DataFusionError::Internal. In normal usage of DataFusion the unwrap should always succeed.

Structs

Column

A named reference to a qualified field in a schema.

ColumnStatistics

Statistics for a column within a relation

Constraints

This object encapsulates a list of functional constraints:

DFSchema

DFSchema wraps an Arrow schema and adds relation names.

FunctionalDependence

This object defines a functional dependence in the schema. A functional dependence defines a relationship between determinant keys and dependent columns. A determinant key is a column, or a set of columns, whose value uniquely determines values of some other (dependent) columns. If two rows have the same determinant key, dependent columns in these rows are necessarily the same. If the determinant key is unique, the set of dependent columns is equal to the entire schema and the determinant key can serve as a primary key. Note that a primary key may “downgrade” into a determinant key due to an operation such as a join, and this object is used to track dependence relationships in such cases. For more information on functional dependencies, see: https://www.scaler.com/topics/dbms/functional-dependency-in-dbms/

FunctionalDependencies

This object encapsulates all functional dependencies in a given relation.

RecursionUnnestOption

Instruction on how to unnest a column (mostly with a list type) such as how to name the output, and how many level it should be unnested

ResolvedTableReference

A fully resolved path to a table of the form “catalog.schema.table”

Statistics

Statistics for a relation Fields are optional and can be inexact because the sources sometimes provide approximate estimates for performance reasons and the transformations output are not always predictable.

UnnestOptions

Options for unnesting a column that contains a list type, replicating values in the other, non nested rows.

Enums

Constraint

This object defines a constraint on a table.

DataFusionError

DataFusion error

Dependency

Describes functional dependency mode.

JoinConstraint

Join constraint

JoinSide

Join side. Stores the referred table side during calculations

JoinType

Join type

ParamValues

The parameter value corresponding to the placeholder

ScalarValue

A dynamically typed, nullable single value.

SchemaError

Schema-related errors

SchemaReference

TableReference

A multi part identifier (path) to a table that may require further resolution (e.g. foo.bar).

Constants

DEFAULT_ARROW_EXTENSION

The default file extension of arrow files

DEFAULT_AVRO_EXTENSION

The default file extension of avro files

DEFAULT_CSV_EXTENSION

The default file extension of csv files

DEFAULT_JSON_EXTENSION

The default file extension of json files

DEFAULT_PARQUET_EXTENSION

The default file extension of parquet files

Traits

ExprSchema

Provides schema information needed by certain methods of Expr (defined in the datafusion-common crate).

GetExt

Define each FileType/FileCompressionType’s extension

ScalarType

Trait used to map a NativeType to a ScalarValue

SchemaExt

DataFusion-specific extensions to Schema.

ToDFSchema

Convenience trait to convert Schema like things to DFSchema and DFSchemaRef with fewer keystrokes

Functions

aggregate_functional_dependencies

Calculates functional dependencies for aggregate output, when there is a GROUP BY expression.

field_not_found

Create a “field not found” DataFusion::SchemaError

get_required_group_by_exprs_indices

Returns indices for the minimal subset of GROUP BY expressions that are functionally equivalent to the original set of GROUP BY expressions.

get_target_functional_dependencies

Returns target indices, for the determinant keys that are inside group by expressions.

project_schema

Applies an optional projection to a SchemaRef, returning the projected schema

qualified_name

unqualified_field_not_found

Convenience wrapper over field_not_found for when there is no qualifier

Type Aliases

DFSchemaRef

A reference-counted reference to a DFSchema.

HashMap

HashSet

Result

Result type for operations that could result in an DataFusionError

SharedResult

Result type for operations that could result in an DataFusionError and needs to be shared (wrapped into Arc).