Crate datafusion_common

Source

Re-exports§

pub use error::field_not_found;
pub use error::unqualified_field_not_found;
pub use error::DataFusionError;
pub use error::Result;
pub use error::SchemaError;
pub use error::SharedResult;
pub use file_options::file_type::GetExt;
pub use file_options::file_type::DEFAULT_ARROW_EXTENSION;
pub use file_options::file_type::DEFAULT_AVRO_EXTENSION;
pub use file_options::file_type::DEFAULT_CSV_EXTENSION;
pub use file_options::file_type::DEFAULT_JSON_EXTENSION;
pub use file_options::file_type::DEFAULT_PARQUET_EXTENSION;
pub use scalar::ScalarType;
pub use scalar::ScalarValue;
pub use stats::ColumnStatistics;
pub use stats::Statistics;
pub use utils::project_schema;
pub use arrow;

Modules§

alias
cast
This module provides DataFusion specific casting functions that provide error handling. They are intended to “never fail” but provide an error message rather than a panic, as the corresponding kernels in arrow-rs such as as_boolean_array do.
config
Runtime configuration, via ConfigOptions
cse
Common Subexpression Elimination logic implemented in CSE can be controlled with a CSEController, that defines how to eliminate common subtrees from a particular TreeNode tree.
display
Types for plan display
error
DataFusion error types
file_options
Options related to how files should be written
format
hash_utils
Functionality used both on logical and physical plans
instant
WASM-compatible Instant wrapper.
parsers
Interval parsing logic
rounding
Floating point rounding mode utility library TODO: Remove this custom implementation and the “libc” dependency when floating-point rounding mode manipulation functions become available in Rust.
scalar
ScalarValue: stores single values
stats
This module provides data structures to represent statistics
test_util
Utility functions to make testing DataFusion based crates easier
tree_node
TreeNode for visiting and rewriting expression and plan trees
types
utils
This module provides the bisect function, which implements binary search.

Macros§

arrow_datafusion_err
arrow_err
assert_batches_eq
Compares formatted output of a record batch with an expected vector of strings, with the result of pretty formatting record batches. This is a macro so errors appear on the correct line
assert_batches_sorted_eq
Compares formatted output of a record batch with an expected vector of strings in a way that order does not matter. This is a macro so errors appear on the correct line
assert_contains
A macro to assert that one string is contained within another with a nice error message if they are not.
assert_not_contains
A macro to assert that one string is NOT contained within another with a nice error message if they are are.
config_datafusion_err
Macro wraps $ERR to add backtrace feature
config_err
Macro wraps Err($ERR) to add backtrace feature
config_field
context
create_array
downcast_value
Downcast an Arrow Array to a concrete type, return an DataFusionError::Internal if the cast is not possible. In normal usage of DataFusion the downcast should always succeed.
exec_datafusion_err
Macro wraps $ERR to add backtrace feature
exec_err
Macro wraps Err($ERR) to add backtrace feature
extensions_options
Convenience macro to create ExtensionsOptions.
internal_datafusion_err
Macro wraps $ERR to add backtrace feature
internal_err
Macro wraps Err($ERR) to add backtrace feature
not_impl_datafusion_err
Macro wraps $ERR to add backtrace feature
not_impl_err
Macro wraps Err($ERR) to add backtrace feature
plan_datafusion_err
Macro wraps $ERR to add backtrace feature
plan_err
Macro wraps Err($ERR) to add backtrace feature
record_batch
Creates a record batch from literal slice of values, suitable for rapid testing and development.
resources_datafusion_err
Macro wraps $ERR to add backtrace feature
resources_err
Macro wraps Err($ERR) to add backtrace feature
schema_datafusion_err
schema_err
sql_datafusion_err
sql_err
substrait_datafusion_err
Macro wraps $ERR to add backtrace feature
substrait_err
Macro wraps Err($ERR) to add backtrace feature
unwrap_or_internal_err
Unwrap an Option if possible. Otherwise return an DataFusionError::Internal. In normal usage of DataFusion the unwrap should always succeed.

Structs§

Column
A named reference to a qualified field in a schema.
Constraints
This object encapsulates a list of functional constraints:
DFSchema
DFSchema wraps an Arrow schema and adds relation names.
FunctionalDependence
This object defines a functional dependence in the schema. A functional dependence defines a relationship between determinant keys and dependent columns. A determinant key is a column, or a set of columns, whose value uniquely determines values of some other (dependent) columns. If two rows have the same determinant key, dependent columns in these rows are necessarily the same. If the determinant key is unique, the set of dependent columns is equal to the entire schema and the determinant key can serve as a primary key. Note that a primary key may “downgrade” into a determinant key due to an operation such as a join, and this object is used to track dependence relationships in such cases. For more information on functional dependencies, see: https://www.scaler.com/topics/dbms/functional-dependency-in-dbms/
FunctionalDependencies
This object encapsulates all functional dependencies in a given relation.
RecursionUnnestOption
Instruction on how to unnest a column (mostly with a list type) such as how to name the output, and how many level it should be unnested
ResolvedTableReference
A fully resolved path to a table of the form “catalog.schema.table”
UnnestOptions
Options for unnesting a column that contains a list type, replicating values in the other, non nested rows.

Enums§

Constraint
This object defines a constraint on a table.
Dependency
Describes functional dependency mode.
JoinConstraint
Join constraint
JoinSide
Join side. Stores the referred table side during calculations
JoinType
Join type
ParamValues
The parameter value corresponding to the placeholder
SchemaReference
TableReference
A multi part identifier (path) to a table that may require further resolution (e.g. foo.bar).

Traits§

ExprSchema
Provides schema information needed by certain methods of Expr (defined in the datafusion-common crate).
SchemaExt
DataFusion-specific extensions to Schema.
ToDFSchema
Convenience trait to convert Schema like things to DFSchema and DFSchemaRef with fewer keystrokes

Functions§

aggregate_functional_dependencies
Calculates functional dependencies for aggregate output, when there is a GROUP BY expression.
get_required_group_by_exprs_indices
Returns indices for the minimal subset of GROUP BY expressions that are functionally equivalent to the original set of GROUP BY expressions.
get_target_functional_dependencies
Returns target indices, for the determinant keys that are inside group by expressions.
qualified_name

Type Aliases§

DFSchemaRef
A reference-counted reference to a DFSchema.
HashMap
HashSet