Module common

re-export of datafusion_common crate

Modules

• alias
• arrow
  A complete, safe, native Rust implementation of Apache Arrow, a cross-language development platform for in-memory data.
• cast
  This module provides DataFusion specific casting functions that provide error handling. They are intended to “never fail” but provide an error message rather than a panic, as the corresponding kernels in arrow-rs such as as_boolean_array do.
• config
  Runtime configuration, via ConfigOptions
• cse
  Common Subexpression Elimination logic implemented in CSE can be controlled with a CSEController, that defines how to eliminate common subtrees from a particular TreeNode tree.
• display
  Types for plan display
• error
  DataFusion error types
• file_options
  Options related to how files should be written
• format
• hash_utils
  Functionality used both on logical and physical plans
• instant
  WASM-compatible Instant wrapper.
• parsers
  Interval parsing logic
• rounding
  Floating point rounding mode utility library TODO: Remove this custom implementation and the “libc” dependency when floating-point rounding mode manipulation functions become available in Rust.
• runtime
  re-export of datafusion_common_runtime crate
• scalar
  ScalarValue: stores single values
• stats
  This module provides data structures to represent statistics
• test_util
  Utility functions to make testing DataFusion based crates easier
• tree_node
  TreeNode for visiting and rewriting expression and plan trees
• types
• utils
  This module provides the bisect function, which implements binary search.

Macros

• _config_datafusion_err
  Macro wraps $ERR to add backtrace feature
• _exec_datafusion_err
  Macro wraps $ERR to add backtrace feature
• _internal_datafusion_err
  Macro wraps $ERR to add backtrace feature
• _not_impl_datafusion_err
  Macro wraps $ERR to add backtrace feature
• _plan_datafusion_err
  Macro wraps $ERR to add backtrace feature
• _resources_datafusion_err
  Macro wraps $ERR to add backtrace feature
• _substrait_datafusion_err
  Macro wraps $ERR to add backtrace feature
• arrow_datafusion_err
• arrow_err
• assert_batches_eq
  Compares formatted output of a record batch with an expected vector of strings, with the result of pretty formatting record batches. This is a macro so errors appear on the correct line
• assert_batches_sorted_eq
  Compares formatted output of a record batch with an expected vector of strings in a way that order does not matter. This is a macro so errors appear on the correct line
• assert_contains
  A macro to assert that one string is contained within another with a nice error message if they are not.
• assert_not_contains
  A macro to assert that one string is NOT contained within another with a nice error message if they are are.
• config_datafusion_err
  Macro wraps $ERR to add backtrace feature
• config_err
  Macro wraps Err($ERR) to add backtrace feature
• config_field
• config_namespace
  A macro that wraps a configuration struct and automatically derives Default and ConfigField for it, allowing it to be used in the ConfigOptions configuration tree
• context
• create_array
• downcast_value
  Downcast an Arrow Array to a concrete type, return an DataFusionError::Internal if the cast is not possible. In normal usage of DataFusion the downcast should always succeed.
• exec_datafusion_err
  Macro wraps $ERR to add backtrace feature
• exec_err
  Macro wraps Err($ERR) to add backtrace feature
• extensions_options
  Convenience macro to create ExtensionsOptions.
• internal_datafusion_err
  Macro wraps $ERR to add backtrace feature
• internal_err
  Macro wraps Err($ERR) to add backtrace feature
• map_until_stop_and_collect
  Transformation helper to process a heterogeneous sequence of tree node containing expressions.
• not_impl_datafusion_err
  Macro wraps $ERR to add backtrace feature
• not_impl_err
  Macro wraps Err($ERR) to add backtrace feature
• plan_datafusion_err
  Macro wraps $ERR to add backtrace feature
• plan_err
  Macro wraps Err($ERR) to add backtrace feature
• record_batch
  Creates a record batch from literal slice of values, suitable for rapid testing and development.
• resources_datafusion_err
  Macro wraps $ERR to add backtrace feature
• resources_err
  Macro wraps Err($ERR) to add backtrace feature
• schema_datafusion_err
• schema_err
• sql_datafusion_err
• sql_err
• substrait_datafusion_err
  Macro wraps $ERR to add backtrace feature
• substrait_err
  Macro wraps Err($ERR) to add backtrace feature
• unwrap_or_internal_err
  Unwrap an Option if possible. Otherwise return an DataFusionError::Internal. In normal usage of DataFusion the unwrap should always succeed.

Structs

• Column
  A named reference to a qualified field in a schema.
• ColumnStatistics
  Statistics for a column within a relation
• Constraints
  This object encapsulates a list of functional constraints:
• DFSchema
  DFSchema wraps an Arrow schema and adds relation names.
• FunctionalDependence
  This object defines a functional dependence in the schema. A functional dependence defines a relationship between determinant keys and dependent columns. A determinant key is a column, or a set of columns, whose value uniquely determines values of some other (dependent) columns. If two rows have the same determinant key, dependent columns in these rows are necessarily the same. If the determinant key is unique, the set of dependent columns is equal to the entire schema and the determinant key can serve as a primary key. Note that a primary key may “downgrade” into a determinant key due to an operation such as a join, and this object is used to track dependence relationships in such cases. For more information on functional dependencies, see: https://www.scaler.com/topics/dbms/functional-dependency-in-dbms/
• FunctionalDependencies
  This object encapsulates all functional dependencies in a given relation.
• RecursionUnnestOption
  Instruction on how to unnest a column (mostly with a list type) such as how to name the output, and how many level it should be unnested
• ResolvedTableReference
  A fully resolved path to a table of the form “catalog.schema.table”
• Statistics
  Statistics for a relation Fields are optional and can be inexact because the sources sometimes provide approximate estimates for performance reasons and the transformations output are not always predictable.
• UnnestOptions
  Options for unnesting a column that contains a list type, replicating values in the other, non nested rows.

Enums

• Constraint
  This object defines a constraint on a table.
• DataFusionError
  DataFusion error
• Dependency
  Describes functional dependency mode.
• JoinConstraint
  Join constraint
• JoinSide
  Join side. Stores the referred table side during calculations
• JoinType
  Join type
• ParamValues
  The parameter value corresponding to the placeholder
• ScalarValue
  A dynamically typed, nullable single value.
• SchemaError
  Schema-related errors
• SchemaReference
• TableReference
  A multi part identifier (path) to a table that may require further resolution (e.g. foo.bar).

Constants

• DEFAULT_ARROW_EXTENSION
  The default file extension of arrow files
• DEFAULT_AVRO_EXTENSION
  The default file extension of avro files
• DEFAULT_CSV_EXTENSION
  The default file extension of csv files
• DEFAULT_JSON_EXTENSION
  The default file extension of json files
• DEFAULT_PARQUET_EXTENSION
  The default file extension of parquet files

Traits

• ExprSchema
  Provides schema information needed by certain methods of Expr (defined in the datafusion-common crate).
• GetExt
  Define each FileType/FileCompressionType’s extension
• ScalarType
  Trait used to map a NativeType to a ScalarValue
• SchemaExt
  DataFusion-specific extensions to Schema.
• ToDFSchema
  Convenience trait to convert Schema like things to DFSchema and DFSchemaRef with fewer keystrokes

Functions

• aggregate_functional_dependencies
  Calculates functional dependencies for aggregate output, when there is a GROUP BY expression.
• field_not_found
  Create a “field not found” DataFusion::SchemaError
• get_required_group_by_exprs_indices
  Returns indices for the minimal subset of GROUP BY expressions that are functionally equivalent to the original set of GROUP BY expressions.
• get_target_functional_dependencies
  Returns target indices, for the determinant keys that are inside group by expressions.
• project_schema
  Applies an optional projection to a SchemaRef, returning the projected schema
• qualified_name
• unqualified_field_not_found
  Convenience wrapper over field_not_found for when there is no qualifier

Type Aliases

• DFSchemaRef
  A reference-counted reference to a DFSchema.
• Result
  Result type for operations that could result in an DataFusionError
• SharedResult
  Result type for operations that could result in an DataFusionError and needs to be shared (wrapped into Arc).