Crate datafusion_functions

Function packages for DataFusion.

This crate contains a collection of various function packages for DataFusion, implemented using the extension API. Users may wish to control which functions are available to control the binary size of their application as well as use dialect specific implementations of functions (e.g. Spark vs Postgres)

Each package is implemented as a separate module, activated by a feature flag.

Available Packages

See the list of modules in this crate for available packages.

Using A Package

You can register all functions in all packages using the register_all function.

To access and use only the functions in a certain package, use the functions() method in each module.

// get the encoding functions
use datafusion_functions::encoding;
for udf in encoding::functions() {
  registry.register_udf(udf)?;
}

Each package also exports an expr_fn submodule to help create Exprs that invoke functions using a fluent style. For example:

// create an Expr that will invoke the encode function
use datafusion_expr::{col, lit};
use datafusion_functions::expr_fn;
// Equivalent to "encode(my_data, 'hex')" in SQL:
let expr = expr_fn::encode(col("my_data"), lit("hex"));

Implementing A New Package

To add a new package to this crate, you should follow the model of existing packages. The high level steps are:

1. Create a new module with the appropriate ScalarUDF implementations.

2. Use the macros in macros to create standard entry points.

3. Add a new feature to Cargo.toml, with any optional dependencies

4. Use the make_package! macro to expose the module when the feature is enabled.

Modules

corecore_expressions

Core datafusion expressions Enabled via feature flag core_expressions “core” DataFusion functions

cryptocrypto_expressions

“crypto” DataFusion functions

datetimedatetime_expressions

Date and time expressions. Contains functions such as to_timestamp Enabled via feature flag datetime_expressions date & time DataFusion functions

encodingencoding_expressions

Encoding expressions. Contains Hex and binary encode and decode functions. Enabled via feature flag encoding_expressions

expr_fn

Fluent-style API for creating Exprs

macros

mathmath_expressions

Mathematical functions. Enabled via feature flag math_expressions “math” DataFusion functions

plannerdatetime_expressions or unicode_expressions

SQL planning extensions like UserDefinedFunctionPlanner

regexregex_expressions

Regular expression functions. Enabled via feature flag regex_expressions “regex” DataFusion functions

stringstring_expressions

“string” DataFusion functions

strings

unicodeunicode_expressions

“unicode” DataFusion functions

utils

Macros

downcast_arg

Downcast an argument to a specific array type, returning an internal error if the cast fails

downcast_named_arg

Downcast a named argument to a specific array type, returning an internal error if the cast fails

Functions

all_default_functions

Return all default functions

register_all

Registers all enabled packages with a FunctionRegistry