Crate datafusion_functions

Function packages for DataFusion.

This crate contains a collection of various function packages for DataFusion, implemented using the extension API. Users may wish to control which functions are available to control the binary size of their application as well as use dialect specific implementations of functions (e.g. Spark vs Postgres)

Each package is implemented as a separate module, activated by a feature flag.

Available Packages

See the list of modules in this crate for available packages.

Using A Package

You can register all functions in all packages using the register_all function.

To access and use only the functions in a certain package, use the functions() method in each module.

// get the encoding functions
use datafusion_functions::encoding;
for udf in encoding::functions() {
  registry.register_udf(udf)?;
}

Each package also exports an expr_fn submodule to help create Exprs that invoke functions using a fluent style. For example:

// create an Expr that will invoke the encode function
use datafusion_expr::{col, lit};
use datafusion_functions::expr_fn;
// Equivalent to "encode(my_data, 'hex')" in SQL:
let expr = expr_fn::encode(col("my_data"), lit("hex"));

Implementing A New Package

To add a new package to this crate, you should follow the model of existing packages. The high level steps are:

1. Create a new module with the appropriate ScalarUDF implementations.

2. Use the macros in macros to create standard entry points.

3. Add a new feature to Cargo.toml, with any optional dependencies

4. Use the make_package! macro to expose the module when the feature is enabled.

Modules

core

Core datafusion expressions Enabled via feature flag core_expressions “core” DataFusion functions

crypto

Disabled. Enable via feature flag crypto_expressions

datetime

Date and time expressions. Contains functions such as to_timestamp Enabled via feature flag datetime_expressions date & time DataFusion functions

encoding

Encoding expressions. Contains Hex and binary encode and decode functions. Enabled via feature flag encoding_expressions

expr_fn

Fluent-style API for creating Exprs

macros

math

Mathematical functions. Enabled via feature flag math_expressions “math” DataFusion functions

planner

SQL planning extensions like UserDefinedFunctionPlanner

regex

Regular expression functions. Enabled via feature flag regex_expressions “regex” DataFusion functions

string

“string” DataFusion functions

strings

unicode

“unicode” DataFusion functions

Macros

downcast_arg

Downcast an argument to a specific array type, returning an internal error if the cast fails

downcast_named_arg

Downcast a named argument to a specific array type, returning an internal error if the cast fails

Functions

all_default_functions

Return all default functions

register_all

Registers all enabled packages with a FunctionRegistry