Enum NativeType

pub enum NativeType {
    Null,
    Boolean,
    Int8,
    Int16,
    Int32,
    Int64,
    UInt8,
    UInt16,
    UInt32,
    UInt64,
    Float16,
    Float32,
    Float64,
    Timestamp(TimeUnit, Option<Arc<str>>),
    Date,
    Time(TimeUnit),
    Duration(TimeUnit),
    Interval(IntervalUnit),
    Binary,
    FixedSizeBinary(i32),
    String,
    List(LogicalFieldRef),
    FixedSizeList(LogicalFieldRef, i32),
    Struct(LogicalFields),
    Union(LogicalUnionFields),
    Decimal(u8, i8),
    Map(LogicalFieldRef),
}

Representation of a type that DataFusion can handle natively. It is a subset of the physical variants in Arrow’s native DataType.

Variants

Null

Null type

Boolean

A boolean type representing the values true and false.

Int8

A signed 8-bit integer.

Int16

A signed 16-bit integer.

Int32

A signed 32-bit integer.

Int64

A signed 64-bit integer.

UInt8

An unsigned 8-bit integer.

UInt16

An unsigned 16-bit integer.

UInt32

An unsigned 32-bit integer.

UInt64

An unsigned 64-bit integer.

Float16

A 16-bit floating point number.

Float32

A 32-bit floating point number.

Float64

A 64-bit floating point number.

Timestamp(TimeUnit, Option<Arc<str>>)

A timestamp with an optional timezone.

Time is measured as a Unix epoch, counting the seconds from 00:00:00.000 on 1 January 1970, excluding leap seconds, as a signed 64-bit integer.

The time zone is a string indicating the name of a time zone, one of:

• As used in the Olson time zone database (the “tz database” or “tzdata”), such as “America/New_York”
• An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30

Timestamps with a non-empty timezone

If a Timestamp column has a non-empty timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in the UTC timezone (the Unix epoch), regardless of the Timestamp’s own timezone.

Therefore, timestamp values with a non-empty timezone correspond to physical points in time together with some additional information about how the data was obtained and/or how to display it (the timezone).

For example, the timestamp value 0 with the timezone string “Europe/Paris” corresponds to “January 1st 1970, 00h00” in the UTC timezone, but the application may prefer to display it as “January 1st 1970, 01h00” in the Europe/Paris timezone (which is the same physical point in time).

One consequence is that timestamp values with a non-empty timezone can be compared and ordered directly, since they all share the same well-known point of reference (the Unix epoch).

Timestamps with an unset / empty timezone

If a Timestamp column has no timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in an unknown timezone.

Therefore, timestamp values without a timezone cannot be meaningfully interpreted as physical points in time, but only as calendar / clock indications (“wall clock time”) in an unspecified timezone.

For example, the timestamp value 0 with an empty timezone string corresponds to “January 1st 1970, 00h00” in an unknown timezone: there is not enough information to interpret it as a well-defined physical point in time.

One consequence is that timestamp values without a timezone cannot be reliably compared or ordered, since they may have different points of reference. In particular, it is not possible to interpret an unset or empty timezone as the same as “UTC”.

Conversion between timezones

If a Timestamp column has a non-empty timezone, changing the timezone to a different non-empty value is a metadata-only operation: the timestamp values need not change as their point of reference remains the same (the Unix epoch).

However, if a Timestamp column has no timezone value, changing it to a non-empty value requires to think about the desired semantics. One possibility is to assume that the original timestamp values are relative to the epoch of the timezone being set; timestamp values should then adjusted to the Unix epoch (for example, changing the timezone from empty to “Europe/Paris” would require converting the timestamp values from “Europe/Paris” to “UTC”, which seems counter-intuitive but is nevertheless correct).

DataType::Timestamp(TimeUnit::Second, None);
DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));

Date

A signed date representing the elapsed time since UNIX epoch (1970-01-01) in days.

Time(TimeUnit)

A signed time representing the elapsed time since midnight in the unit of TimeUnit.

Duration(TimeUnit)

Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.

Interval(IntervalUnit)

A “calendar” interval which models types that don’t necessarily have a precise duration without the context of a base timestamp (e.g. days can differ in length during day light savings time transitions).

Binary

Opaque binary data of variable length.

FixedSizeBinary(i32)

Opaque binary data of fixed size. Enum parameter specifies the number of bytes per value.

String

A variable-length string in Unicode with UTF-8 encoding.

List(LogicalFieldRef)

A list of some logical data type with variable length.

FixedSizeList(LogicalFieldRef, i32)

A list of some logical data type with fixed length.

Struct(LogicalFields)

A nested type that contains a number of sub-fields.

Union(LogicalUnionFields)

A nested type that can represent slots of differing types.

Decimal(u8, i8)

Decimal value with precision and scale

• precision is the total number of digits
• scale is the number of digits past the decimal

For example the number 123.45 has precision 5 and scale 2.

In certain situations, scale could be negative number. For negative scale, it is the number of padding 0 to the right of the digits.

For example the number 12300 could be treated as a decimal has precision 3 and scale -2.

Map(LogicalFieldRef)

A Map is a type that an association between a key and a value.

The key and value types are not constrained, but keys should be hashable and unique.

In a field with Map type, key type and the second the value type. The names of the child fields may be respectively “entries”, “key”, and “value”, but this is not enforced.

Trait Implementations

impl Clone for NativeType

fn clone(&self) -> NativeType

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for NativeType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl From<&DataType> for NativeType

fn from(value: &DataType) -> Self

Converts to this type from the input type.

impl From<DataType> for NativeType

fn from(value: DataType) -> Self

Converts to this type from the input type.

impl Hash for NativeType

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl LogicalType for NativeType

fn native(&self) -> &NativeType

Get the native backing type of this logical type.

fn signature(&self) -> TypeSignature<'_>

Get the unique type signature for this logical type. Logical types with identical signatures are considered equal.

fn default_cast_for(&self, origin: &DataType) -> Result<DataType>

Get the default physical type to cast origin to in order to obtain a physical type that is logically compatible with this logical type.

impl Ord for NativeType

fn cmp(&self, other: &NativeType) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized,

Restrict a value to a certain interval.

impl PartialEq for NativeType

fn eq(&self, other: &NativeType) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for NativeType

fn partial_cmp(&self, other: &NativeType) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

Tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

Tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

Tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

Tests greater than or equal to (for self and other) and is used by the >= operator.

impl Eq for NativeType

impl StructuralPartialEq for NativeType