Enum arrow_schema::DataType

pub enum DataType {
    Null,
    Boolean,
    Int8,
    Int16,
    Int32,
    Int64,
    UInt8,
    UInt16,
    UInt32,
    UInt64,
    Float16,
    Float32,
    Float64,
    Timestamp(TimeUnit, Option<Arc<str>>),
    Date32,
    Date64,
    Time32(TimeUnit),
    Time64(TimeUnit),
    Duration(TimeUnit),
    Interval(IntervalUnit),
    Binary,
    FixedSizeBinary(i32),
    LargeBinary,
    BinaryView,
    Utf8,
    LargeUtf8,
    Utf8View,
    List(FieldRef),
    ListView(FieldRef),
    FixedSizeList(FieldRef, i32),
    LargeList(FieldRef),
    LargeListView(FieldRef),
    Struct(Fields),
    Union(UnionFields, UnionMode),
    Dictionary(Box<DataType>, Box<DataType>),
    Decimal128(u8, i8),
    Decimal256(u8, i8),
    Map(FieldRef, bool),
    RunEndEncoded(FieldRef, FieldRef),
}

Datatypes supported by this implementation of Apache Arrow.

The variants of this enum include primitive fixed size types as well as parametric or nested types. See Schema.fbs for Arrow’s specification.

Examples

Primitive types

// create a new 32-bit signed integer
let data_type = DataType::Int32;

Nested Types

// create a new list of 32-bit signed integers directly
let list_data_type = DataType::List(Arc::new(Field::new("item", DataType::Int32, true)));
// Create the same list type with constructor
let list_data_type2 = DataType::new_list(DataType::Int32, true);
assert_eq!(list_data_type, list_data_type2);

Dictionary Types

// String Dictionary (key type Int32 and value type Utf8)
let data_type = DataType::Dictionary(Box::new(DataType::Int32), Box::new(DataType::Utf8));

Timestamp Types

// timestamp with millisecond precision without timezone specified
let data_type = DataType::Timestamp(TimeUnit::Millisecond, None);
// timestamp with nanosecond precision in UTC timezone
let data_type = DataType::Timestamp(TimeUnit::Nanosecond, Some("UTC".into()));

Display and FromStr

The Display and FromStr implementations for DataType are human-readable, parseable, and reversible.

let data_type = DataType::Dictionary(Box::new(DataType::Int32), Box::new(DataType::Utf8));
let data_type_string = data_type.to_string();
assert_eq!(data_type_string, "Dictionary(Int32, Utf8)");
// display can be parsed back into the original type
let parsed_data_type: DataType = data_type.to_string().parse().unwrap();
assert_eq!(data_type, parsed_data_type);

Nested Support

Currently, the Rust implementation supports the following nested types:

• List<T>
• LargeList<T>
• FixedSizeList<T>
• Struct<T, U, V, ...>
• Union<T, U, V, ...>
• Map<K, V>

Nested types can themselves be nested within other arrays. For more information on these types please see the physical memory layout of Apache Arrow

Variants

Null

Null type

Boolean

A boolean datatype representing the values true and false.

Int8

A signed 8-bit integer.

Int16

A signed 16-bit integer.

Int32

A signed 32-bit integer.

Int64

A signed 64-bit integer.

UInt8

An unsigned 8-bit integer.

UInt16

An unsigned 16-bit integer.

UInt32

An unsigned 32-bit integer.

UInt64

An unsigned 64-bit integer.

Float16

A 16-bit floating point number.

Float32

A 32-bit floating point number.

Float64

A 64-bit floating point number.

Timestamp(TimeUnit, Option<Arc<str>>)

A timestamp with an optional timezone.

Time is measured as a Unix epoch, counting the seconds from 00:00:00.000 on 1 January 1970, excluding leap seconds, as a signed 64-bit integer.

The time zone is a string indicating the name of a time zone, one of:

• As used in the Olson time zone database (the “tz database” or “tzdata”), such as “America/New_York”
• An absolute time zone offset of the form +XX:XX or -XX:XX, such as +07:30

Timestamps with a non-empty timezone

If a Timestamp column has a non-empty timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in the UTC timezone (the Unix epoch), regardless of the Timestamp’s own timezone.

Therefore, timestamp values with a non-empty timezone correspond to physical points in time together with some additional information about how the data was obtained and/or how to display it (the timezone).

For example, the timestamp value 0 with the timezone string “Europe/Paris” corresponds to “January 1st 1970, 00h00” in the UTC timezone, but the application may prefer to display it as “January 1st 1970, 01h00” in the Europe/Paris timezone (which is the same physical point in time).

One consequence is that timestamp values with a non-empty timezone can be compared and ordered directly, since they all share the same well-known point of reference (the Unix epoch).

Timestamps with an unset / empty timezone

If a Timestamp column has no timezone value, its epoch is 1970-01-01 00:00:00 (January 1st 1970, midnight) in an unknown timezone.

Therefore, timestamp values without a timezone cannot be meaningfully interpreted as physical points in time, but only as calendar / clock indications (“wall clock time”) in an unspecified timezone.

For example, the timestamp value 0 with an empty timezone string corresponds to “January 1st 1970, 00h00” in an unknown timezone: there is not enough information to interpret it as a well-defined physical point in time.

One consequence is that timestamp values without a timezone cannot be reliably compared or ordered, since they may have different points of reference. In particular, it is not possible to interpret an unset or empty timezone as the same as “UTC”.

Conversion between timezones

If a Timestamp column has a non-empty timezone, changing the timezone to a different non-empty value is a metadata-only operation: the timestamp values need not change as their point of reference remains the same (the Unix epoch).

However, if a Timestamp column has no timezone value, changing it to a non-empty value requires to think about the desired semantics. One possibility is to assume that the original timestamp values are relative to the epoch of the timezone being set; timestamp values should then adjusted to the Unix epoch (for example, changing the timezone from empty to “Europe/Paris” would require converting the timestamp values from “Europe/Paris” to “UTC”, which seems counter-intuitive but is nevertheless correct).

DataType::Timestamp(TimeUnit::Second, None);
DataType::Timestamp(TimeUnit::Second, Some("literal".into()));
DataType::Timestamp(TimeUnit::Second, Some("string".to_string().into()));

Date32

A signed 32-bit date representing the elapsed time since UNIX epoch (1970-01-01) in days.

Date64

A signed 64-bit date representing the elapsed time since UNIX epoch (1970-01-01) in milliseconds.

According to the specification (see Schema.fbs), this should be treated as the number of days, in milliseconds, since the UNIX epoch. Therefore, values must be evenly divisible by 86_400_000 (the number of milliseconds in a standard day).

The reason for this is for compatibility with other language’s native libraries, such as Java, which historically lacked a dedicated date type and only supported timestamps.

Practically, validation that values of this type are evenly divisible by 86_400_000 is not enforced by this library for performance and usability reasons. Date64 values will be treated similarly to the Timestamp(TimeUnit::Millisecond, None) type, in that its values will be printed showing the time of day if the value does not represent an exact day, and arithmetic can be done at the millisecond granularity to change the time represented.

Users should prefer using Date32 to cleanly represent the number of days, or one of the Timestamp variants to include time as part of the representation, depending on their use case.

For more details, see #5288.

Time32(TimeUnit)

A signed 32-bit time representing the elapsed time since midnight in the unit of TimeUnit. Must be either seconds or milliseconds.

Time64(TimeUnit)

A signed 64-bit time representing the elapsed time since midnight in the unit of TimeUnit. Must be either microseconds or nanoseconds.

Duration(TimeUnit)

Measure of elapsed time in either seconds, milliseconds, microseconds or nanoseconds.

Interval(IntervalUnit)

A “calendar” interval which models types that don’t necessarily have a precise duration without the context of a base timestamp (e.g. days can differ in length during day light savings time transitions).

Binary

Opaque binary data of variable length.

A single Binary array can store up to i32::MAX bytes of binary data in total.

FixedSizeBinary(i32)

Opaque binary data of fixed size. Enum parameter specifies the number of bytes per value.

LargeBinary

Opaque binary data of variable length and 64-bit offsets.

A single LargeBinary array can store up to i64::MAX bytes of binary data in total.

BinaryView

(NOT YET FULLY SUPPORTED) Opaque binary data of variable length.

Note this data type is not yet fully supported. Using it with arrow APIs may result in panics.

Logically the same as Self::Binary, but the internal representation uses a view struct that contains the string length and either the string’s entire data inline (for small strings) or an inlined prefix, an index of another buffer, and an offset pointing to a slice in that buffer (for non-small strings).

Utf8

A variable-length string in Unicode with UTF-8 encoding.

A single Utf8 array can store up to i32::MAX bytes of string data in total.

LargeUtf8

A variable-length string in Unicode with UFT-8 encoding and 64-bit offsets.

A single LargeUtf8 array can store up to i64::MAX bytes of string data in total.

Utf8View

(NOT YET FULLY SUPPORTED) A variable-length string in Unicode with UTF-8 encoding

Note this data type is not yet fully supported. Using it with arrow APIs may result in panics.

Logically the same as Self::Utf8, but the internal representation uses a view struct that contains the string length and either the string’s entire data inline (for small strings) or an inlined prefix, an index of another buffer, and an offset pointing to a slice in that buffer (for non-small strings).

List(FieldRef)

A list of some logical data type with variable length.

A single List array can store up to i32::MAX elements in total.

ListView(FieldRef)

(NOT YET FULLY SUPPORTED) A list of some logical data type with variable length.

Note this data type is not yet fully supported. Using it with arrow APIs may result in panics.

The ListView layout is defined by three buffers: a validity bitmap, an offsets buffer, and an additional sizes buffer. Sizes and offsets are both 32 bits for this type

FixedSizeList(FieldRef, i32)

A list of some logical data type with fixed length.

LargeList(FieldRef)

A list of some logical data type with variable length and 64-bit offsets.

A single LargeList array can store up to i64::MAX elements in total.

LargeListView(FieldRef)

(NOT YET FULLY SUPPORTED) A list of some logical data type with variable length and 64-bit offsets.

Note this data type is not yet fully supported. Using it with arrow APIs may result in panics.

The LargeListView layout is defined by three buffers: a validity bitmap, an offsets buffer, and an additional sizes buffer. Sizes and offsets are both 64 bits for this type

Struct(Fields)

A nested datatype that contains a number of sub-fields.

Union(UnionFields, UnionMode)

A nested datatype that can represent slots of differing types. Components:

1. UnionFields
2. The type of union (Sparse or Dense)

Dictionary(Box<DataType>, Box<DataType>)

A dictionary encoded array (key_type, value_type), where each array element is an index of key_type into an associated dictionary of value_type.

Dictionary arrays are used to store columns of value_type that contain many repeated values using less memory, but with a higher CPU overhead for some operations.

This type mostly used to represent low cardinality string arrays or a limited set of primitive types as integers.

Decimal128(u8, i8)

Exact 128-bit width decimal value with precision and scale

• precision is the total number of digits
• scale is the number of digits past the decimal

For example the number 123.45 has precision 5 and scale 2.

In certain situations, scale could be negative number. For negative scale, it is the number of padding 0 to the right of the digits.

For example the number 12300 could be treated as a decimal has precision 3 and scale -2.

Decimal256(u8, i8)

Exact 256-bit width decimal value with precision and scale

• precision is the total number of digits
• scale is the number of digits past the decimal

For example the number 123.45 has precision 5 and scale 2.

In certain situations, scale could be negative number. For negative scale, it is the number of padding 0 to the right of the digits.

For example the number 12300 could be treated as a decimal has precision 3 and scale -2.

Map(FieldRef, bool)

A Map is a logical nested type that is represented as

List<entries: Struct<key: K, value: V>>

The keys and values are each respectively contiguous. The key and value types are not constrained, but keys should be hashable and unique. Whether the keys are sorted can be set in the bool after the Field.

In a field with Map type, the field has a child Struct field, which then has two children: key type and the second the value type. The names of the child fields may be respectively “entries”, “key”, and “value”, but this is not enforced.

RunEndEncoded(FieldRef, FieldRef)

A run-end encoding (REE) is a variation of run-length encoding (RLE). These encodings are well-suited for representing data containing sequences of the same value, called runs. Each run is represented as a value and an integer giving the index in the array where the run ends.

A run-end encoded array has no buffers by itself, but has two child arrays. The first child array, called the run ends array, holds either 16, 32, or 64-bit signed integers. The actual values of each run are held in the second child array.

These child arrays are prescribed the standard names of “run_ends” and “values” respectively.

Implementations

impl DataType

pub fn is_primitive(&self) -> bool

Returns true if the type is primitive: (numeric, temporal).

pub fn is_numeric(&self) -> bool

Returns true if this type is numeric: (UInt*, Int*, Float*, Decimal*).

pub fn is_temporal(&self) -> bool

Returns true if this type is temporal: (Date*, Time*, Duration, or Interval).

pub fn is_floating(&self) -> bool

Returns true if this type is floating: (Float*).

pub fn is_integer(&self) -> bool

Returns true if this type is integer: (Int*, UInt*).

pub fn is_signed_integer(&self) -> bool

Returns true if this type is signed integer: (Int*).

pub fn is_unsigned_integer(&self) -> bool

Returns true if this type is unsigned integer: (UInt*).

pub fn is_dictionary_key_type(&self) -> bool

Returns true if this type is valid as a dictionary key

pub fn is_run_ends_type(&self) -> bool

Returns true if this type is valid for run-ends array in RunArray

pub fn is_nested(&self) -> bool

Returns true if this type is nested (List, FixedSizeList, LargeList, Struct, Union, or Map), or a dictionary of a nested type

pub fn is_null(&self) -> bool

Returns true if this type is DataType::Null.

pub fn equals_datatype(&self, other: &DataType) -> bool

Compares the datatype with another, ignoring nested field names and metadata.

pub fn primitive_width(&self) -> Option<usize>

Returns the bit width of this type if it is a primitive type

Returns None if not a primitive type

pub fn size(&self) -> usize

Return size of this instance in bytes.

Includes the size of Self.

pub fn contains(&self, other: &DataType) -> bool

Check to see if self is a superset of other

If DataType is a nested type, then it will check to see if the nested type is a superset of the other nested type else it will check to see if the DataType is equal to the other DataType

pub fn new_list(data_type: DataType, nullable: bool) -> Self

Create a DataType::List with elements of the specified type and nullability, and conventionally named inner Field ("item").

To specify field level metadata, construct the inner Field directly via Field::new or Field::new_list_field.

pub fn new_large_list(data_type: DataType, nullable: bool) -> Self

Create a DataType::LargeList with elements of the specified type and nullability, and conventionally named inner Field ("item").

To specify field level metadata, construct the inner Field directly via Field::new or Field::new_list_field.

pub fn new_fixed_size_list( data_type: DataType, size: i32, nullable: bool, ) -> Self

Create a DataType::FixedSizeList with elements of the specified type, size and nullability, and conventionally named inner Field ("item").

To specify field level metadata, construct the inner Field directly via Field::new or Field::new_list_field.

Trait Implementations

impl Clone for DataType

fn clone(&self) -> DataType

Returns a copy of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for DataType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for DataType

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl FromStr for DataType

Parses str into a DataType.

This is the reverse of DataType’s Display impl, and maintains the invariant that DataType::try_from(&data_type.to_string()).unwrap() == data_type

Example

use arrow_schema::DataType;

let data_type: DataType = "Int32".parse().unwrap();
assert_eq!(data_type, DataType::Int32);

type Err = ArrowError

The associated error which can be returned from parsing.

fn from_str(s: &str) -> Result<Self, Self::Err>

Parses a string s to return a value of this type.

impl Hash for DataType

fn hash<__H: Hasher>(&self, state: &mut __H)

Feeds this value into the given Hasher.

fn hash_slice<H>(data: &[Self], state: &mut H)

where H: Hasher, Self: Sized,

Feeds a slice of this type into the given Hasher.

impl Ord for DataType

fn cmp(&self, other: &DataType) -> Ordering

This method returns an Ordering between self and other.

fn max(self, other: Self) -> Self

where Self: Sized,

Compares and returns the maximum of two values.

fn min(self, other: Self) -> Self

where Self: Sized,

Compares and returns the minimum of two values.

fn clamp(self, min: Self, max: Self) -> Self

where Self: Sized + PartialOrd,

Restrict a value to a certain interval.

impl PartialEq for DataType

fn eq(&self, other: &DataType) -> bool

This method tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

This method tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl PartialOrd for DataType

fn partial_cmp(&self, other: &DataType) -> Option<Ordering>

This method returns an ordering between self and other values if one exists.

fn lt(&self, other: &Rhs) -> bool

This method tests less than (for self and other) and is used by the < operator.

fn le(&self, other: &Rhs) -> bool

This method tests less than or equal to (for self and other) and is used by the <= operator.

fn gt(&self, other: &Rhs) -> bool

This method tests greater than (for self and other) and is used by the > operator.

fn ge(&self, other: &Rhs) -> bool

This method tests greater than or equal to (for self and other) and is used by the >= operator.

impl TryFrom<&DataType> for FFI_ArrowSchema

fn try_from(dtype: &DataType) -> Result<Self, ArrowError>

See CDataInterface docs

type Error = ArrowError

The type returned in the event of a conversion error.

impl TryFrom<&FFI_ArrowSchema> for DataType

fn try_from(c_schema: &FFI_ArrowSchema) -> Result<Self, ArrowError>

See CDataInterface docs

type Error = ArrowError

The type returned in the event of a conversion error.

impl TryFrom<&str> for DataType

type Error = ArrowError

The type returned in the event of a conversion error.

fn try_from(value: &str) -> Result<Self, Self::Error>

Performs the conversion.

impl TryFrom<DataType> for FFI_ArrowSchema

type Error = ArrowError

The type returned in the event of a conversion error.

fn try_from(dtype: DataType) -> Result<Self, ArrowError>

Performs the conversion.

impl Eq for DataType

impl StructuralPartialEq for DataType