Struct ChunkedArray

Source

pub struct ChunkedArray<T: PolarsDataType> { /* private fields */ }

Expand description

§ChunkedArray

Every Series contains a ChunkedArray<T>. Unlike Series, ChunkedArrays are typed. This allows us to apply closures to the data and collect the results to a ChunkedArray of the same type T. Below we use an apply to use the cosine function to the values of a ChunkedArray.

fn apply_cosine_and_cast(ca: &Float32Chunked) -> Float32Chunked {
    ca.apply_values(|v| v.cos())
}

§Conversion between Series and ChunkedArrays

Conversion from a Series to a ChunkedArray is effortless.

fn to_chunked_array(series: &Series) -> PolarsResult<&Int32Chunked>{
    series.i32()
}

fn to_series(ca: Int32Chunked) -> Series {
    ca.into_series()
}

§Iterators

ChunkedArrays fully support Rust native Iterator and DoubleEndedIterator traits, thereby giving access to all the excellent methods available for Iterators.


fn iter_forward(ca: &Float32Chunked) {
    ca.iter()
        .for_each(|opt_v| println!("{:?}", opt_v))
}

fn iter_backward(ca: &Float32Chunked) {
    ca.iter()
        .rev()
        .for_each(|opt_v| println!("{:?}", opt_v))
}

§Memory layout

ChunkedArrays use Apache Arrow as backend for the memory layout. Arrows memory is immutable which makes it possible to make multiple zero copy (sub)-views from a single array.

To be able to append data, Polars uses chunks to append new memory locations, hence the ChunkedArray<T> data structure. Appends are cheap, because it will not lead to a full reallocation of the whole array (as could be the case with a Rust Vec).

However, multiple chunks in a ChunkedArray will slow down many operations that need random access because we have an extra indirection and indexes need to be mapped to the proper chunk. Arithmetic may also be slowed down by this. When multiplying two ChunkedArrays with different chunk sizes they cannot utilize SIMD for instance.

If you want to have predictable performance (no unexpected re-allocation of memory), it is advised to call the ChunkedArray::rechunk after multiple append operations.

See also ChunkedArray::extend for appends within a chunk.

§Invariants

A ChunkedArray should always have at least a single ArrayRef.
The PolarsDataType T should always map to the correct ArrowDataType in the ArrayRef chunks.
Nested datatypes such as List and Array store the physical types instead of the logical type given by the datatype.

Struct ChunkedArrayCopy item path

§ChunkedArray

§Conversion between Series and ChunkedArrays

§Iterators

§Memory layout

§Invariants

Implementations§

impl ChunkedArray<BooleanType>

pub fn sum(&self) -> Option<IdxSize>

pub fn min(&self) -> Option<bool>

pub fn max(&self) -> Option<bool>

pub fn mean(&self) -> Option<f64>

impl ChunkedArray<BinaryType>

pub fn max_binary(&self) -> Option<&[u8]>

pub fn min_binary(&self) -> Option<&[u8]>

impl<T> ChunkedArray<T>where T: PolarsDataType<IsNested = FalseT, IsObject = FalseT>, for<'a> T::Physical<'a>: TotalOrd,

pub fn append(&mut self, other: &Self) -> PolarsResult<()>

impl<T> ChunkedArray<T>where T: PolarsDataType,

pub fn apply_nonnull_values_generic<'a, U, K, F>( &'a self, dtype: DataType, op: F, ) -> ChunkedArray<U>where U: PolarsDataType, F: FnMut(T::Physical<'a>) -> K, U::Array: ArrayFromIterDtype<K> + ArrayFromIterDtype<Option<K>>,

pub fn try_apply_nonnull_values_generic<'a, U, K, F, E>( &'a self, op: F, ) -> Result<ChunkedArray<U>, E>where U: PolarsDataType, F: FnMut(T::Physical<'a>) -> Result<K, E>, U::Array: ArrayFromIter<K> + ArrayFromIter<Option<K>>,

pub fn apply_into_string_amortized<'a, F>(&'a self, f: F) -> StringChunkedwhere F: FnMut(T::Physical<'a>, &mut String),

pub fn try_apply_into_string_amortized<'a, F, E>( &'a self, f: F, ) -> Result<StringChunked, E>where F: FnMut(T::Physical<'a>, &mut String) -> Result<(), E>,

impl<T: PolarsNumericType> ChunkedArray<T>

pub fn cast_and_apply_in_place<F, S>(&self, f: F) -> ChunkedArray<S>where F: Fn(S::Native) -> S::Native + Copy, S: PolarsNumericType,

pub fn apply_in_place<F>(self, f: F) -> Selfwhere F: Fn(T::Native) -> T::Native + Copy,

impl<T: PolarsNumericType> ChunkedArray<T>

pub fn apply_mut<F>(&mut self, f: F)where F: Fn(T::Native) -> T::Native + Copy,

impl ChunkedArray<StringType>

pub fn apply_mut<'a, F>(&'a self, f: F) -> Selfwhere F: FnMut(&'a str) -> &'a str,

impl ChunkedArray<BinaryType>

pub fn apply_mut<'a, F>(&'a self, f: F) -> Selfwhere F: FnMut(&'a [u8]) -> &'a [u8],

impl ChunkedArray<StringType>

pub unsafe fn apply_views<F: FnMut(View, &str) -> View + Copy>( &self, update_view: F, ) -> Self

§Safety

impl ChunkedArray<Float32Type>

pub fn apply_as_ints<F>(&self, f: F) -> Serieswhere F: Fn(&Series) -> Series,

impl ChunkedArray<Float64Type>

pub fn apply_as_ints<F>(&self, f: F) -> Serieswhere F: Fn(&Series) -> Series,

impl ChunkedArray<BooleanType>

pub fn num_trues(&self) -> usize

pub fn num_falses(&self) -> usize

impl<T: PolarsDataType> ChunkedArray<T>

pub fn len(&self) -> usize

pub fn null_count(&self) -> usize

pub unsafe fn set_null_count(&mut self, null_count: IdxSize)

§Safety

pub fn is_empty(&self) -> bool

pub fn rechunk(&self) -> Self

pub fn rechunk_validity(&self) -> Option<Bitmap>

pub fn split_at(&self, offset: i64) -> (Self, Self)

pub fn slice(&self, offset: i64, length: usize) -> Self

pub fn limit(&self, num_elements: usize) -> Selfwhere Self: Sized,

pub fn head(&self, length: Option<usize>) -> Selfwhere Self: Sized,

pub fn tail(&self, length: Option<usize>) -> Selfwhere Self: Sized,

pub fn prune_empty_chunks(&mut self)

impl ChunkedArray<StringType>

pub fn to_decimal(&self, infer_length: usize) -> PolarsResult<Series>

impl<T> ChunkedArray<T>where T: PolarsNumericType,

pub fn extend(&mut self, other: &Self) -> PolarsResult<()>

impl<T> ChunkedArray<T>where T: PolarsDataType,

pub fn for_each<'a, F>(&'a self, op: F)where F: FnMut(Option<T::Physical<'a>>),

impl ChunkedArray<ListType>

pub fn full_null_with_dtype( name: PlSmallStr, length: usize, inner_dtype: &DataType, ) -> ListChunked

impl ChunkedArray<UInt32Type>

pub fn with_nullable_idx<T, F: FnOnce(&IdxCa) -> T>( idx: &[NullableIdxSize], f: F, ) -> T

impl<T: PolarsDataType> ChunkedArray<T>

pub fn is_null(&self) -> BooleanChunked

pub fn is_not_null(&self) -> BooleanChunked

impl<T> ChunkedArray<T>where ChunkedArray<T>: IntoSeries, T: PolarsFloatType, T::Native: Float + IsFloat + SubAssign + Pow<T::Native, Output = T::Native>,

pub fn rolling_map_float<F>( &self, window_size: usize, f: F, ) -> PolarsResult<Self>where F: FnMut(&mut ChunkedArray<T>) -> Option<T::Native>,

impl ChunkedArray<BinaryType>

pub unsafe fn to_string_unchecked(&self) -> StringChunked

§Safety

impl ChunkedArray<StringType>

pub fn as_binary(&self) -> BinaryChunked

impl ChunkedArray<BooleanType>

pub fn any(&self) -> bool

pub fn all(&self) -> bool

pub fn any_kleene(&self) -> Option<bool>

pub fn all_kleene(&self) -> Option<bool>

Struct ChunkedArray

impl<T> ChunkedArray<T>
where T: PolarsDataType<IsNested = FalseT, IsObject = FalseT>, for<'a> T::Physical<'a>: TotalOrd,

impl<T> ChunkedArray<T>
where T: PolarsDataType,

pub fn apply_nonnull_values_generic<'a, U, K, F>( &'a self, dtype: DataType, op: F, ) -> ChunkedArray<U>
where U: PolarsDataType, F: FnMut(T::Physical<'a>) -> K, U::Array: ArrayFromIterDtype<K> + ArrayFromIterDtype<Option<K>>,

pub fn try_apply_nonnull_values_generic<'a, U, K, F, E>( &'a self, op: F, ) -> Result<ChunkedArray<U>, E>
where U: PolarsDataType, F: FnMut(T::Physical<'a>) -> Result<K, E>, U::Array: ArrayFromIter<K> + ArrayFromIter<Option<K>>,

pub fn apply_into_string_amortized<'a, F>(&'a self, f: F) -> StringChunked
where F: FnMut(T::Physical<'a>, &mut String),

pub fn try_apply_into_string_amortized<'a, F, E>( &'a self, f: F, ) -> Result<StringChunked, E>
where F: FnMut(T::Physical<'a>, &mut String) -> Result<(), E>,

pub fn cast_and_apply_in_place<F, S>(&self, f: F) -> ChunkedArray<S>
where F: Fn(S::Native) -> S::Native + Copy, S: PolarsNumericType,

pub fn apply_in_place<F>(self, f: F) -> Self
where F: Fn(T::Native) -> T::Native + Copy,

pub fn apply_mut<F>(&mut self, f: F)
where F: Fn(T::Native) -> T::Native + Copy,

pub fn apply_mut<'a, F>(&'a self, f: F) -> Self
where F: FnMut(&'a str) -> &'a str,

pub fn apply_mut<'a, F>(&'a self, f: F) -> Self
where F: FnMut(&'a [u8]) -> &'a [u8],

pub fn apply_as_ints<F>(&self, f: F) -> Series
where F: Fn(&Series) -> Series,

pub fn apply_as_ints<F>(&self, f: F) -> Series
where F: Fn(&Series) -> Series,

pub fn limit(&self, num_elements: usize) -> Self
where Self: Sized,

pub fn head(&self, length: Option<usize>) -> Self
where Self: Sized,

pub fn tail(&self, length: Option<usize>) -> Self
where Self: Sized,

impl<T> ChunkedArray<T>
where T: PolarsNumericType,

impl<T> ChunkedArray<T>
where T: PolarsDataType,

pub fn for_each<'a, F>(&'a self, op: F)
where F: FnMut(Option<T::Physical<'a>>),

impl<T> ChunkedArray<T>
where ChunkedArray<T>: IntoSeries, T: PolarsFloatType, T::Native: Float + IsFloat + SubAssign + Pow<T::Native, Output = T::Native>,

pub fn rolling_map_float<F>( &self, window_size: usize, f: F, ) -> PolarsResult<Self>
where F: FnMut(&mut ChunkedArray<T>) -> Option<T::Native>,

impl<T> ChunkedArray<T>
where T: PolarsFloatType, T::Native: Float,

impl<T> ChunkedArray<T>
where T: PolarsFloatType, T::Native: Float + Canonical,

impl<T> ChunkedArray<T>
where T: PolarsDataType,

impl<T> ChunkedArray<T>
where T: PolarsNumericType,

pub fn to_ndarray<N>(&self) -> PolarsResult<Array2<N::Native>>
where N: PolarsNumericType,

impl<T> ChunkedArray<T>
where T: PolarsDataType, for<'a> <T::Array as StaticArray>::ValueT<'a>: AsRef<[u8]>,

impl<T> ChunkedArray<T>
where T: PolarsDataType,

pub fn with_chunk<A>(name: PlSmallStr, arr: A) -> Self
where A: Array, T: PolarsDataType<Array = A>,

pub fn with_chunk_like<A>(ca: &Self, arr: A) -> Self
where A: Array, T: PolarsDataType<Array = A>,

pub fn from_chunk_iter<I>(name: PlSmallStr, iter: I) -> Self
where I: IntoIterator, T: PolarsDataType<Array = <I as IntoIterator>::Item>, <I as IntoIterator>::Item: Array,

pub fn from_chunk_iter_like<I>(ca: &Self, iter: I) -> Self
where I: IntoIterator, T: PolarsDataType<Array = <I as IntoIterator>::Item>, <I as IntoIterator>::Item: Array,

pub fn try_from_chunk_iter<I, A, E>( name: PlSmallStr, iter: I, ) -> Result<Self, E>
where I: IntoIterator<Item = Result<A, E>>, T: PolarsDataType<Array = A>, A: Array,

impl<T> ChunkedArray<T>
where T: PolarsNumericType,

pub fn apply_amortized_generic<F, K, V>(&self, f: F) -> ChunkedArray<V>
where V: PolarsDataType, F: FnMut(Option<AmortSeries>) -> Option<K> + Copy, V::Array: ArrayFromIter<Option<K>>,

pub fn try_apply_amortized_generic<F, K, V>( &self, f: F, ) -> PolarsResult<ChunkedArray<V>>
where V: PolarsDataType, F: FnMut(Option<AmortSeries>) -> PolarsResult<Option<K>> + Copy, V::Array: ArrayFromIter<Option<K>>,

pub fn for_each_amortized<F>(&self, f: F)
where F: FnMut(Option<AmortSeries>),

pub fn zip_and_apply_amortized<'a, T, I, F>( &'a self, ca: &'a ChunkedArray<T>, f: F, ) -> Self
where T: PolarsDataType, &'a ChunkedArray<T>: IntoIterator<IntoIter = I>, I: TrustedLen<Item = Option<T::Physical<'a>>>, F: FnMut(Option<AmortSeries>, Option<T::Physical<'a>>) -> Option<Series>,

pub fn binary_zip_and_apply_amortized<'a, T, U, F>( &'a self, ca1: &'a ChunkedArray<T>, ca2: &'a ChunkedArray<U>, f: F, ) -> Self
where T: PolarsDataType, U: PolarsDataType, F: FnMut(Option<AmortSeries>, Option<T::Physical<'a>>, Option<U::Physical<'a>>) -> Option<Series>,

pub fn apply_amortized<F>(&self, f: F) -> Self
where F: FnMut(AmortSeries) -> Series,

pub fn try_apply_amortized<F>(&self, f: F) -> PolarsResult<Self>
where F: FnMut(AmortSeries) -> PolarsResult<Series>,