Module delta_datafusion

Datafusion integration for Delta Table

Example:

use std::sync::Arc;
use datafusion::execution::context::SessionContext;

async {
  let mut ctx = SessionContext::new();
  let table = deltalake_core::open_table("./tests/data/simple_table")
      .await
      .unwrap();
  ctx.register_table("demo", Arc::new(table)).unwrap();

  let batches = ctx
      .sql("SELECT * FROM demo").await.unwrap()
      .collect()
      .await.unwrap();
};

Modules

• cdf
  Logical operators and physical executions for CDF
• expr
  Utility functions for Datafusion’s Expressions
• logical
  Logical Operations for DataFusion
• physical
  Physical Operations for DataFusion
• planner
  Custom planners for datafusion so that you can convert custom nodes, can be used to trace custom metrics in an operation

Structs

• DeltaColumn
  A wrapper for Deltafusion’s Column to preserve case-sensitivity during string conversion
• DeltaDataChecker
  Responsible for checking batches of data conform to table’s invariants, constraints and nullability.
• DeltaLogicalCodec
  Does serde on DeltaTables
• DeltaParserOptions
  A wrapper for sql_parser’s ParserOptions to capture sane default table defaults
• DeltaPhysicalCodec
  A codec for deltalake physical plans
• DeltaScan
  A wrapper for parquet scans
• DeltaScanConfig
  Include additional metadata columns during a DeltaScan
• DeltaScanConfigBuilder
  Used to specify if additional metadata columns are exposed to the user
• DeltaSessionConfig
  A wrapper for Deltafusion’s SessionConfig to capture sane default table defaults
• DeltaSessionContext
  A wrapper for Deltafusion’s SessionContext to capture sane default table defaults
• DeltaTableFactory
  Responsible for creating deltatables
• DeltaTableProvider
  A Delta table provider that enables additional metadata columns to be included during the scan
• FindFiles
  Representing the result of the find_files function.

Traits

• DataFusionMixins
  Convience trait for calling common methods on snapshot heirarchies

Functions

• find_files
  Finds files in a snapshot that match the provided predicate.