Crate hdrs

hdrs is a HDFS Native Client in Rust based on hdfs-sys.

Examples

use std::io::{Read, Write};

use hdrs::Client;
use hdrs::ClientBuilder;
let fs = ClientBuilder::new("default").connect()?;

let mut f = fs
    .open_file()
    .write(true)
    .create(true)
    .open("/tmp/hello.txt")?;
let n = f.write("Hello, World!".as_bytes())?;

let mut f = fs.open_file().read(true).open("/tmp/hello.txt")?;
let mut buf = vec![0; 1024];
let n = f.read(&mut buf)?;

let _ = fs.remove_file("/tmp/hello.txt")?;

Features

• async_file: Enable async operation support
• vendored: Ignore lib loading logic, enforce to complie and staticly link libhdfs

Compiletime

hdrs depends on hdfs-sys which links libjvm to work.

Please make sure JAVA_HOME is set correctly:

export JAVA_HOME=/path/to/java
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}

• Enable vendored feature to compile libhdfs and link in static.
• Specify HDFS_LIB_DIR or HADOOP_HOME to load from specified path instead of compile.
• Specify HDFS_STATIC=1 to link libhdfs in static.
• And finally, we will fallback to compile libhdfs and link in static.

Runtime

hdrs depends on hdfs-sys which uses JNI to call functions provided by jars that provided by hadoop releases.

Please also make sure HADOOP_HOME, LD_LIBRARY_PATH, CLASSPATH is set correctly during runtime:

export HADOOP_HOME=/path/to/hadoop
export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${LD_LIBRARY_PATH}
export CLASSPATH=$(${HADOOP_HOME}/bin/hadoop classpath --glob)

If libhdfs is configued to link dynamiclly, please also add ${HADOOP_HOME}/lib/native in LD_LIBRARY_PATH to make sure linker can find libhdfs.so:

export LD_LIBRARY_PATH=${JAVA_HOME}/lib/server:${HADOOP_HOME}/lib/native:${LD_LIBRARY_PATH}

Structs

AsyncFile

Async version of file.

Client

Client holds the underlying connection to hdfs clusters.

ClientBuilder

The builder of connecting to hdfs clusters.

File

File will hold the underlying pointer to hdfsFile.

Metadata

Metadata of a path.

OpenOptions

Options and flags which can be used to configure how a file is opened.

Readdir

Readdir is an Iterator of metadata, returned by read_dir