Struct bytes_utils::SegmentedBuf
source · pub struct SegmentedBuf<B> { /* private fields */ }
Expand description
A concatenation of multiple buffers into a large one, without copying the bytes over.
Note that this doesn’t provide a continuous slice view into them, it is split into the segments of the original smaller buffers.
This variants drop the inner buffers as they are exhausted and new ones can be added. But it internally keeps a VecDeque, therefore needs a heap allocation. If you don’t need the extending behaviour, but want to avoid the allocation, the SegmentedSlice can be used instead.
Why
This can be used, for example, if data of unknown length is coming over the network (for example, the bodies in hyper act a bit like this, it returns a stream of Bytes buffers). One might want to accumulate the whole body before acting on it, possibly by parsing it through serde or prost. Options would include:
- Have a
Vec<u8>
and extend it with each chunk. This needlessly copy the bytes every time and reallocates if the vector grows too large. - Repeatedly use chain, but this changes the type of the whole buffer, therefore needs to be boxed.
- Use hyper::body::aggregate to create a Buf implementation that concatenates all of them together, but lacks any kind of flexibility (like protecting against loading too much data into memory).
This type allows for concatenating multiple buffers, either all at once, or by incrementally pushing more buffers to the end.
Heterogeneous buffers
This expects all the buffers are of the same type. If different-typed buffers are needed, one
needs to use dynamic dispatch, either something like SegmentedBuf<Box<Buf>>
or
SegmentedBuf<&mut Buf>
.
Example
let mut buf = SegmentedBuf::new();
buf.push(Bytes::from("Hello"));
buf.push(Bytes::from(" "));
buf.push(Bytes::from("World"));
assert_eq!(3, buf.segments());
assert_eq!(11, buf.remaining());
assert_eq!(b"Hello", buf.chunk());
let mut out = String::new();
buf.reader().read_to_string(&mut out).expect("Doesn't cause IO errors");
assert_eq!("Hello World", out);
FIFO behaviour
The buffers are dropped once their data are completely consumed. Additionally, it is possible to add more buffers to the end, even while some of the previous buffers were partially or fully consumed. That makes it usable as kind of a queue (that operates on the buffers, not individual bytes).
let mut buf = SegmentedBuf::new();
buf.push(Bytes::from("Hello"));
assert_eq!(1, buf.segments());
let mut out = [0; 3];
buf.copy_to_slice(&mut out);
assert_eq!(&out, b"Hel");
assert_eq!(2, buf.remaining());
assert_eq!(1, buf.segments());
buf.push(Bytes::from("World"));
assert_eq!(7, buf.remaining());
assert_eq!(2, buf.segments());
buf.copy_to_slice(&mut out);
assert_eq!(&out, b"loW");
assert_eq!(4, buf.remaining());
assert_eq!(1, buf.segments());
Optimizations
The copy_to_bytes method tries to avoid copies by delegating into the underlying buffer if possible (if the whole request can be fulfilled using only a single buffer). If that one is optimized (for example, the Bytes returns a shared instance instead of making a copy), the copying is avoided. If the request is across a buffer boundary, a copy is made.
The chunks_vectored will properly output as many slices as possible, not just 1 as the default implementation does.
Implementations§
source§impl<B> SegmentedBuf<B>
impl<B> SegmentedBuf<B>
sourcepub fn into_inner(self) -> VecDeque<B>
pub fn into_inner(self) -> VecDeque<B>
Returns the yet unconsumed sequence of buffers.
Trait Implementations§
source§impl<B: Buf> Buf for SegmentedBuf<B>
impl<B: Buf> Buf for SegmentedBuf<B>
source§fn remaining(&self) -> usize
fn remaining(&self) -> usize
source§fn chunk(&self) -> &[u8] ⓘ
fn chunk(&self) -> &[u8] ⓘ
Buf::remaining()
. Note that this can return shorter slice (this allows
non-continuous internal representation). Read moresource§fn copy_to_bytes(&mut self, len: usize) -> Bytes
fn copy_to_bytes(&mut self, len: usize) -> Bytes
source§fn chunks_vectored<'a>(&'a self, dst: &mut [IoSlice<'a>]) -> usize
fn chunks_vectored<'a>(&'a self, dst: &mut [IoSlice<'a>]) -> usize
source§fn has_remaining(&self) -> bool
fn has_remaining(&self) -> bool
source§fn copy_to_slice(&mut self, dst: &mut [u8])
fn copy_to_slice(&mut self, dst: &mut [u8])
source§fn get_u16(&mut self) -> u16
fn get_u16(&mut self) -> u16
self
in big-endian byte order. Read moresource§fn get_u16_le(&mut self) -> u16
fn get_u16_le(&mut self) -> u16
self
in little-endian byte order. Read moresource§fn get_u16_ne(&mut self) -> u16
fn get_u16_ne(&mut self) -> u16
self
in native-endian byte order. Read moresource§fn get_i16(&mut self) -> i16
fn get_i16(&mut self) -> i16
self
in big-endian byte order. Read moresource§fn get_i16_le(&mut self) -> i16
fn get_i16_le(&mut self) -> i16
self
in little-endian byte order. Read moresource§fn get_i16_ne(&mut self) -> i16
fn get_i16_ne(&mut self) -> i16
self
in native-endian byte order. Read moresource§fn get_u32(&mut self) -> u32
fn get_u32(&mut self) -> u32
self
in the big-endian byte order. Read moresource§fn get_u32_le(&mut self) -> u32
fn get_u32_le(&mut self) -> u32
self
in the little-endian byte order. Read moresource§fn get_u32_ne(&mut self) -> u32
fn get_u32_ne(&mut self) -> u32
self
in native-endian byte order. Read moresource§fn get_i32(&mut self) -> i32
fn get_i32(&mut self) -> i32
self
in big-endian byte order. Read moresource§fn get_i32_le(&mut self) -> i32
fn get_i32_le(&mut self) -> i32
self
in little-endian byte order. Read moresource§fn get_i32_ne(&mut self) -> i32
fn get_i32_ne(&mut self) -> i32
self
in native-endian byte order. Read moresource§fn get_u64(&mut self) -> u64
fn get_u64(&mut self) -> u64
self
in big-endian byte order. Read moresource§fn get_u64_le(&mut self) -> u64
fn get_u64_le(&mut self) -> u64
self
in little-endian byte order. Read moresource§fn get_u64_ne(&mut self) -> u64
fn get_u64_ne(&mut self) -> u64
self
in native-endian byte order. Read moresource§fn get_i64(&mut self) -> i64
fn get_i64(&mut self) -> i64
self
in big-endian byte order. Read moresource§fn get_i64_le(&mut self) -> i64
fn get_i64_le(&mut self) -> i64
self
in little-endian byte order. Read moresource§fn get_i64_ne(&mut self) -> i64
fn get_i64_ne(&mut self) -> i64
self
in native-endian byte order. Read moresource§fn get_u128(&mut self) -> u128
fn get_u128(&mut self) -> u128
self
in big-endian byte order. Read moresource§fn get_u128_le(&mut self) -> u128
fn get_u128_le(&mut self) -> u128
self
in little-endian byte order. Read moresource§fn get_u128_ne(&mut self) -> u128
fn get_u128_ne(&mut self) -> u128
self
in native-endian byte order. Read moresource§fn get_i128(&mut self) -> i128
fn get_i128(&mut self) -> i128
self
in big-endian byte order. Read moresource§fn get_i128_le(&mut self) -> i128
fn get_i128_le(&mut self) -> i128
self
in little-endian byte order. Read moresource§fn get_i128_ne(&mut self) -> i128
fn get_i128_ne(&mut self) -> i128
self
in native-endian byte order. Read moresource§fn get_uint(&mut self, nbytes: usize) -> u64
fn get_uint(&mut self, nbytes: usize) -> u64
self
in big-endian byte order. Read moresource§fn get_uint_le(&mut self, nbytes: usize) -> u64
fn get_uint_le(&mut self, nbytes: usize) -> u64
self
in little-endian byte order. Read moresource§fn get_uint_ne(&mut self, nbytes: usize) -> u64
fn get_uint_ne(&mut self, nbytes: usize) -> u64
self
in native-endian byte order. Read moresource§fn get_int(&mut self, nbytes: usize) -> i64
fn get_int(&mut self, nbytes: usize) -> i64
self
in big-endian byte order. Read moresource§fn get_int_le(&mut self, nbytes: usize) -> i64
fn get_int_le(&mut self, nbytes: usize) -> i64
self
in little-endian byte order. Read moresource§fn get_int_ne(&mut self, nbytes: usize) -> i64
fn get_int_ne(&mut self, nbytes: usize) -> i64
self
in native-endian byte order. Read moresource§fn get_f32(&mut self) -> f32
fn get_f32(&mut self) -> f32
self
in big-endian byte order. Read moresource§fn get_f32_le(&mut self) -> f32
fn get_f32_le(&mut self) -> f32
self
in little-endian byte order. Read moresource§fn get_f32_ne(&mut self) -> f32
fn get_f32_ne(&mut self) -> f32
self
in native-endian byte order. Read moresource§fn get_f64(&mut self) -> f64
fn get_f64(&mut self) -> f64
self
in big-endian byte order. Read moresource§fn get_f64_le(&mut self) -> f64
fn get_f64_le(&mut self) -> f64
self
in little-endian byte order. Read moresource§fn get_f64_ne(&mut self) -> f64
fn get_f64_ne(&mut self) -> f64
self
in native-endian byte order. Read moresource§impl<B: Clone> Clone for SegmentedBuf<B>
impl<B: Clone> Clone for SegmentedBuf<B>
source§fn clone(&self) -> SegmentedBuf<B>
fn clone(&self) -> SegmentedBuf<B>
1.0.0 · source§fn clone_from(&mut self, source: &Self)
fn clone_from(&mut self, source: &Self)
source
. Read moresource§impl<B: Debug> Debug for SegmentedBuf<B>
impl<B: Debug> Debug for SegmentedBuf<B>
source§impl<B> Default for SegmentedBuf<B>
impl<B> Default for SegmentedBuf<B>
source§impl<B: Buf> Extend<B> for SegmentedBuf<B>
impl<B: Buf> Extend<B> for SegmentedBuf<B>
source§fn extend<T: IntoIterator<Item = B>>(&mut self, iter: T)
fn extend<T: IntoIterator<Item = B>>(&mut self, iter: T)
source§fn extend_one(&mut self, item: A)
fn extend_one(&mut self, item: A)
extend_one
)source§fn extend_reserve(&mut self, additional: usize)
fn extend_reserve(&mut self, additional: usize)
extend_one
)