pew 0.2.3

A benchmarking library for Rust based on google/benchmark
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157

# Pew

Pew is a benchmarking library written in Rust based of
[Google's C++ Benchmarking library](https://github.com/google/benchmark). It
is currently in very alpha stages (I'd consider it an MVP). It was built to be
able to do the following (which you cannot do in the rust benchmarking library):

1) Pause and unpause the benchmark timer
2) Run multiple benchmarks by specifying a range of arguments
3) Creating some initial state that gets passed to all runs of the benchmark

The benchmark will run for at least 1 second (or the user specified
`--min_duration`) and at least 8 runs (or the user specified `--min_runs`).
The average of these runs is output as the `Time (ns)` column.

More comprehensive docs can be found [here](https://docs.rs/pew/).

[TODO](https://github.com/akshaynanavati/pew/issues/4): Output other state, not
just the mean.

## Installation

This is available on [crates.io](https://crates.io/crates/pew). You can link it
as a library, or `cargo install` to also get the transpose script (see [output](#output)).

Add pew as a dependency in your `Cargo.toml`. Then, add this to your crate root:

```
#[macro_use]
extern crate pew;
```

I usually create benchmarks in the `bin/` directory and run them with
`cargo run --bin <benchmark-name> --release`.

## Usage

View the docs [here](https://docs.rs/pew/).

## Example

This can be used as follows:

```
fn main() {
    Benchmark::with_name("range_bench")
        .with_range(1 << 10, 1 << 20, 4)
        .with_generator(generator)
        .with_bench(pew_bench!(bm_vector1))
        .with_bench(pew_bench!(bm_vector2))
        .with_bench(pew_bench!(bm_vector3))
        .run();
}
```

There are more complete examples in the `examples/` directory of how to use this.

## Output

The output is a comma separated list of benchmark results (this is what
`cargo cargo run --example example1` will output):

```
Name,Time (ns)
range_bench/bm_vector_range/1024,104715
range_bench/bm_vector_range/4096,554838
range_bench/bm_vector_range/16384,2068971
range_bench/bm_vector_range/65536,7739376
range_bench/bm_vector_range/262144,31389948
range_bench/bm_vector_range/1048576,114633815
gen_bench/bm_vector_gen/1024,123643
gen_bench/bm_vector_gen/4096,545581
gen_bench/bm_vector_gen/16384,2590869
gen_bench/bm_vector_gen/65536,7799209
gen_bench/bm_vector_gen/262144,29498657
gen_bench/bm_vector_gen/1048576,113458415
```

You can also pass a `--filter` flag to the benchmark which would only run
benchmarks who's name contains the filter string. For example, running
`cargo cargo run --example example1 -- --filter gen` will output:

```
gen_bench/bm_vector_gen/1024,123643
gen_bench/bm_vector_gen/4096,545581
gen_bench/bm_vector_gen/16384,2590869
gen_bench/bm_vector_gen/65536,7799209
gen_bench/bm_vector_gen/262144,29498657
gen_bench/bm_vector_gen/1048576,113458415
```

while running `cargo cargo run --example example1 -- --filter 1024` will output:

```
range_bench/bm_vector_range/1024,104715
gen_bench/bm_vector_gen/1024,123643
```

Oftentimes I run multiple benchmarks on the same range and plot the results.
I find it is easier to plot the results by transposing the output. There is
a binary `pew_transpose` that does just that. It can be run by piping the
output of the benchmark code:

```
cargo run --example example1 | pew_transpose
```

The transposed output of the
above would be:

```
Size,bm_vector_range,bm_vector_gen
1024,105974,106845
4096,418835,409143
16384,1646391,1655855
65536,6566094,6668369
262144,27025772,26948818
1048576,108144239,107596626
```

[TODO](https://github.com/akshaynanavati/pew/issues/2): Add more output types
(e.g. pretty printed, JSON, etc).

## Cli

The Cli offers convenience flags and options while running benchmarks:

```
Akshay Nanavati <akshay.nanavati1@gmail.com>
A benchmarking library for Rust based on google/benchmark

USAGE:
    example1 [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -V, --version    Prints version information

OPTIONS:
    -f, --filter <FILTER>             Only run benchmarks that contain this string
    -d, --min_duration <RUN_UNTIL>    Run benchmarks till this time (in s) and then output average [default: 1]
    -r, --min_runs <MIN_RUNS>         Run benchmarks for at least this many runs [default: 8]
```

These can be passed to the main binary that is running your benchmark.

## License

This code is licensed under [Apache 2.0](https://www.apache.org/licenses/LICENSE-2.0).

## Contributing

This library is in very early stages and I'd love any contributions I can get. This is my
first time writing a becnhmarking library (and a library in Rust) and would love input from
those who are more experienced.

A good starting point would be the issues (which are also linked here as TODOs).