# Hotsauce S3 Benchmark Version 0.1 ## Introduction hsbench is an S3 compatable benchmark originally based on [wasabi-tech/s3-benchmark](https://github.com/wasabi-tech/s3-benchmark). While there are already several S3 compatable benchmark suites available, each has various tradeoffs. What seemed to be missing was a simple yet extremely fast benchmark that could easily be scripted into orchestration frameworks to run tests at scale. hsbench tries to fill that niche. The current release of hsbench is considered to be alpha level software and may contain bugs. ## Features hsbench tries to improve on the original Wasabi s3-benchmark in the following ways: * Threads can distribute IOS across an arbitrary number of buckets. * Tests can be run individually and externally coordinated across multiple clients. * Intermediate results are logged periodically at user-defined intervals. * Min/avg/max/percentile latency results are included. * Test length can be limited either by duration or maximum number of objects. * Object prefixes can be set to test large object names (12 bytes reserved for uniqueness) * Bucket/Object prefixes can be used to allow multiple clients to target the same buckets ## Limitations * hsbench does not yet log log/csv/json output (will be added pending community feedback) * hsbench has no built-in provisions for making graphs * hsbench does not yet support mixed IO workloads * hsbench is still in alpha and options/output may change at any moment ## Prerequisites To leverage this tool, the following prerequisites apply: * Git development environment * Go version 1.7 or newer * Access to an S3 compatable storage service (AWS, Ceph, etc) ## Install The easiest way to install hsbench is to use go's built in github support: ``` go get github.com/markhpc/hsbench ``` Then in the hsbench src directory run 'go build': ``` $ pwd /home/perf/go/src/github.com/markhpc/hsbench $ go build ``` ## Usage ``` $ ./hsbench --help USAGE: ./hsbench [OPTIONS] OPTIONS: -a string Access key -b int Number of buckets to distribute IOs across (default 1) -bp string Prefix for buckets (default "hotsauce_bench") -d int Maximum test duration in seconds <-1 for unlimited> (default 60) -j string Write JSON output to this file -l int Number of times to repeat test (default 1) -m string Run modes in order. See NOTES for more info (default "cxipgdx") -n int Maximum number of objects <-1 for unlimited> (default -1) -o string Write CSV output to this file -op string Prefix for objects -r string Region for testing (default "us-east-1") -ri float Number of seconds between report intervals (default 1) -s string Secret key -t int Number of threads to run (default 1) -u string URL for host with method prefix -z string Size of objects in bytes with postfix K, M, and G (default "1M") NOTES: - Valid mode types for the -m mode string are: c: clear all existing objects from buckets (requires lookups) x: delete buckets i: initialize buckets p: put objects in buckets g: get objects from buckets d: delete objects from buckets These modes are processed in-order and can be repeated, ie "ippgd" will initialize the buckets, put the objects, reput the objects, get the objects, and then delete the objects. The repeat flag will repeat this whole process the specified number of times. ``` ## Example Benchmark Below is an example run of the benchmark using a 10s test duration, 10 threads, 10 buckets, and a 4K object size against a Ceph RadosGW backed by a single Ceph OSD running on an Intel P3700 NVMe device. ``` $ ./hsbench -a 3JZ0SVK94Z55OZU5J1N0 -s OdzEPyDDZ0ls1haDUu1NVWkJDcnG74Lb7XylfXRM -u http://127.0.0.1:7480 -z 4K -d 10 -t 10 -b 10 2019/08/19 11:21:06 Hotsauce S3 Benchmark Version 0.1 2019/08/19 11:21:06 Parameters: 2019/08/19 11:21:06 url=http://127.0.0.1:7480 2019/08/19 11:21:06 object_prefix= 2019/08/19 11:21:06 bucket_prefix=hotsauce_bench 2019/08/19 11:21:06 region=us-east-1 2019/08/19 11:21:06 modes=cxipgdx 2019/08/19 11:21:06 output= 2019/08/19 11:21:06 json_output= 2019/08/19 11:21:06 object_count=-1 2019/08/19 11:21:06 bucket_count=10 2019/08/19 11:21:06 duration=10 2019/08/19 11:21:06 threads=10 2019/08/19 11:21:06 loops=1 2019/08/19 11:21:06 size=4K 2019/08/19 11:21:06 interval=1.000000 2019/08/19 11:21:06 Running Loop 0 BUCKET CLEAR TEST 2019/08/19 11:21:06 Loop: 0, Int: TOTAL, Dur(s): 0.0, Mode: BCLR, Ops: 0, MB/s: 0.00, IO/s: 0, Lat(ms): [ min: 0.0, avg: 0.0, 99%: 0.0, max: 0.0 ], Slowdowns: 0 2019/08/19 11:21:06 Running Loop 0 BUCKET DELETE TEST 2019/08/19 11:21:06 Loop: 0, Int: TOTAL, Dur(s): 0.0, Mode: BDEL, Ops: 0, MB/s: 0.00, IO/s: 0, Lat(ms): [ min: 0.0, avg: 0.0, 99%: 0.0, max: 0.0 ], Slowdowns: 0 2019/08/19 11:21:06 Running Loop 0 BUCKET INIT TEST 2019/08/19 11:21:06 Loop: 0, Int: TOTAL, Dur(s): 0.0, Mode: BINIT, Ops: 10, MB/s: 0.00, IO/s: 944, Lat(ms): [ min: 7.9, avg: 9.3, 99%: 10.4, max: 10.4 ], Slowdowns: 0 2019/08/19 11:21:06 Running Loop 0 OBJECT PUT TEST 2019/08/19 11:21:07 Loop: 0, Int: 0, Dur(s): 1.0, Mode: PUT, Ops: 5209, MB/s: 20.35, IO/s: 5209, Lat(ms): [ min: 1.3, avg: 1.9, 99%: 3.5, max: 8.4 ], Slowdowns: 0 2019/08/19 11:21:08 Loop: 0, Int: 1, Dur(s): 1.0, Mode: PUT, Ops: 5076, MB/s: 19.83, IO/s: 5076, Lat(ms): [ min: 1.2, avg: 2.0, 99%: 3.1, max: 42.3 ], Slowdowns: 0 2019/08/19 11:21:09 Loop: 0, Int: 2, Dur(s): 1.0, Mode: PUT, Ops: 4319, MB/s: 16.87, IO/s: 4319, Lat(ms): [ min: 1.3, avg: 2.3, 99%: 4.4, max: 58.0 ], Slowdowns: 0 2019/08/19 11:21:10 Loop: 0, Int: 3, Dur(s): 1.0, Mode: PUT, Ops: 4288, MB/s: 16.75, IO/s: 4288, Lat(ms): [ min: 1.3, avg: 2.3, 99%: 3.5, max: 63.1 ], Slowdowns: 0 2019/08/19 11:21:11 Loop: 0, Int: 4, Dur(s): 1.0, Mode: PUT, Ops: 4549, MB/s: 17.77, IO/s: 4549, Lat(ms): [ min: 1.3, avg: 2.2, 99%: 6.8, max: 57.9 ], Slowdowns: 0 2019/08/19 11:21:12 Loop: 0, Int: 5, Dur(s): 1.0, Mode: PUT, Ops: 4447, MB/s: 17.37, IO/s: 4447, Lat(ms): [ min: 1.3, avg: 2.2, 99%: 3.5, max: 58.8 ], Slowdowns: 0 2019/08/19 11:21:13 Loop: 0, Int: 6, Dur(s): 1.0, Mode: PUT, Ops: 4260, MB/s: 16.64, IO/s: 4260, Lat(ms): [ min: 1.3, avg: 2.4, 99%: 5.8, max: 58.4 ], Slowdowns: 0 2019/08/19 11:21:14 Loop: 0, Int: 7, Dur(s): 1.0, Mode: PUT, Ops: 5202, MB/s: 20.32, IO/s: 5202, Lat(ms): [ min: 1.3, avg: 1.9, 99%: 3.1, max: 10.3 ], Slowdowns: 0 2019/08/19 11:21:15 Loop: 0, Int: 8, Dur(s): 1.0, Mode: PUT, Ops: 5213, MB/s: 20.36, IO/s: 5213, Lat(ms): [ min: 1.3, avg: 1.9, 99%: 3.0, max: 13.3 ], Slowdowns: 0 2019/08/19 11:21:16 Loop: 0, Int: 9, Dur(s): 1.0, Mode: PUT, Ops: 5210, MB/s: 20.35, IO/s: 5210, Lat(ms): [ min: 1.2, avg: 1.9, 99%: 3.0, max: 9.6 ], Slowdowns: 0 2019/08/19 11:21:16 Loop: 0, Int: TOTAL, Dur(s): 10.0, Mode: PUT, Ops: 47783, MB/s: 18.66, IO/s: 4777, Lat(ms): [ min: 1.2, avg: 2.1, 99%: 3.4, max: 63.1 ], Slowdowns: 0 2019/08/19 11:21:16 Running Loop 0 OBJECT GET TEST 2019/08/19 11:21:17 Loop: 0, Int: 0, Dur(s): 1.0, Mode: GET, Ops: 1211, MB/s: 4.73, IO/s: 1211, Lat(ms): [ min: 0.6, avg: 0.9, 99%: 1.5, max: 2.0 ], Slowdowns: 0 2019/08/19 11:21:18 Loop: 0, Int: 1, Dur(s): 1.0, Mode: GET, Ops: 1182, MB/s: 4.62, IO/s: 1182, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.3, max: 2.1 ], Slowdowns: 0 2019/08/19 11:21:19 Loop: 0, Int: 2, Dur(s): 1.0, Mode: GET, Ops: 1110, MB/s: 4.34, IO/s: 1110, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 1.5 ], Slowdowns: 0 2019/08/19 11:21:20 Loop: 0, Int: 3, Dur(s): 1.0, Mode: GET, Ops: 1072, MB/s: 4.19, IO/s: 1072, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 1.3 ], Slowdowns: 0 2019/08/19 11:21:21 Loop: 0, Int: 4, Dur(s): 1.0, Mode: GET, Ops: 1098, MB/s: 4.29, IO/s: 1098, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 3.9 ], Slowdowns: 0 2019/08/19 11:21:22 Loop: 0, Int: 5, Dur(s): 1.0, Mode: GET, Ops: 1115, MB/s: 4.36, IO/s: 1115, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 1.4 ], Slowdowns: 0 2019/08/19 11:21:23 Loop: 0, Int: 6, Dur(s): 1.0, Mode: GET, Ops: 1110, MB/s: 4.34, IO/s: 1110, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.1, max: 1.5 ], Slowdowns: 0 2019/08/19 11:21:24 Loop: 0, Int: 7, Dur(s): 1.0, Mode: GET, Ops: 1079, MB/s: 4.21, IO/s: 1079, Lat(ms): [ min: 0.6, avg: 0.7, 99%: 1.1, max: 1.7 ], Slowdowns: 0 2019/08/19 11:21:25 Loop: 0, Int: 8, Dur(s): 1.0, Mode: GET, Ops: 1089, MB/s: 4.25, IO/s: 1089, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 1.6 ], Slowdowns: 0 2019/08/19 11:21:26 Loop: 0, Int: 9, Dur(s): 1.0, Mode: GET, Ops: 1156, MB/s: 4.52, IO/s: 1156, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 5.4 ], Slowdowns: 0 2019/08/19 11:21:26 Loop: 0, Int: TOTAL, Dur(s): 10.0, Mode: GET, Ops: 11222, MB/s: 4.37, IO/s: 1118, Lat(ms): [ min: 0.6, avg: 0.8, 99%: 1.2, max: 5.4 ], Slowdowns: 0 2019/08/19 11:21:26 Running Loop 0 OBJECT DELETE TEST 2019/08/19 11:21:27 Loop: 0, Int: 0, Dur(s): 1.0, Mode: DEL, Ops: 5673, MB/s: 22.16, IO/s: 5673, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.7, max: 10.0 ], Slowdowns: 0 2019/08/19 11:21:28 Loop: 0, Int: 1, Dur(s): 1.0, Mode: DEL, Ops: 5597, MB/s: 21.86, IO/s: 5597, Lat(ms): [ min: 1.1, avg: 1.8, 99%: 2.9, max: 10.5 ], Slowdowns: 0 2019/08/19 11:21:29 Loop: 0, Int: 2, Dur(s): 1.0, Mode: DEL, Ops: 5123, MB/s: 20.01, IO/s: 5123, Lat(ms): [ min: 1.1, avg: 1.9, 99%: 3.2, max: 67.1 ], Slowdowns: 0 2019/08/19 11:21:30 Loop: 0, Int: 3, Dur(s): 1.0, Mode: DEL, Ops: 5547, MB/s: 21.67, IO/s: 5547, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.7, max: 15.2 ], Slowdowns: 0 2019/08/19 11:21:31 Loop: 0, Int: 4, Dur(s): 1.0, Mode: DEL, Ops: 5604, MB/s: 21.89, IO/s: 5604, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.7, max: 11.6 ], Slowdowns: 0 2019/08/19 11:21:32 Loop: 0, Int: 5, Dur(s): 1.0, Mode: DEL, Ops: 5610, MB/s: 21.91, IO/s: 5610, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.8, max: 8.4 ], Slowdowns: 0 2019/08/19 11:21:33 Loop: 0, Int: 6, Dur(s): 1.0, Mode: DEL, Ops: 5526, MB/s: 21.59, IO/s: 5526, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.8, max: 11.6 ], Slowdowns: 0 2019/08/19 11:21:34 Loop: 0, Int: 7, Dur(s): 1.0, Mode: DEL, Ops: 5538, MB/s: 21.63, IO/s: 5538, Lat(ms): [ min: 1.2, avg: 1.8, 99%: 2.8, max: 8.7 ], Slowdowns: 0 2019/08/19 11:21:35 Loop: 0, Int: TOTAL, Dur(s): 8.6, Mode: DEL, Ops: 47783, MB/s: 21.61, IO/s: 5532, Lat(ms): [ min: 1.1, avg: 1.8, 99%: 2.8, max: 67.1 ], Slowdowns: 0 2019/08/19 11:21:35 Running Loop 0 BUCKET DELETE TEST 2019/08/19 11:21:35 Loop: 0, Int: TOTAL, Dur(s): 0.0, Mode: BDEL, Ops: 10, MB/s: 0.00, IO/s: 250, Lat(ms): [ min: 35.1, avg: 37.1, 99%: 39.4, max: 39.4 ], Slowdowns: 0 ``` One notable point is that like the s3-benchmark program it is based on, hsbench has relatively low CPU overhead compared to some other S3 benchmarks. During the 4K PUT phase of the above test: ``` 22628 root 20 0 6700416 5.308g 28072 S 413.3 8.4 676:28.86 ceph-osd 23329 root 20 0 1658700 253068 21320 S 333.3 0.4 530:06.76 radosgw 9017 perf 20 0 2184580 33612 6568 S 173.3 0.1 0:05.25 hsbench ```