Commit Graph

20 Commits (49be604db04f8eff8a2103f7f92530a5efe04406)

Author SHA1 Message Date
Elias Naur 49be604db0
Add an efficient implementation of shard counts up to 65536 (#191)
This is a O(n*log n) implementation of Reed-Solomon
codes, ported from the C++ library https://github.com/catid/leopard.

The implementation is based on the paper

"Novel Polynomial Basis with Fast Fourier Transform
and Its Application to Reed-Solomon Erasure Codes"

Several TODOs are left for future commits:

- Performance optimizations, in particular SIMD and multiple goroutines
- Documentation
- Detailed Tests
- Merging of reedSolomonFF16 and reedSolomon types
- Turn the straight C++ port into more idiomatic Go

This change also bumps one race testing timeout, to ensure adequate
time on CI.
2022-07-21 14:27:10 +02:00
Klaus Post abb309aca7
Fix stream allocations (#129)
Numbers speak for themselves:

```
benchmark                                old ns/op     new ns/op     delta
BenchmarkStreamEncode10x2x10000-32       4792420       7937          -99.83%
BenchmarkStreamEncode100x20x10000-32     38424066      473285        -98.77%
BenchmarkStreamEncode17x3x1M-32          8195036       1482191       -81.91%
BenchmarkStreamEncode10x4x16M-32         21356715      18051773      -15.47%
BenchmarkStreamEncode5x2x1M-32           3295827       412301        -87.49%
BenchmarkStreamEncode10x2x1M-32          5249011       798828        -84.78%
BenchmarkStreamEncode10x4x1M-32          6392974       904818        -85.85%
BenchmarkStreamEncode50x20x1M-32         29083474      7199282       -75.25%
BenchmarkStreamEncode17x3x16M-32         32451850      28036421      -13.61%
BenchmarkStreamVerify10x2x10000-32       4858416       12988         -99.73%
BenchmarkStreamVerify50x5x50000-32       17047361      377003        -97.79%
BenchmarkStreamVerify10x2x1M-32          4869964       887214        -81.78%
BenchmarkStreamVerify5x2x1M-32           3282999       591669        -81.98%
BenchmarkStreamVerify10x4x1M-32          5824392       1230888       -78.87%
BenchmarkStreamVerify50x20x1M-32         27301648      6204613       -77.27%
BenchmarkStreamVerify10x4x16M-32         8508963       18845695      +121.48%

benchmark                                old MB/s     new MB/s     speedup
BenchmarkStreamEncode10x2x10000-32       20.87        12599.82     603.73x
BenchmarkStreamEncode100x20x10000-32     26.03        2112.89      81.17x
BenchmarkStreamEncode17x3x1M-32          2175.19      12026.65     5.53x
BenchmarkStreamEncode10x4x16M-32         7855.71      9293.94      1.18x
BenchmarkStreamEncode5x2x1M-32           1590.76      12716.14     7.99x
BenchmarkStreamEncode10x2x1M-32          1997.66      13126.43     6.57x
BenchmarkStreamEncode10x4x1M-32          1640.20      11588.81     7.07x
BenchmarkStreamEncode50x20x1M-32         1802.70      7282.50      4.04x
BenchmarkStreamEncode17x3x16M-32         8788.80      10172.93     1.16x
BenchmarkStreamVerify10x2x10000-32       20.58        7699.20      374.11x
BenchmarkStreamVerify50x5x50000-32       293.30       13262.49     45.22x
BenchmarkStreamVerify10x2x1M-32          2153.15      11818.75     5.49x
BenchmarkStreamVerify5x2x1M-32           1596.98      8861.17      5.55x
BenchmarkStreamVerify10x4x1M-32          1800.32      8518.86      4.73x
BenchmarkStreamVerify50x20x1M-32         1920.35      8449.97      4.40x
BenchmarkStreamVerify10x4x16M-32         19717.11     8902.41      0.45x
```
2020-05-05 16:35:35 +02:00
Christian Muehlhaeuser 4681100338 Removed unused struct members (#106)
creads & cwrites both seem to be unused.
2019-09-27 16:31:11 -07:00
Klaus Post 09979cdf93 Start documentation with method name.
Replaces #92
2019-02-15 15:31:43 +01:00
Frank Wessels 0de37d7697 Add ReconstructData interface method (#57)
* Add ReconstructData interface method to allow reconstruction of any missing data shards
* Add support for just reconstructing data shards only to SteamEncoder.Reconstruct()
2017-07-20 12:15:46 +02:00
Klaus Post 0dd0a0e50c Fix error grammar
Fixes #56
2017-07-16 17:00:58 +02:00
Klaus Post 5abf0ee302 Add options (#46)
* Add options

Make constants changeable as options.

The API remains backwards compatible.

* Update documentation.

* Fix line endings

* fmt

* fmt

* Use functions for parameters.

Much neater.
2017-02-19 11:13:22 +01:00
Harshavardhana df175d2921 ErrShortData shouldn't be returned for data less than dataShards.
The reasoning behind this is that if we have a data block number
of 10, and parity of 10.  Restricting input such that files of
size < 10Bytes should be errored out doesn't seem like the right
approach.

Most erasure subsystems will have static data and parity blocks,
in such case erroring out is not correct since reedsolomon itself
doesn't provide this limitation (please correct me here if i am
wrong :-)).

So removing the check itself is not a problem since most of the
data after the split would be padded with zeros, which is okay
and should be left as application optimization if they wish to
pack small files in this range.

ErrShortData will be still returned in case if the size of data
is empty, or in case of streaming if the size == 0.
2016-04-29 20:38:45 -07:00
klauspost e125c8bc8c Fix comment. 2015-11-03 12:36:28 +01:00
klauspost 94aef4a146 Add stream order information. 2015-11-03 12:10:37 +01:00
klauspost 5dedbfc3ff Update streaming documentation. 2015-10-27 15:59:47 +01:00
Klaus Post 4c33f7e599 Remove duplicated package documentation. 2015-10-27 15:39:19 +01:00
klauspost 64a2a9b848 Add performance note. 2015-10-27 14:54:40 +01:00
klauspost 7d91d8d953 Create separate read/write errors. 2015-10-27 11:11:50 +01:00
klauspost ff9a5aff03 Remove StreamError for shard size mismatches
Since the result is pretty random (especially on concurrent reads),
we just return the plan ErrShardSize.
2015-10-27 10:54:08 +01:00
klauspost e78a382960 Add specific error for streams, to help identify the faulty stream. 2015-10-27 10:36:29 +01:00
klauspost 0500314cc5 Add concurrent reads/writes to streaming interface. 2015-10-26 15:59:02 +01:00
klauspost 8940ce33cc Return ErrShardNoData if there is no data in stream. 2015-10-26 14:31:10 +01:00
Klaus Post 32af654e33 Trim output shards to be equal size to input shards on last read. 2015-10-24 12:42:04 +02:00
klauspost 3fada41987 Add streaming API 2015-10-23 16:14:55 +02:00