cb7a0b5aef
Do fast by one multiplication ( #130 )
...
When multiplying by one we can use faster math.
2020-05-06 11:14:25 +02:00
65df535980
Make single goroutine encodes more efficient ( #122 )
...
Calculate the optimal per round size to keep data in cache when not using WithAutoGoroutines.
```
λ benchcmp before.txt after.txt
benchmark old ns/op new ns/op delta
BenchmarkParallel_8x8x05M-16 675225 321053 -52.45%
BenchmarkParallel_20x10x05M-16 3471988 600740 -82.70%
BenchmarkParallel_8x8x1M-16 3948606 728093 -81.56%
BenchmarkParallel_8x8x8M-16 47361588 5976467 -87.38%
BenchmarkParallel_8x8x32M-16 195044200 24365474 -87.51%
benchmark old MB/s new MB/s speedup
BenchmarkParallel_8x8x05M-16 6211.71 13064.22 2.10x
BenchmarkParallel_20x10x05M-16 3020.10 17454.73 5.78x
BenchmarkParallel_8x8x1M-16 2124.45 11521.34 5.42x
BenchmarkParallel_8x8x8M-16 1416.95 11228.85 7.92x
BenchmarkParallel_8x8x32M-16 1376.28 11017.04 8.00x
```
2020-05-03 19:37:22 +02:00
d2cfcb8065
Add commandline arg to disable asm for tests. ( #116 )
...
* Add commandline test args
2020-04-22 15:38:21 +02:00
0abe9de20c
Update tests ( #115 )
...
Don't create new slices.
2020-02-21 11:30:44 -08:00
ec2eb9fb8c
Split: Reduce memory allocation ( #103 )
...
* [Split] Reduce memory allocation in Split function
2019-06-25 16:28:24 +02:00
f5e73dcfe2
Split blocks into size divisible by 16
...
Older systems (typically without AVX2) are more sensitive to misaligned load+stores.
Add parameter to automatically set the number of goroutines.
name old time/op new time/op delta
Encode10x2x10000-8 18.4µs ± 1% 16.1µs ± 1% -12.43% (p=0.000 n=9+9)
Encode100x20x10000-8 692µs ± 1% 608µs ± 1% -12.10% (p=0.000 n=10+10)
Encode17x3x1M-8 1.78ms ± 5% 1.49ms ± 1% -16.63% (p=0.000 n=10+10)
Encode10x4x16M-8 21.5ms ± 5% 19.6ms ± 4% -8.74% (p=0.000 n=10+9)
Encode5x2x1M-8 343µs ± 2% 267µs ± 2% -22.22% (p=0.000 n=9+10)
Encode10x2x1M-8 858µs ± 5% 701µs ± 5% -18.34% (p=0.000 n=10+10)
Encode10x4x1M-8 1.34ms ± 1% 1.16ms ± 1% -13.19% (p=0.000 n=9+9)
Encode50x20x1M-8 30.3ms ± 4% 25.0ms ± 2% -17.51% (p=0.000 n=10+8)
Encode17x3x16M-8 26.9ms ± 1% 24.5ms ± 4% -9.13% (p=0.000 n=8+10)
name old speed new speed delta
Encode10x2x10000-8 5.45GB/s ± 1% 6.22GB/s ± 1% +14.20% (p=0.000 n=9+9)
Encode100x20x10000-8 1.44GB/s ± 1% 1.64GB/s ± 1% +13.77% (p=0.000 n=10+10)
Encode17x3x1M-8 10.0GB/s ± 5% 12.0GB/s ± 1% +19.88% (p=0.000 n=10+10)
Encode10x4x16M-8 7.81GB/s ± 5% 8.56GB/s ± 5% +9.58% (p=0.000 n=10+9)
Encode5x2x1M-8 15.3GB/s ± 2% 19.6GB/s ± 2% +28.57% (p=0.000 n=9+10)
Encode10x2x1M-8 12.2GB/s ± 5% 15.0GB/s ± 5% +22.45% (p=0.000 n=10+10)
Encode10x4x1M-8 7.84GB/s ± 1% 9.03GB/s ± 1% +15.19% (p=0.000 n=9+9)
Encode50x20x1M-8 1.73GB/s ± 4% 2.09GB/s ± 4% +20.59% (p=0.000 n=10+9)
Encode17x3x16M-8 10.6GB/s ± 1% 11.7GB/s ± 4% +10.12% (p=0.000 n=8+10)
2017-11-18 22:00:55 +01:00
61c22eab55
Cauchy Matrix option ( #70 )
...
* Experimental Cauchy Matrix
Experimental support for Cauchy style matrix
http://web.eecs.utk.edu/~plank/plank/papers/CS-05-569.pdf
All matrices appear reversible.
* Remove Go 1.5 and 1.6 from CI tests.
* Fix comment.
* Increase max number of goroutines+docs.
2017-10-01 14:02:11 +02:00
ddcafc661e
Allow reconstructing into pre-allocated memory. ( #66 )
...
This changes the interface of Reconstruct and ReconstructData to accept
slices of zero length but sufficient capacity for shards to reconstruct,
and reslices them instead of allocating new memory.
2017-09-20 21:08:24 +02:00
d78bf472d8
add Update parity function ( #60 )
...
Add Update parity function
2017-08-20 11:42:39 +02:00
dc6af2dce5
Minor cleanup ( #61 )
...
* Remove some benchmarks
* Format tables a bit.
* Doc cleanup
2017-08-13 22:38:27 +02:00
0de37d7697
Add ReconstructData interface method ( #57 )
...
* Add ReconstructData interface method to allow reconstruction of any missing data shards
* Add support for just reconstructing data shards only to SteamEncoder.Reconstruct()
2017-07-20 12:15:46 +02:00
18d548df63
Add support for PAR1 ( #55 )
...
PAR1 is a file format which uses a Reed-Solomon code similar
to the current one, except it uses a different (flawed) coding
matrix.
Add support for it via a WithPAR1Matrix option, so that this code
can be used to encode/decode PAR1 files. Also add the option to
existing tests, and add a test demonstrating the flaw in PAR1's
coding matrix.
Also fix an mistakenly inverted test in testOpts().
Incidentally, PAR1 is obsoleted by PAR2, which uses GF(2^16)
and tries to fix the flaw in the coding matrix; however, PAR2's
coding matrix is still flawed! The real solution is to build the
coding matrix like in this repository.
PAR1 spec:
http://parchive.sourceforge.net/docs/specifications/parity-volume-spec-1.0/article-spec.html
Paper describing the (flawed) Reed-Solomon code used by PAR1:
http://web.eecs.utk.edu/~plank/plank/papers/CS-96-332.html
2017-06-20 20:24:57 +02:00
87c4e5ae75
Allow 256 total shards ( #54 )
...
* Allow 256 total shards
2017-06-19 11:26:52 +02:00
5abf0ee302
Add options ( #46 )
...
* Add options
Make constants changeable as options.
The API remains backwards compatible.
* Update documentation.
* Fix line endings
* fmt
* fmt
* Use functions for parameters.
Much neater.
2017-02-19 11:13:22 +01:00
c54154da9e
Add Inverse Matrix caching in a Thread-Safe Lookup Tree ( #36 )
...
* Add matrix inversion caching
* Benchmark and Parallel Benchmark tests for Reconstruct
2016-09-12 21:31:07 +02:00
b1c8b4b073
Make Join return an error if a reconstruction is required first
...
If one or more required data shards are still nil and we can't correctly join
them before a reconstruction, return ErrReconstructRequired.
2016-08-05 19:23:08 +02:00
ba30981088
Add checks for data and parity to not exceed 255 shards in total.
...
Fixes #16
2016-06-03 01:31:01 -07:00
9f0bea8a29
Tests: backport go1.6 rand.Read for speedup tests
2016-04-07 18:34:47 +08:00
976a24f33b
Move examples to separate file/package
...
This makes the reedsolomon package prefix show up in the documentation examples.
+ StreamEncoder example.
2015-11-03 12:12:42 +01:00
86bd0f239b
seed RNG in TestSplitJoin
2015-08-08 18:20:40 -04:00
458f451fc2
add codeSomeShardsP test
2015-08-08 13:52:00 -04:00
bb7bd0036a
fully test Split/Join functions
2015-08-08 13:51:11 -04:00
64b705bbf6
fully test Reconstruct function
...
Well, I can't figure out how to trigger the Invert error.
It may not be possible; need more domain knowledge to be sure.
2015-08-08 13:50:18 -04:00
f81ea8daaf
fully test Verify function
2015-08-08 13:50:18 -04:00
0238782585
fully test Encode function
2015-08-08 13:50:18 -04:00
10fbe96890
use slice literal
2015-08-06 22:56:32 -04:00
640ab74d9d
fully test the New function
2015-08-06 22:47:11 -04:00
d31049df42
Add another example that shows that sets can be xor'ed and still remain valid.
2015-06-23 14:35:16 +02:00
8ebf356efb
The number of data shards must be below 257. Check that and update documentation.
2015-06-23 13:39:57 +02:00
6861078d3b
Add more information to example.
2015-06-22 15:52:10 +02:00
0cb21eccc5
Rename example function.
2015-06-22 15:48:52 +02:00
7794948a5b
Add split/merge example.
2015-06-22 15:44:22 +02:00
619e2b7d65
Add benchmark with 17 data shards and 3 parity shards with 16MB each, and correct comments.
2015-06-21 17:07:17 +02:00
ab50161bb9
Update benchmarks.
2015-06-20 20:51:26 +02:00
36a0e57744
Begin docs.
2015-06-20 13:10:51 +02:00
d54843ee41
Add Encoder example (and test)
2015-06-20 11:29:26 +02:00
c5de03551c
Minor adjustments for golint.
2015-06-20 10:11:33 +02:00
cf70107291
Add verification test that also tests failure.
2015-06-19 19:20:44 +02:00
e3aca6cd9d
Shorten the variable names and make an encoder interface, so it isn't possible to create it without calling New.
2015-06-19 18:54:58 +02:00
67f8d8b8c7
Add another benchmark.
2015-06-19 18:25:48 +02:00
2b4171b9e6
Initial version
2015-06-19 16:31:24 +02:00
Klaus Post