Andreas Auernhammer
48a4fd05f1
fix unnecessary memory alloc in Split ( #59 )
...
Split divided the data into `DataShards` blocks and allocates all parity blocks.
This change adds a check whether the capacity of data is large enough to hold all
data and parity blocks. It only allocates parity blocks if necessary.
2017-07-22 16:16:58 +02:00
Frank Wessels
0de37d7697
Add ReconstructData interface method ( #57 )
...
* Add ReconstructData interface method to allow reconstruction of any missing data shards
* Add support for just reconstructing data shards only to SteamEncoder.Reconstruct()
2017-07-20 12:15:46 +02:00
Klaus Post
0dd0a0e50c
Fix error grammar
...
Fixes #56
2017-07-16 17:00:58 +02:00
Fred Akalin
18d548df63
Add support for PAR1 ( #55 )
...
PAR1 is a file format which uses a Reed-Solomon code similar
to the current one, except it uses a different (flawed) coding
matrix.
Add support for it via a WithPAR1Matrix option, so that this code
can be used to encode/decode PAR1 files. Also add the option to
existing tests, and add a test demonstrating the flaw in PAR1's
coding matrix.
Also fix an mistakenly inverted test in testOpts().
Incidentally, PAR1 is obsoleted by PAR2, which uses GF(2^16)
and tries to fix the flaw in the coding matrix; however, PAR2's
coding matrix is still flawed! The real solution is to build the
coding matrix like in this repository.
PAR1 spec:
http://parchive.sourceforge.net/docs/specifications/parity-volume-spec-1.0/article-spec.html
Paper describing the (flawed) Reed-Solomon code used by PAR1:
http://web.eecs.utk.edu/~plank/plank/papers/CS-96-332.html
2017-06-20 20:24:57 +02:00
Fred Akalin
87c4e5ae75
Allow 256 total shards ( #54 )
...
* Allow 256 total shards
2017-06-19 11:26:52 +02:00
Klaus Post
5abf0ee302
Add options ( #46 )
...
* Add options
Make constants changeable as options.
The API remains backwards compatible.
* Update documentation.
* Fix line endings
* fmt
* fmt
* Use functions for parameters.
Much neater.
2017-02-19 11:13:22 +01:00
Peter C
c54154da9e
Add Inverse Matrix caching in a Thread-Safe Lookup Tree ( #36 )
...
* Add matrix inversion caching
* Benchmark and Parallel Benchmark tests for Reconstruct
2016-09-12 21:31:07 +02:00
Christian Muehlhaeuser
b1c8b4b073
Make Join return an error if a reconstruction is required first
...
If one or more required data shards are still nil and we can't correctly join
them before a reconstruction, return ErrReconstructRequired.
2016-08-05 19:23:08 +02:00
Harshavardhana
ba30981088
Add checks for data and parity to not exceed 255 shards in total.
...
Fixes #16
2016-06-03 01:31:01 -07:00
Klaus Post
4fadad8564
Update reedsolomon.go
...
Fix comment
2016-05-01 12:00:51 +02:00
Klaus Post
ed06f926b9
Merge pull request #20 from harshavardhana/fix
...
ErrShortData shouldn't be returned for data less than dataShards.
2016-05-01 11:58:47 +02:00
Harshavardhana
df175d2921
ErrShortData shouldn't be returned for data less than dataShards.
...
The reasoning behind this is that if we have a data block number
of 10, and parity of 10. Restricting input such that files of
size < 10Bytes should be errored out doesn't seem like the right
approach.
Most erasure subsystems will have static data and parity blocks,
in such case erroring out is not correct since reedsolomon itself
doesn't provide this limitation (please correct me here if i am
wrong :-)).
So removing the check itself is not a problem since most of the
data after the split would be padded with zeros, which is okay
and should be left as application optimization if they wish to
pack small files in this range.
ErrShortData will be still returned in case if the size of data
is empty, or in case of streaming if the size == 0.
2016-04-29 20:38:45 -07:00
Harshavardhana
0b630aea27
use bytes.Equal rather than bytes.Compare
2016-04-29 14:12:03 -07:00
xiaost
4048a541c8
Optimized encoding & decoding goroutines number
...
hardware: E5-2630 v2 (Intel x86-64 with ssse3)
software: linux, go1.6, GOMAXPROCS=2
Performances before after change
BenchmarkEncode10x2x10000-2 2884.95 MB/s 2837.93 MB/s 0.98x
BenchmarkEncode100x20x10000-2 593.93 MB/s 577.17 MB/s 0.97x
BenchmarkEncode17x3x1M-2 2903.74 MB/s 5197.99 MB/s 1.80x
BenchmarkEncode10x4x16M-2 1992.13 MB/s 3689.69 MB/s 1.85x
BenchmarkEncode5x2x1M-2 2883.78 MB/s 7506.19 MB/s 2.60x
BenchmarkEncode10x2x1M-2 3205.63 MB/s 7848.12 MB/s 2.45x
BenchmarkEncode10x4x1M-2 2218.35 MB/s 3998.35 MB/s 1.80x
BenchmarkEncode50x20x1M-2 579.24 MB/s 641.08 MB/s 1.11x
BenchmarkEncode17x3x16M-2 2652.36 MB/s 4775.41 MB/s 1.80x
BenchmarkVerify10x2x10000-2 1327.27 MB/s 1837.41 MB/s 1.38x
BenchmarkVerify50x5x50000-2 1481.89 MB/s 2684.57 MB/s 1.81x
BenchmarkVerify10x2x1M-2 1553.91 MB/s 5704.71 MB/s 3.67x
BenchmarkVerify5x2x1M-2 939.90 MB/s 4949.30 MB/s 5.26x
BenchmarkVerify10x4x1M-2 956.89 MB/s 3191.01 MB/s 3.33x
BenchmarkVerify50x20x1M-2 490.49 MB/s 823.46 MB/s 1.68x
BenchmarkVerify10x4x16M-2 1078.03 MB/s 3196.97 MB/s 2.97x
BenchmarkStreamEncode10x2x10000-2 2.40 MB/s 12.10 MB/s 5.04x
BenchmarkStreamEncode100x20x10000-2 6.72 MB/s 10.72 MB/s 1.60x
BenchmarkStreamEncode17x3x1M-2 390.75 MB/s 845.08 MB/s 2.16x
BenchmarkStreamEncode10x4x16M-2 1175.93 MB/s 1803.71 MB/s 1.53x
BenchmarkStreamEncode5x2x1M-2 207.85 MB/s 790.02 MB/s 3.80x
BenchmarkStreamEncode10x2x1M-2 296.77 MB/s 872.41 MB/s 2.94x
BenchmarkStreamEncode10x4x1M-2 264.43 MB/s 699.25 MB/s 2.64x
BenchmarkStreamEncode50x20x1M-2 284.93 MB/s 414.65 MB/s 1.46x
BenchmarkStreamEncode17x3x16M-2 1439.13 MB/s 1933.42 MB/s 1.34x
BenchmarkStreamVerify10x2x10000-2 2.33 MB/s 12.07 MB/s 5.18x
BenchmarkStreamVerify50x5x50000-2 86.53 MB/s 136.02 MB/s 1.57x
BenchmarkStreamVerify10x2x1M-2 315.65 MB/s 909.44 MB/s 2.88x
BenchmarkStreamVerify5x2x1M-2 180.45 MB/s 772.42 MB/s 4.28x
BenchmarkStreamVerify10x4x1M-2 310.35 MB/s 779.26 MB/s 2.51x
BenchmarkStreamVerify50x20x1M-2 547.23 MB/s 773.74 MB/s 1.41x
BenchmarkStreamVerify10x4x16M-2 4128.01 MB/s 6606.43 MB/s 1.60x
2016-04-12 15:41:22 +08:00
klauspost
180472d98f
Make documentation conform to go vet.
2015-11-03 12:09:36 +01:00
lukechampine
295bf27a3d
fix Split panic
2015-08-08 16:38:55 -04:00
lukechampine
0bd572bc5b
tweak Split/Join functions
2015-08-08 13:51:12 -04:00
lukechampine
64b705bbf6
fully test Reconstruct function
...
Well, I can't figure out how to trigger the Invert error.
It may not be possible; need more domain knowledge to be sure.
2015-08-08 13:50:18 -04:00
lukechampine
cf985d4451
remove unreachable checkShards case
...
this case would be caught by shardSize anyway
2015-08-08 13:50:18 -04:00
lukechampine
5784cfa7ff
remove impossible errors
2015-08-06 22:46:27 -04:00
klauspost
8ebf356efb
The number of data shards must be below 257. Check that and update documentation.
2015-06-23 13:39:57 +02:00
klauspost
5c2ef3ae72
Always check/return errors.
2015-06-23 12:16:26 +02:00
klauspost
7381e0b7b5
- Only run multiple goroutines if size is bigger than splitsize.
...
- Update docs
2015-06-23 11:18:29 +02:00
klauspost
83703c37ac
Add package documentation and clarify interface docs.
2015-06-22 15:12:05 +02:00
Klaus Post
5aa37c3492
Add AMD64 SSE3 Galois multiplication. Approximately 5-10x faster.
...
BenchmarkEncode10x2x10000 333.31 5827.17 17.48x
BenchmarkEncode10x2x10000-2 431.20 2802.53 6.50x
BenchmarkEncode10x2x10000-4 553.98 2432.95 4.39x
BenchmarkEncode10x2x10000-8 585.79 3469.61 5.92x
BenchmarkEncode100x20x10000 32.59 583.40 17.90x
BenchmarkEncode100x20x10000-2 59.52 726.70 12.21x
BenchmarkEncode100x20x10000-4 108.04 1363.25 12.62x
BenchmarkEncode100x20x10000-8 113.76 1274.62 11.20x
BenchmarkEncode17x3x1M 215.28 3141.85 14.59x
BenchmarkEncode17x3x1M-2 398.76 3650.12 9.15x
BenchmarkEncode17x3x1M-4 655.32 6071.11 9.26x
BenchmarkEncode17x3x1M-8 832.16 6616.47 7.95x
BenchmarkEncode10x4x16M 154.48 1357.30 8.79x
BenchmarkEncode10x4x16M-2 295.62 2377.92 8.04x
BenchmarkEncode10x4x16M-4 529.89 3519.49 6.64x
BenchmarkEncode10x4x16M-8 632.11 4521.90 7.15x
BenchmarkEncode5x2x1M 327.87 4879.09 14.88x
BenchmarkEncode5x2x1M-2 576.11 2599.20 4.51x
BenchmarkEncode5x2x1M-4 1043.65 3559.12 3.41x
BenchmarkEncode5x2x1M-8 1227.77 4255.34 3.47x
BenchmarkEncode10x2x1M 321.24 4574.68 14.24x
BenchmarkEncode10x2x1M-2 587.73 3100.28 5.28x
BenchmarkEncode10x2x1M-4 1101.96 4770.32 4.33x
BenchmarkEncode10x2x1M-8 1217.08 5812.17 4.78x
BenchmarkEncode10x4x1M 155.34 2037.27 13.11x
BenchmarkEncode10x4x1M-2 298.38 2470.97 8.28x
BenchmarkEncode10x4x1M-4 548.67 3603.15 6.57x
BenchmarkEncode10x4x1M-8 625.23 4827.42 7.72x
BenchmarkEncode50x20x1M 31.37 347.65 11.08x
BenchmarkEncode50x20x1M-2 59.81 713.28 11.93x
BenchmarkEncode50x20x1M-4 105.34 1175.47 11.16x
BenchmarkEncode50x20x1M-8 123.84 1491.91 12.05x
BenchmarkEncode17x3x16M 209.55 1861.59 8.88x
BenchmarkEncode17x3x16M-2 394.19 3331.73 8.45x
BenchmarkEncode17x3x16M-4 643.30 4942.74 7.68x
BenchmarkEncode17x3x16M-8 839.64 6213.43 7.40x
2015-06-21 21:23:22 +02:00
Klaus Post
17e9fa30f0
Add Join function for join data shards.
2015-06-21 13:25:12 +02:00
Klaus Post
437e364842
Adjust splitsize:
...
benchmark old ns/op new ns/op delta
BenchmarkEncode10x2x10000-2 243613 229413 -5.83%
BenchmarkEncode100x20x10000-2 23041318 19311104 -16.19%
BenchmarkEncode17x3x1M-2 54469780 49602836 -8.94%
BenchmarkEncode10x4x16M-2 674538600 647037000 -4.08%
Bigger sizes (1024) yeilds slower less speedup.
2015-06-20 20:32:52 +02:00
Klaus Post
9f6744582c
Also refactor Verify as well as multithreaded options.
...
benchmark old MB/s new MB/s speedup
BenchmarkEncode10x2x10000 182.29 308.59 1.69x
BenchmarkEncode100x20x10000 14.41 30.29 2.10x
BenchmarkEncode17x3x1M 38.52 196.43 5.10x
BenchmarkEncode10x4x16M 23.78 148.58 6.25x
2015-06-20 20:00:25 +02:00
Klaus Post
50a83296f4
Restructure to make one of the galois multiplication parts constant for the main loop.
2015-06-20 18:46:06 +02:00
Klaus Post
921adcb5d5
Use range to avoid one bound check per galMultiply:
...
benchmark old MB/s new MB/s speedup
BenchmarkVerify10x2x10000-2 235.24 253.36 1.08x
BenchmarkVerify50x5x50000-2 76.78 94.87 1.24x
BenchmarkVerify10x2x1M-2 180.90 209.73 1.16x
BenchmarkVerify5x2x1M-2 173.22 202.89 1.17x
BenchmarkVerify10x4x1M-2 71.51 118.20 1.65x
BenchmarkVerify50x20x1M-2 11.27 12.84 1.14x
BenchmarkVerify10x4x16M-2 44.74 50.07 1.12x
2015-06-20 14:51:11 +02:00
Klaus Post
419c6cc9e9
Add Splitter to help split data into shards.
2015-06-20 11:27:03 +02:00
Klaus Post
c5de03551c
Minor adjustments for golint.
2015-06-20 10:11:33 +02:00
Klaus Post
fdb7664a06
Hide internal matrix errors and document the used ones.
2015-06-20 10:08:59 +02:00
Klaus Post
481b39ba1c
Use a mutex to check if other goroutines have detected a difference. This avoid rare "close of closed channel" panic.
2015-06-19 19:15:31 +02:00
Klaus Post
e3aca6cd9d
Shorten the variable names and make an encoder interface, so it isn't possible to create it without calling New.
2015-06-19 18:54:58 +02:00
Klaus Post
8296e44d17
Update documentation, and test if all shards are zero-sized.
2015-06-19 18:21:38 +02:00
Klaus Post
755a781502
Remove unused types.
2015-06-19 17:27:22 +02:00
klauspost
2b4171b9e6
Initial version
2015-06-19 16:31:24 +02:00