c54154da9e
Add Inverse Matrix caching in a Thread-Safe Lookup Tree ( #36 )
...
* Add matrix inversion caching
* Benchmark and Parallel Benchmark tests for Reconstruct
2016-09-12 21:31:07 +02:00
b1c8b4b073
Make Join return an error if a reconstruction is required first
...
If one or more required data shards are still nil and we can't correctly join
them before a reconstruction, return ErrReconstructRequired.
2016-08-05 19:23:08 +02:00
ba30981088
Add checks for data and parity to not exceed 255 shards in total.
...
Fixes #16
2016-06-03 01:31:01 -07:00
4fadad8564
Update reedsolomon.go
...
Fix comment
2016-05-01 12:00:51 +02:00
ed06f926b9
Merge pull request #20 from harshavardhana/fix
...
ErrShortData shouldn't be returned for data less than dataShards.
2016-05-01 11:58:47 +02:00
df175d2921
ErrShortData shouldn't be returned for data less than dataShards.
...
The reasoning behind this is that if we have a data block number
of 10, and parity of 10. Restricting input such that files of
size < 10Bytes should be errored out doesn't seem like the right
approach.
Most erasure subsystems will have static data and parity blocks,
in such case erroring out is not correct since reedsolomon itself
doesn't provide this limitation (please correct me here if i am
wrong :-)).
So removing the check itself is not a problem since most of the
data after the split would be padded with zeros, which is okay
and should be left as application optimization if they wish to
pack small files in this range.
ErrShortData will be still returned in case if the size of data
is empty, or in case of streaming if the size == 0.
2016-04-29 20:38:45 -07:00
0b630aea27
use bytes.Equal rather than bytes.Compare
2016-04-29 14:12:03 -07:00
4048a541c8
Optimized encoding & decoding goroutines number
...
hardware: E5-2630 v2 (Intel x86-64 with ssse3)
software: linux, go1.6, GOMAXPROCS=2
Performances before after change
BenchmarkEncode10x2x10000-2 2884.95 MB/s 2837.93 MB/s 0.98x
BenchmarkEncode100x20x10000-2 593.93 MB/s 577.17 MB/s 0.97x
BenchmarkEncode17x3x1M-2 2903.74 MB/s 5197.99 MB/s 1.80x
BenchmarkEncode10x4x16M-2 1992.13 MB/s 3689.69 MB/s 1.85x
BenchmarkEncode5x2x1M-2 2883.78 MB/s 7506.19 MB/s 2.60x
BenchmarkEncode10x2x1M-2 3205.63 MB/s 7848.12 MB/s 2.45x
BenchmarkEncode10x4x1M-2 2218.35 MB/s 3998.35 MB/s 1.80x
BenchmarkEncode50x20x1M-2 579.24 MB/s 641.08 MB/s 1.11x
BenchmarkEncode17x3x16M-2 2652.36 MB/s 4775.41 MB/s 1.80x
BenchmarkVerify10x2x10000-2 1327.27 MB/s 1837.41 MB/s 1.38x
BenchmarkVerify50x5x50000-2 1481.89 MB/s 2684.57 MB/s 1.81x
BenchmarkVerify10x2x1M-2 1553.91 MB/s 5704.71 MB/s 3.67x
BenchmarkVerify5x2x1M-2 939.90 MB/s 4949.30 MB/s 5.26x
BenchmarkVerify10x4x1M-2 956.89 MB/s 3191.01 MB/s 3.33x
BenchmarkVerify50x20x1M-2 490.49 MB/s 823.46 MB/s 1.68x
BenchmarkVerify10x4x16M-2 1078.03 MB/s 3196.97 MB/s 2.97x
BenchmarkStreamEncode10x2x10000-2 2.40 MB/s 12.10 MB/s 5.04x
BenchmarkStreamEncode100x20x10000-2 6.72 MB/s 10.72 MB/s 1.60x
BenchmarkStreamEncode17x3x1M-2 390.75 MB/s 845.08 MB/s 2.16x
BenchmarkStreamEncode10x4x16M-2 1175.93 MB/s 1803.71 MB/s 1.53x
BenchmarkStreamEncode5x2x1M-2 207.85 MB/s 790.02 MB/s 3.80x
BenchmarkStreamEncode10x2x1M-2 296.77 MB/s 872.41 MB/s 2.94x
BenchmarkStreamEncode10x4x1M-2 264.43 MB/s 699.25 MB/s 2.64x
BenchmarkStreamEncode50x20x1M-2 284.93 MB/s 414.65 MB/s 1.46x
BenchmarkStreamEncode17x3x16M-2 1439.13 MB/s 1933.42 MB/s 1.34x
BenchmarkStreamVerify10x2x10000-2 2.33 MB/s 12.07 MB/s 5.18x
BenchmarkStreamVerify50x5x50000-2 86.53 MB/s 136.02 MB/s 1.57x
BenchmarkStreamVerify10x2x1M-2 315.65 MB/s 909.44 MB/s 2.88x
BenchmarkStreamVerify5x2x1M-2 180.45 MB/s 772.42 MB/s 4.28x
BenchmarkStreamVerify10x4x1M-2 310.35 MB/s 779.26 MB/s 2.51x
BenchmarkStreamVerify50x20x1M-2 547.23 MB/s 773.74 MB/s 1.41x
BenchmarkStreamVerify10x4x16M-2 4128.01 MB/s 6606.43 MB/s 1.60x
2016-04-12 15:41:22 +08:00
180472d98f
Make documentation conform to go vet.
2015-11-03 12:09:36 +01:00
295bf27a3d
fix Split panic
2015-08-08 16:38:55 -04:00
0bd572bc5b
tweak Split/Join functions
2015-08-08 13:51:12 -04:00
64b705bbf6
fully test Reconstruct function
...
Well, I can't figure out how to trigger the Invert error.
It may not be possible; need more domain knowledge to be sure.
2015-08-08 13:50:18 -04:00
cf985d4451
remove unreachable checkShards case
...
this case would be caught by shardSize anyway
2015-08-08 13:50:18 -04:00
5784cfa7ff
remove impossible errors
2015-08-06 22:46:27 -04:00
8ebf356efb
The number of data shards must be below 257. Check that and update documentation.
2015-06-23 13:39:57 +02:00
5c2ef3ae72
Always check/return errors.
2015-06-23 12:16:26 +02:00
7381e0b7b5
- Only run multiple goroutines if size is bigger than splitsize.
...
- Update docs
2015-06-23 11:18:29 +02:00
83703c37ac
Add package documentation and clarify interface docs.
2015-06-22 15:12:05 +02:00
5aa37c3492
Add AMD64 SSE3 Galois multiplication. Approximately 5-10x faster.
...
BenchmarkEncode10x2x10000 333.31 5827.17 17.48x
BenchmarkEncode10x2x10000-2 431.20 2802.53 6.50x
BenchmarkEncode10x2x10000-4 553.98 2432.95 4.39x
BenchmarkEncode10x2x10000-8 585.79 3469.61 5.92x
BenchmarkEncode100x20x10000 32.59 583.40 17.90x
BenchmarkEncode100x20x10000-2 59.52 726.70 12.21x
BenchmarkEncode100x20x10000-4 108.04 1363.25 12.62x
BenchmarkEncode100x20x10000-8 113.76 1274.62 11.20x
BenchmarkEncode17x3x1M 215.28 3141.85 14.59x
BenchmarkEncode17x3x1M-2 398.76 3650.12 9.15x
BenchmarkEncode17x3x1M-4 655.32 6071.11 9.26x
BenchmarkEncode17x3x1M-8 832.16 6616.47 7.95x
BenchmarkEncode10x4x16M 154.48 1357.30 8.79x
BenchmarkEncode10x4x16M-2 295.62 2377.92 8.04x
BenchmarkEncode10x4x16M-4 529.89 3519.49 6.64x
BenchmarkEncode10x4x16M-8 632.11 4521.90 7.15x
BenchmarkEncode5x2x1M 327.87 4879.09 14.88x
BenchmarkEncode5x2x1M-2 576.11 2599.20 4.51x
BenchmarkEncode5x2x1M-4 1043.65 3559.12 3.41x
BenchmarkEncode5x2x1M-8 1227.77 4255.34 3.47x
BenchmarkEncode10x2x1M 321.24 4574.68 14.24x
BenchmarkEncode10x2x1M-2 587.73 3100.28 5.28x
BenchmarkEncode10x2x1M-4 1101.96 4770.32 4.33x
BenchmarkEncode10x2x1M-8 1217.08 5812.17 4.78x
BenchmarkEncode10x4x1M 155.34 2037.27 13.11x
BenchmarkEncode10x4x1M-2 298.38 2470.97 8.28x
BenchmarkEncode10x4x1M-4 548.67 3603.15 6.57x
BenchmarkEncode10x4x1M-8 625.23 4827.42 7.72x
BenchmarkEncode50x20x1M 31.37 347.65 11.08x
BenchmarkEncode50x20x1M-2 59.81 713.28 11.93x
BenchmarkEncode50x20x1M-4 105.34 1175.47 11.16x
BenchmarkEncode50x20x1M-8 123.84 1491.91 12.05x
BenchmarkEncode17x3x16M 209.55 1861.59 8.88x
BenchmarkEncode17x3x16M-2 394.19 3331.73 8.45x
BenchmarkEncode17x3x16M-4 643.30 4942.74 7.68x
BenchmarkEncode17x3x16M-8 839.64 6213.43 7.40x
2015-06-21 21:23:22 +02:00
17e9fa30f0
Add Join function for join data shards.
2015-06-21 13:25:12 +02:00
437e364842
Adjust splitsize:
...
benchmark old ns/op new ns/op delta
BenchmarkEncode10x2x10000-2 243613 229413 -5.83%
BenchmarkEncode100x20x10000-2 23041318 19311104 -16.19%
BenchmarkEncode17x3x1M-2 54469780 49602836 -8.94%
BenchmarkEncode10x4x16M-2 674538600 647037000 -4.08%
Bigger sizes (1024) yeilds slower less speedup.
2015-06-20 20:32:52 +02:00
9f6744582c
Also refactor Verify as well as multithreaded options.
...
benchmark old MB/s new MB/s speedup
BenchmarkEncode10x2x10000 182.29 308.59 1.69x
BenchmarkEncode100x20x10000 14.41 30.29 2.10x
BenchmarkEncode17x3x1M 38.52 196.43 5.10x
BenchmarkEncode10x4x16M 23.78 148.58 6.25x
2015-06-20 20:00:25 +02:00
50a83296f4
Restructure to make one of the galois multiplication parts constant for the main loop.
2015-06-20 18:46:06 +02:00
921adcb5d5
Use range to avoid one bound check per galMultiply:
...
benchmark old MB/s new MB/s speedup
BenchmarkVerify10x2x10000-2 235.24 253.36 1.08x
BenchmarkVerify50x5x50000-2 76.78 94.87 1.24x
BenchmarkVerify10x2x1M-2 180.90 209.73 1.16x
BenchmarkVerify5x2x1M-2 173.22 202.89 1.17x
BenchmarkVerify10x4x1M-2 71.51 118.20 1.65x
BenchmarkVerify50x20x1M-2 11.27 12.84 1.14x
BenchmarkVerify10x4x16M-2 44.74 50.07 1.12x
2015-06-20 14:51:11 +02:00
419c6cc9e9
Add Splitter to help split data into shards.
2015-06-20 11:27:03 +02:00
c5de03551c
Minor adjustments for golint.
2015-06-20 10:11:33 +02:00
fdb7664a06
Hide internal matrix errors and document the used ones.
2015-06-20 10:08:59 +02:00
481b39ba1c
Use a mutex to check if other goroutines have detected a difference. This avoid rare "close of closed channel" panic.
2015-06-19 19:15:31 +02:00
e3aca6cd9d
Shorten the variable names and make an encoder interface, so it isn't possible to create it without calling New.
2015-06-19 18:54:58 +02:00
8296e44d17
Update documentation, and test if all shards are zero-sized.
2015-06-19 18:21:38 +02:00
755a781502
Remove unused types.
2015-06-19 17:27:22 +02:00
2b4171b9e6
Initial version
2015-06-19 16:31:24 +02:00
Peter C