Commit Graph

218 Commits (01b307ec91a7a7587226724141df961f3a9b8d26)

Author SHA1 Message Date
Klaus Post 7d9453e171
Update README.md 2018-05-04 15:02:00 +02:00
Klaus Post 0b30fa71cc
Merge pull request #75 from ernado/patch-1
Fix typo in README.md
2017-12-19 14:34:37 +01:00
Aleksandr Razumov 19a926a71b
Fix typo in README.md 2017-12-15 19:01:33 +03:00
Klaus Post 6db5e38e85
Merge pull request #74 from klauspost/align-blocks
Split blocks into size divisible by 16, add WithAutoGoroutines
2017-12-10 21:00:05 +01:00
Klaus Post f5e73dcfe2 Split blocks into size divisible by 16
Older systems (typically without AVX2) are more sensitive to misaligned load+stores.

Add parameter to automatically set the number of goroutines.

name                  old time/op    new time/op    delta
Encode10x2x10000-8      18.4µs ± 1%    16.1µs ± 1%  -12.43%    (p=0.000 n=9+9)
Encode100x20x10000-8     692µs ± 1%     608µs ± 1%  -12.10%  (p=0.000 n=10+10)
Encode17x3x1M-8         1.78ms ± 5%    1.49ms ± 1%  -16.63%  (p=0.000 n=10+10)
Encode10x4x16M-8        21.5ms ± 5%    19.6ms ± 4%   -8.74%   (p=0.000 n=10+9)
Encode5x2x1M-8           343µs ± 2%     267µs ± 2%  -22.22%   (p=0.000 n=9+10)
Encode10x2x1M-8          858µs ± 5%     701µs ± 5%  -18.34%  (p=0.000 n=10+10)
Encode10x4x1M-8         1.34ms ± 1%    1.16ms ± 1%  -13.19%    (p=0.000 n=9+9)
Encode50x20x1M-8        30.3ms ± 4%    25.0ms ± 2%  -17.51%   (p=0.000 n=10+8)
Encode17x3x16M-8        26.9ms ± 1%    24.5ms ± 4%   -9.13%   (p=0.000 n=8+10)

name                  old speed      new speed      delta
Encode10x2x10000-8    5.45GB/s ± 1%  6.22GB/s ± 1%  +14.20%    (p=0.000 n=9+9)
Encode100x20x10000-8  1.44GB/s ± 1%  1.64GB/s ± 1%  +13.77%  (p=0.000 n=10+10)
Encode17x3x1M-8       10.0GB/s ± 5%  12.0GB/s ± 1%  +19.88%  (p=0.000 n=10+10)
Encode10x4x16M-8      7.81GB/s ± 5%  8.56GB/s ± 5%   +9.58%   (p=0.000 n=10+9)
Encode5x2x1M-8        15.3GB/s ± 2%  19.6GB/s ± 2%  +28.57%   (p=0.000 n=9+10)
Encode10x2x1M-8       12.2GB/s ± 5%  15.0GB/s ± 5%  +22.45%  (p=0.000 n=10+10)
Encode10x4x1M-8       7.84GB/s ± 1%  9.03GB/s ± 1%  +15.19%    (p=0.000 n=9+9)
Encode50x20x1M-8      1.73GB/s ± 4%  2.09GB/s ± 4%  +20.59%   (p=0.000 n=10+9)
Encode17x3x16M-8      10.6GB/s ± 1%  11.7GB/s ± 4%  +10.12%   (p=0.000 n=8+10)
2017-11-18 22:00:55 +01:00
Nick Heindl e52c150f96 Fix some small typos in README (#71) 2017-11-18 16:17:31 +01:00
Frank Wessels 3610933d2f Use AVX2 SIMD assembly instructions in favor of BYTE sequences. (#73)
* Use AVX2 SIMD assembly instructions in favor of BYTE sequences.
2017-11-18 16:17:10 +01:00
Klaus Post 6bb6130ff6 Add laste new feature to doc. 2017-10-01 14:06:06 +02:00
Klaus Post 61c22eab55 Cauchy Matrix option (#70)
* Experimental Cauchy Matrix

Experimental support for Cauchy style matrix

http://web.eecs.utk.edu/~plank/plank/papers/CS-05-569.pdf

All matrices appear reversible.

* Remove Go 1.5 and 1.6 from CI tests.

* Fix comment.

* Increase max number of goroutines+docs.
2017-10-01 14:02:11 +02:00
David Reiss ddcafc661e Allow reconstructing into pre-allocated memory. (#66)
This changes the interface of Reconstruct and ReconstructData to accept
slices of zero length but sufficient capacity for shards to reconstruct,
and reslices them instead of allocating new memory.
2017-09-20 21:08:24 +02:00
Klaus Post 87ba8262ab Add Go 1.9 to Travis. 2017-08-26 11:54:10 +02:00
Klaus Post 985e396eec Asmfmt. 2017-08-26 11:51:49 +02:00
Klaus Post c71640765a Update docs before release, when #62 is ready. (#63)
* Update docs before release, when #62 is ready.

* Update README.md
2017-08-26 11:48:42 +02:00
Frank Wessels 7b88f42e61 Add NEON support for ARM64 (#62)
* Add support for arm64 using NEON instructions

Specifically using the PMULL/PMULL2 polynomial multiplication instructions followed by a reduction step (actually two steps).

* Add ARM performance numbers

* Formatting for performance table

* Refactoring of NEON version and 256-bit wide version

* Expand test slice beyond 32 (for AVX2 and NEON) and test galMulSliceXor explicitly.

* Fix ARM code with missing function.

* Fix missing newline
2017-08-26 11:47:42 +02:00
chenzhongtao d78bf472d8 add Update parity function (#60)
Add Update parity function
2017-08-20 11:42:39 +02:00
Klaus Post dc6af2dce5 Minor cleanup (#61)
* Remove some benchmarks
* Format tables a bit.
* Doc cleanup
2017-08-13 22:38:27 +02:00
Andreas Auernhammer 48a4fd05f1 fix unnecessary memory alloc in Split (#59)
Split divided the data into `DataShards` blocks and allocates all parity blocks.

This change adds a check whether the capacity of data is large enough to hold all
data and parity blocks. It only allocates parity blocks if necessary.
2017-07-22 16:16:58 +02:00
Klaus Post 82ee2d9869 Update README.md 2017-07-20 12:24:02 +02:00
Frank Wessels 0de37d7697 Add ReconstructData interface method (#57)
* Add ReconstructData interface method to allow reconstruction of any missing data shards
* Add support for just reconstructing data shards only to SteamEncoder.Reconstruct()
2017-07-20 12:15:46 +02:00
Klaus Post 0dd0a0e50c Fix error grammar
Fixes #56
2017-07-16 17:00:58 +02:00
Fred Akalin 18d548df63 Add support for PAR1 (#55)
PAR1 is a file format which uses a Reed-Solomon code similar
to the current one, except it uses a different (flawed) coding
matrix.

Add support for it via a WithPAR1Matrix option, so that this code
can be used to encode/decode PAR1 files. Also add the option to
existing tests, and add a test demonstrating the flaw in PAR1's
coding matrix.

Also fix an mistakenly inverted test in testOpts().

Incidentally, PAR1 is obsoleted by PAR2, which uses GF(2^16)
and tries to fix the flaw in the coding matrix; however, PAR2's
coding matrix is still flawed! The real solution is to build the
coding matrix like in this repository.

PAR1 spec:
http://parchive.sourceforge.net/docs/specifications/parity-volume-spec-1.0/article-spec.html

Paper describing the (flawed) Reed-Solomon code used by PAR1:
http://web.eecs.utk.edu/~plank/plank/papers/CS-96-332.html
2017-06-20 20:24:57 +02:00
Fred Akalin 87c4e5ae75 Allow 256 total shards (#54)
* Allow 256 total shards
2017-06-19 11:26:52 +02:00
timchenxiaoyu d4658f22be fix example error (#53) 2017-06-06 22:26:01 +02:00
Klaus Post dde6ad55c5 Set correct field in WithMinSplitSize
Fixes #51
2017-05-28 12:38:06 +02:00
Klaus Post 5abf0ee302 Add options (#46)
* Add options

Make constants changeable as options.

The API remains backwards compatible.

* Update documentation.

* Fix line endings

* fmt

* fmt

* Use functions for parameters.

Much neater.
2017-02-19 11:13:22 +01:00
Klaus Post c056598956 Merge pull request #39 from jesselucas/patch-1
Update README.md to fix small typos.
2017-01-05 16:18:16 +01:00
Jesse Lucas ff2f89b6ca Update README.md to fix small typos. 2017-01-05 00:16:24 -05:00
Klaus Post d0a56f72c0 Update README.md 2016-10-28 09:13:20 +02:00
Klaus Post 9998b4cb21 Update README.md 2016-10-28 09:00:26 +02:00
Peter C c54154da9e Add Inverse Matrix caching in a Thread-Safe Lookup Tree (#36)
* Add matrix inversion caching
* Benchmark and Parallel Benchmark tests for Reconstruct
2016-09-12 21:31:07 +02:00
Klaus Post fac1884d47 Merge pull request #34 from muesli/master
Make Join return an error if a reconstruction is required first
2016-08-22 14:13:59 +02:00
Christian Muehlhaeuser b1c8b4b073 Make Join return an error if a reconstruction is required first
If one or more required data shards are still nil and we can't correctly join
them before a reconstruction, return ErrReconstructRequired.
2016-08-05 19:23:08 +02:00
Klaus Post 9b772b54b3 Merge pull request #30 from hackintoshrao/matrix-test
Tests: Coverage and enhancement for matrix_test.go
2016-08-01 19:20:09 +02:00
Klaus Post 5eab34543e Merge pull request #31 from klauspost/use-gofmt-for-tests
Use gofmt instead of goimports for format teting
2016-08-01 19:19:22 +02:00
Klaus Post 6978874dc1 Use gofmt instead of goimports for format teting. 2016-08-01 18:15:38 +02:00
Karthic Rao f89ef0c3a3 Coverage and enhancement for matrix_test.go 2016-07-26 15:41:13 +05:30
Frank 467733eb9c Add generated byte assembler using asm2plan9s
Add recompilable assembler using asm2plan9s
2016-07-06 21:06:00 +02:00
Klaus Post cebd11778b Merge pull request #26 from fwessels/removeJMP
Removed unnecessary JMP instruction
2016-07-06 11:08:32 +02:00
frankw d4000061f2 Removed unnecessary JMP instruction 2016-07-06 09:39:02 +02:00
Klaus Post dd5af9e1ff Merge pull request #25 from gyuho/slice_allocation
matrix.go: preallocate slice for String() method
2016-07-05 16:08:46 +02:00
Gyu-Ho Lee 7a64c70cbc matrix.go: preallocate slice for String() method
To avoid slice growth with append operations.
2016-06-04 22:12:17 -07:00
Klaus Post 2fde63d288 Merge pull request #24 from harshavardhana/parity
Add checks for data and parity to not exceed 255 shards in total.
2016-06-04 08:05:20 +02:00
Klaus Post b118551f15 Merge pull request #23 from harshavardhana/appveyor
Submit a new appveyor CI config.
2016-06-04 08:04:55 +02:00
Harshavardhana ba30981088 Add checks for data and parity to not exceed 255 shards in total.
Fixes #16
2016-06-03 01:31:01 -07:00
Harshavardhana 25fa063ea5 Submit a new appveyor CI config.
Fixes #21
2016-06-03 00:57:56 -07:00
Klaus Post 4fadad8564 Update reedsolomon.go
Fix comment
2016-05-01 12:00:51 +02:00
Klaus Post ed06f926b9 Merge pull request #20 from harshavardhana/fix
ErrShortData shouldn't be returned for data less than dataShards.
2016-05-01 11:58:47 +02:00
Klaus Post d61a5ff262 Merge pull request #19 from harshavardhana/simple
use bytes.Equal rather than bytes.Compare
2016-05-01 11:57:20 +02:00
Harshavardhana df175d2921 ErrShortData shouldn't be returned for data less than dataShards.
The reasoning behind this is that if we have a data block number
of 10, and parity of 10.  Restricting input such that files of
size < 10Bytes should be errored out doesn't seem like the right
approach.

Most erasure subsystems will have static data and parity blocks,
in such case erroring out is not correct since reedsolomon itself
doesn't provide this limitation (please correct me here if i am
wrong :-)).

So removing the check itself is not a problem since most of the
data after the split would be padded with zeros, which is okay
and should be left as application optimization if they wish to
pack small files in this range.

ErrShortData will be still returned in case if the size of data
is empty, or in case of streaming if the size == 0.
2016-04-29 20:38:45 -07:00
Harshavardhana 0b630aea27 use bytes.Equal rather than bytes.Compare 2016-04-29 14:12:03 -07:00