Commit Graph

161 Commits (6bb6130ff6a76a904c1841707d65603aec9cc288)

Author SHA1 Message Date
Klaus Post 6bb6130ff6 Add laste new feature to doc. 2017-10-01 14:06:06 +02:00
Klaus Post 61c22eab55 Cauchy Matrix option (#70)
* Experimental Cauchy Matrix

Experimental support for Cauchy style matrix

http://web.eecs.utk.edu/~plank/plank/papers/CS-05-569.pdf

All matrices appear reversible.

* Remove Go 1.5 and 1.6 from CI tests.

* Fix comment.

* Increase max number of goroutines+docs.
2017-10-01 14:02:11 +02:00
David Reiss ddcafc661e Allow reconstructing into pre-allocated memory. (#66)
This changes the interface of Reconstruct and ReconstructData to accept
slices of zero length but sufficient capacity for shards to reconstruct,
and reslices them instead of allocating new memory.
2017-09-20 21:08:24 +02:00
Klaus Post 87ba8262ab Add Go 1.9 to Travis. 2017-08-26 11:54:10 +02:00
Klaus Post 985e396eec Asmfmt. 2017-08-26 11:51:49 +02:00
Klaus Post c71640765a Update docs before release, when #62 is ready. (#63)
* Update docs before release, when #62 is ready.

* Update README.md
2017-08-26 11:48:42 +02:00
Frank Wessels 7b88f42e61 Add NEON support for ARM64 (#62)
* Add support for arm64 using NEON instructions

Specifically using the PMULL/PMULL2 polynomial multiplication instructions followed by a reduction step (actually two steps).

* Add ARM performance numbers

* Formatting for performance table

* Refactoring of NEON version and 256-bit wide version

* Expand test slice beyond 32 (for AVX2 and NEON) and test galMulSliceXor explicitly.

* Fix ARM code with missing function.

* Fix missing newline
2017-08-26 11:47:42 +02:00
chenzhongtao d78bf472d8 add Update parity function (#60)
Add Update parity function
2017-08-20 11:42:39 +02:00
Klaus Post dc6af2dce5 Minor cleanup (#61)
* Remove some benchmarks
* Format tables a bit.
* Doc cleanup
2017-08-13 22:38:27 +02:00
Andreas Auernhammer 48a4fd05f1 fix unnecessary memory alloc in Split (#59)
Split divided the data into `DataShards` blocks and allocates all parity blocks.

This change adds a check whether the capacity of data is large enough to hold all
data and parity blocks. It only allocates parity blocks if necessary.
2017-07-22 16:16:58 +02:00
Klaus Post 82ee2d9869 Update README.md 2017-07-20 12:24:02 +02:00
Frank Wessels 0de37d7697 Add ReconstructData interface method (#57)
* Add ReconstructData interface method to allow reconstruction of any missing data shards
* Add support for just reconstructing data shards only to SteamEncoder.Reconstruct()
2017-07-20 12:15:46 +02:00
Klaus Post 0dd0a0e50c Fix error grammar
Fixes #56
2017-07-16 17:00:58 +02:00
Fred Akalin 18d548df63 Add support for PAR1 (#55)
PAR1 is a file format which uses a Reed-Solomon code similar
to the current one, except it uses a different (flawed) coding
matrix.

Add support for it via a WithPAR1Matrix option, so that this code
can be used to encode/decode PAR1 files. Also add the option to
existing tests, and add a test demonstrating the flaw in PAR1's
coding matrix.

Also fix an mistakenly inverted test in testOpts().

Incidentally, PAR1 is obsoleted by PAR2, which uses GF(2^16)
and tries to fix the flaw in the coding matrix; however, PAR2's
coding matrix is still flawed! The real solution is to build the
coding matrix like in this repository.

PAR1 spec:
http://parchive.sourceforge.net/docs/specifications/parity-volume-spec-1.0/article-spec.html

Paper describing the (flawed) Reed-Solomon code used by PAR1:
http://web.eecs.utk.edu/~plank/plank/papers/CS-96-332.html
2017-06-20 20:24:57 +02:00
Fred Akalin 87c4e5ae75 Allow 256 total shards (#54)
* Allow 256 total shards
2017-06-19 11:26:52 +02:00
timchenxiaoyu d4658f22be fix example error (#53) 2017-06-06 22:26:01 +02:00
Klaus Post dde6ad55c5 Set correct field in WithMinSplitSize
Fixes #51
2017-05-28 12:38:06 +02:00
Klaus Post 5abf0ee302 Add options (#46)
* Add options

Make constants changeable as options.

The API remains backwards compatible.

* Update documentation.

* Fix line endings

* fmt

* fmt

* Use functions for parameters.

Much neater.
2017-02-19 11:13:22 +01:00
Klaus Post c056598956 Merge pull request #39 from jesselucas/patch-1
Update README.md to fix small typos.
2017-01-05 16:18:16 +01:00
Jesse Lucas ff2f89b6ca Update README.md to fix small typos. 2017-01-05 00:16:24 -05:00
Klaus Post d0a56f72c0 Update README.md 2016-10-28 09:13:20 +02:00
Klaus Post 9998b4cb21 Update README.md 2016-10-28 09:00:26 +02:00
Peter C c54154da9e Add Inverse Matrix caching in a Thread-Safe Lookup Tree (#36)
* Add matrix inversion caching
* Benchmark and Parallel Benchmark tests for Reconstruct
2016-09-12 21:31:07 +02:00
Klaus Post fac1884d47 Merge pull request #34 from muesli/master
Make Join return an error if a reconstruction is required first
2016-08-22 14:13:59 +02:00
Christian Muehlhaeuser b1c8b4b073 Make Join return an error if a reconstruction is required first
If one or more required data shards are still nil and we can't correctly join
them before a reconstruction, return ErrReconstructRequired.
2016-08-05 19:23:08 +02:00
Klaus Post 9b772b54b3 Merge pull request #30 from hackintoshrao/matrix-test
Tests: Coverage and enhancement for matrix_test.go
2016-08-01 19:20:09 +02:00
Klaus Post 5eab34543e Merge pull request #31 from klauspost/use-gofmt-for-tests
Use gofmt instead of goimports for format teting
2016-08-01 19:19:22 +02:00
Klaus Post 6978874dc1 Use gofmt instead of goimports for format teting. 2016-08-01 18:15:38 +02:00
Karthic Rao f89ef0c3a3 Coverage and enhancement for matrix_test.go 2016-07-26 15:41:13 +05:30
Frank 467733eb9c Add generated byte assembler using asm2plan9s
Add recompilable assembler using asm2plan9s
2016-07-06 21:06:00 +02:00
Klaus Post cebd11778b Merge pull request #26 from fwessels/removeJMP
Removed unnecessary JMP instruction
2016-07-06 11:08:32 +02:00
frankw d4000061f2 Removed unnecessary JMP instruction 2016-07-06 09:39:02 +02:00
Klaus Post dd5af9e1ff Merge pull request #25 from gyuho/slice_allocation
matrix.go: preallocate slice for String() method
2016-07-05 16:08:46 +02:00
Gyu-Ho Lee 7a64c70cbc matrix.go: preallocate slice for String() method
To avoid slice growth with append operations.
2016-06-04 22:12:17 -07:00
Klaus Post 2fde63d288 Merge pull request #24 from harshavardhana/parity
Add checks for data and parity to not exceed 255 shards in total.
2016-06-04 08:05:20 +02:00
Klaus Post b118551f15 Merge pull request #23 from harshavardhana/appveyor
Submit a new appveyor CI config.
2016-06-04 08:04:55 +02:00
Harshavardhana ba30981088 Add checks for data and parity to not exceed 255 shards in total.
Fixes #16
2016-06-03 01:31:01 -07:00
Harshavardhana 25fa063ea5 Submit a new appveyor CI config.
Fixes #21
2016-06-03 00:57:56 -07:00
Klaus Post 4fadad8564 Update reedsolomon.go
Fix comment
2016-05-01 12:00:51 +02:00
Klaus Post ed06f926b9 Merge pull request #20 from harshavardhana/fix
ErrShortData shouldn't be returned for data less than dataShards.
2016-05-01 11:58:47 +02:00
Klaus Post d61a5ff262 Merge pull request #19 from harshavardhana/simple
use bytes.Equal rather than bytes.Compare
2016-05-01 11:57:20 +02:00
Harshavardhana df175d2921 ErrShortData shouldn't be returned for data less than dataShards.
The reasoning behind this is that if we have a data block number
of 10, and parity of 10.  Restricting input such that files of
size < 10Bytes should be errored out doesn't seem like the right
approach.

Most erasure subsystems will have static data and parity blocks,
in such case erroring out is not correct since reedsolomon itself
doesn't provide this limitation (please correct me here if i am
wrong :-)).

So removing the check itself is not a problem since most of the
data after the split would be padded with zeros, which is okay
and should be left as application optimization if they wish to
pack small files in this range.

ErrShortData will be still returned in case if the size of data
is empty, or in case of streaming if the size == 0.
2016-04-29 20:38:45 -07:00
Harshavardhana 0b630aea27 use bytes.Equal rather than bytes.Compare 2016-04-29 14:12:03 -07:00
Klaus Post 55f0000302 Merge pull request #18 from xiaost/optimized-goroutines-number
Optimized encoding & decoding goroutines number
2016-04-12 10:54:07 +02:00
xiaost 4048a541c8 Optimized encoding & decoding goroutines number
hardware: E5-2630 v2  (Intel x86-64 with ssse3)
software: linux, go1.6, GOMAXPROCS=2

Performances                          before          after         change

BenchmarkEncode10x2x10000-2           2884.95 MB/s    2837.93 MB/s  0.98x
BenchmarkEncode100x20x10000-2          593.93 MB/s     577.17 MB/s  0.97x
BenchmarkEncode17x3x1M-2              2903.74 MB/s    5197.99 MB/s  1.80x
BenchmarkEncode10x4x16M-2             1992.13 MB/s    3689.69 MB/s  1.85x
BenchmarkEncode5x2x1M-2               2883.78 MB/s    7506.19 MB/s  2.60x
BenchmarkEncode10x2x1M-2              3205.63 MB/s    7848.12 MB/s  2.45x
BenchmarkEncode10x4x1M-2              2218.35 MB/s    3998.35 MB/s  1.80x
BenchmarkEncode50x20x1M-2              579.24 MB/s     641.08 MB/s  1.11x
BenchmarkEncode17x3x16M-2             2652.36 MB/s    4775.41 MB/s  1.80x
BenchmarkVerify10x2x10000-2           1327.27 MB/s    1837.41 MB/s  1.38x
BenchmarkVerify50x5x50000-2           1481.89 MB/s    2684.57 MB/s  1.81x
BenchmarkVerify10x2x1M-2              1553.91 MB/s    5704.71 MB/s  3.67x
BenchmarkVerify5x2x1M-2                939.90 MB/s    4949.30 MB/s  5.26x
BenchmarkVerify10x4x1M-2               956.89 MB/s    3191.01 MB/s  3.33x
BenchmarkVerify50x20x1M-2              490.49 MB/s     823.46 MB/s  1.68x
BenchmarkVerify10x4x16M-2             1078.03 MB/s    3196.97 MB/s  2.97x
BenchmarkStreamEncode10x2x10000-2        2.40 MB/s      12.10 MB/s  5.04x
BenchmarkStreamEncode100x20x10000-2      6.72 MB/s      10.72 MB/s  1.60x
BenchmarkStreamEncode17x3x1M-2         390.75 MB/s     845.08 MB/s  2.16x
BenchmarkStreamEncode10x4x16M-2       1175.93 MB/s    1803.71 MB/s  1.53x
BenchmarkStreamEncode5x2x1M-2          207.85 MB/s     790.02 MB/s  3.80x
BenchmarkStreamEncode10x2x1M-2         296.77 MB/s     872.41 MB/s  2.94x
BenchmarkStreamEncode10x4x1M-2         264.43 MB/s     699.25 MB/s  2.64x
BenchmarkStreamEncode50x20x1M-2        284.93 MB/s     414.65 MB/s  1.46x
BenchmarkStreamEncode17x3x16M-2       1439.13 MB/s    1933.42 MB/s  1.34x
BenchmarkStreamVerify10x2x10000-2        2.33 MB/s      12.07 MB/s  5.18x
BenchmarkStreamVerify50x5x50000-2       86.53 MB/s     136.02 MB/s  1.57x
BenchmarkStreamVerify10x2x1M-2         315.65 MB/s     909.44 MB/s  2.88x
BenchmarkStreamVerify5x2x1M-2          180.45 MB/s     772.42 MB/s  4.28x
BenchmarkStreamVerify10x4x1M-2         310.35 MB/s     779.26 MB/s  2.51x
BenchmarkStreamVerify50x20x1M-2        547.23 MB/s     773.74 MB/s  1.41x
BenchmarkStreamVerify10x4x16M-2       4128.01 MB/s    6606.43 MB/s  1.60x
2016-04-12 15:41:22 +08:00
Klaus Post ff5a6511bc Merge pull request #17 from xiaost/master
Tests: backport go1.6 rand.Read for speedup tests
2016-04-07 13:02:14 +02:00
xiaost 9f0bea8a29 Tests: backport go1.6 rand.Read for speedup tests 2016-04-07 18:34:47 +08:00
Klaus Post 58c5128227 Update .travis.yml
Remove golint
2016-03-21 18:45:58 +01:00
Klaus Post 458f5ce468 Update .travis.yml
Add OSX test. Don't use sudo
2016-02-19 15:32:27 +01:00
Klaus Post 56791bf235 Update .travis.yml
Add Go 1.6 to CI test.
2016-02-19 15:22:04 +01:00