Klaus Post
abb309aca7
Fix stream allocations ( #129 )
...
Numbers speak for themselves:
```
benchmark old ns/op new ns/op delta
BenchmarkStreamEncode10x2x10000-32 4792420 7937 -99.83%
BenchmarkStreamEncode100x20x10000-32 38424066 473285 -98.77%
BenchmarkStreamEncode17x3x1M-32 8195036 1482191 -81.91%
BenchmarkStreamEncode10x4x16M-32 21356715 18051773 -15.47%
BenchmarkStreamEncode5x2x1M-32 3295827 412301 -87.49%
BenchmarkStreamEncode10x2x1M-32 5249011 798828 -84.78%
BenchmarkStreamEncode10x4x1M-32 6392974 904818 -85.85%
BenchmarkStreamEncode50x20x1M-32 29083474 7199282 -75.25%
BenchmarkStreamEncode17x3x16M-32 32451850 28036421 -13.61%
BenchmarkStreamVerify10x2x10000-32 4858416 12988 -99.73%
BenchmarkStreamVerify50x5x50000-32 17047361 377003 -97.79%
BenchmarkStreamVerify10x2x1M-32 4869964 887214 -81.78%
BenchmarkStreamVerify5x2x1M-32 3282999 591669 -81.98%
BenchmarkStreamVerify10x4x1M-32 5824392 1230888 -78.87%
BenchmarkStreamVerify50x20x1M-32 27301648 6204613 -77.27%
BenchmarkStreamVerify10x4x16M-32 8508963 18845695 +121.48%
benchmark old MB/s new MB/s speedup
BenchmarkStreamEncode10x2x10000-32 20.87 12599.82 603.73x
BenchmarkStreamEncode100x20x10000-32 26.03 2112.89 81.17x
BenchmarkStreamEncode17x3x1M-32 2175.19 12026.65 5.53x
BenchmarkStreamEncode10x4x16M-32 7855.71 9293.94 1.18x
BenchmarkStreamEncode5x2x1M-32 1590.76 12716.14 7.99x
BenchmarkStreamEncode10x2x1M-32 1997.66 13126.43 6.57x
BenchmarkStreamEncode10x4x1M-32 1640.20 11588.81 7.07x
BenchmarkStreamEncode50x20x1M-32 1802.70 7282.50 4.04x
BenchmarkStreamEncode17x3x16M-32 8788.80 10172.93 1.16x
BenchmarkStreamVerify10x2x10000-32 20.58 7699.20 374.11x
BenchmarkStreamVerify50x5x50000-32 293.30 13262.49 45.22x
BenchmarkStreamVerify10x2x1M-32 2153.15 11818.75 5.49x
BenchmarkStreamVerify5x2x1M-32 1596.98 8861.17 5.55x
BenchmarkStreamVerify10x4x1M-32 1800.32 8518.86 4.73x
BenchmarkStreamVerify50x20x1M-32 1920.35 8449.97 4.40x
BenchmarkStreamVerify10x4x16M-32 19717.11 8902.41 0.45x
```
2020-05-05 16:35:35 +02:00
Christian Muehlhaeuser
4681100338
Removed unused struct members ( #106 )
...
creads & cwrites both seem to be unused.
2019-09-27 16:31:11 -07:00
Klaus Post
09979cdf93
Start documentation with method name.
...
Replaces #92
2019-02-15 15:31:43 +01:00
Frank Wessels
0de37d7697
Add ReconstructData interface method ( #57 )
...
* Add ReconstructData interface method to allow reconstruction of any missing data shards
* Add support for just reconstructing data shards only to SteamEncoder.Reconstruct()
2017-07-20 12:15:46 +02:00
Klaus Post
0dd0a0e50c
Fix error grammar
...
Fixes #56
2017-07-16 17:00:58 +02:00
Klaus Post
5abf0ee302
Add options ( #46 )
...
* Add options
Make constants changeable as options.
The API remains backwards compatible.
* Update documentation.
* Fix line endings
* fmt
* fmt
* Use functions for parameters.
Much neater.
2017-02-19 11:13:22 +01:00
Harshavardhana
df175d2921
ErrShortData shouldn't be returned for data less than dataShards.
...
The reasoning behind this is that if we have a data block number
of 10, and parity of 10. Restricting input such that files of
size < 10Bytes should be errored out doesn't seem like the right
approach.
Most erasure subsystems will have static data and parity blocks,
in such case erroring out is not correct since reedsolomon itself
doesn't provide this limitation (please correct me here if i am
wrong :-)).
So removing the check itself is not a problem since most of the
data after the split would be padded with zeros, which is okay
and should be left as application optimization if they wish to
pack small files in this range.
ErrShortData will be still returned in case if the size of data
is empty, or in case of streaming if the size == 0.
2016-04-29 20:38:45 -07:00
klauspost
e125c8bc8c
Fix comment.
2015-11-03 12:36:28 +01:00
klauspost
94aef4a146
Add stream order information.
2015-11-03 12:10:37 +01:00
klauspost
5dedbfc3ff
Update streaming documentation.
2015-10-27 15:59:47 +01:00
Klaus Post
4c33f7e599
Remove duplicated package documentation.
2015-10-27 15:39:19 +01:00
klauspost
64a2a9b848
Add performance note.
2015-10-27 14:54:40 +01:00
klauspost
7d91d8d953
Create separate read/write errors.
2015-10-27 11:11:50 +01:00
klauspost
ff9a5aff03
Remove StreamError for shard size mismatches
...
Since the result is pretty random (especially on concurrent reads),
we just return the plan ErrShardSize.
2015-10-27 10:54:08 +01:00
klauspost
e78a382960
Add specific error for streams, to help identify the faulty stream.
2015-10-27 10:36:29 +01:00
klauspost
0500314cc5
Add concurrent reads/writes to streaming interface.
2015-10-26 15:59:02 +01:00
klauspost
8940ce33cc
Return ErrShardNoData if there is no data in stream.
2015-10-26 14:31:10 +01:00
Klaus Post
32af654e33
Trim output shards to be equal size to input shards on last read.
2015-10-24 12:42:04 +02:00
klauspost
3fada41987
Add streaming API
2015-10-23 16:14:55 +02:00