b26f308191
These are variants on S3. S3 uses the "pure" S3 interface, e.g. using Multi-Part-Upload. The "plus" variant enables EMC-extensions in the aws4c library. This allows the N:N case to use "append", in the case where "transfer_size" != "block_size" for IOR. In pure S3, the N:N case will fail, because the EMC-extensions won't be enabled, and appending (which attempts to use the EMC byte-range tricks to do this) will throw an error. In the S3_EMC alg, N:1 uses EMCs other byte-range tricks to write different parts of an N:1 file, and also uses append to write the parts of an N:N file. Preliminary tests show these EMC extensions look to improve BW by ~20%. I put all three algs in aiori-S3.c, because it seemed some code was getting reused. Not sure if that's still going to make sense after the TBD, below. TBD: Recently realized that the "pure' S3 shouldn't be trying to use appends for anything. In the N:N case, it should just use MPU, within each file. Then, there's no need for S3_plus. We just have S3, which does MPU for all writes where transfer_size != block_size, and uses (standard) byte-range reads for reading. Then S3_EMC uses "appends for N:N writes, and byte-range writes for N:1 writes. This separates the code for the two algs a little more, but we might still want them in the same file. |
||
|---|---|---|
| config | ||
| contrib | ||
| doc | ||
| scripts | ||
| src | ||
| testing | ||
| .gitignore | ||
| COPYRIGHT | ||
| ChangeLog | ||
| META | ||
| Makefile.am | ||
| README | ||
| bootstrap | ||
| configure.ac | ||
| lanl_aws4c.tgz | ||
README
Building -------- 0. If "configure" is missing from the top level directory, you probably retrieved this code directly from the repository. Run "./bootstrap". If your versions of the autotools are not new enough to run this script, download and official tarball in which the configure script is already provided. 1. Run "./configure" See "./configure --help" for configuration options. 2. Run "make" 3. Optionally, run "make install". The installation prefix can be changed as an option to the "configure" script.