mdtest/src
Jeffrey Thornton Inman 4368cc2dc4 Fixed striding/segmentation in N:N and N:1 cases, for multi-part-upload.
All ranks locally capture and accumulate Etags for the parts they are
writing.  In the N:1 cases, these are ethen collected by rank 0, via
MPI_Gather.  This is effectively an organization matching the "segmented"
layout.  If data was written segmented, then rank0 assigns part-numbers to
with appropriate offsets to correspond to what would've been used by each
rank when writing a given etag.  If data was written strided, then etags
must also be accessed in strided order, to build the XML that will be sent.

TBD: Once the total volume of etag data exceeds the size of memory at rank
0, we'll need to impose a more-sophisticated technique.  One idea is to
thread the MPI comms differently from the libcurl comms, so that multiple
gathers can be staged incrementally, while sending a single stream of XML
data tot he servers.  For example, the libcurl write-function could
interact with the MPI prog to allow the appearance of a single stream of
data.
2014-12-02 09:10:32 -07:00
..
win Add header to set code preferences for vi and emacs. 2012-01-08 19:43:41 -08:00
Makefile.am S3 with Multi-Part Upload for N:1 is working. 2014-10-27 13:29:44 -06:00
aiori-HDF5.c Define H5_USER_16_API to enable HDF5 backwards compatibility 2013-07-18 15:01:15 -07:00
aiori-HDFS.c Fixed bug in aiori-HDFS.c where it was calling "free(fd);", which is bad. 2014-08-28 15:39:44 -06:00
aiori-MPIIO.c Initial commit of the new aiori-HDFS module. 2014-08-13 17:02:43 -06:00
aiori-NCMPI.c Add header to set code preferences for vi and emacs. 2012-01-08 19:43:41 -08:00
aiori-PLFS.c First-cut at PLFS support, integrating Brett K's code. 2014-09-18 11:20:37 -06:00
aiori-POSIX.c First-cut at PLFS support, integrating Brett K's code. 2014-09-18 11:20:37 -06:00
aiori-S3.c Fixed striding/segmentation in N:N and N:1 cases, for multi-part-upload. 2014-12-02 09:10:32 -07:00
aiori.h Algorithms 'S3', 'S3_plus', and 'S3_EMC' all available. 2014-10-29 16:04:30 -06:00
ior.c Fixed striding/segmentation in N:N and N:1 cases, for multi-part-upload. 2014-12-02 09:10:32 -07:00
ior.h Algorithms 'S3', 'S3_plus', and 'S3_EMC' all available. 2014-10-29 16:04:30 -06:00
iordef.h Initial commit of the new aiori-HDFS module. 2014-08-13 17:02:43 -06:00
parse_options.c Don't have IOR_RDWR in open-flags, if only opening for reading or writing. 2014-08-25 15:51:28 -06:00
parse_options.h Add header to set code preferences for vi and emacs. 2012-01-08 19:43:41 -08:00
utilities.c First-cut at PLFS support, integrating Brett K's code. 2014-09-18 11:20:37 -06:00
utilities.h First-cut at PLFS support, integrating Brett K's code. 2014-09-18 11:20:37 -06:00