For now, and compatibility, IOR options can still be set/internally accessed using the backends init_xfer_options.
This should be removed in the long run to strip away this dependency.
On systems where numTasks is not evenly divisible by 'tasksPerNode' we were
seeing some nodes reading multiple files while others read none after
reordering.
Commonly all nodes have the same number of tasks but there is nothing
requiring that to be the case. Imagine having 64 tasks running against 4
nodes which can run 20 tasks each. Here you get three groups of 20 and one
group of 4. On this sytem nodes running in the group of 4 were previously
getting tasksPerNode of 4 which meant they reordered tasks differently than
the nodes which got tasksPerNode of 20.
The key to fixing this is ensuring that every node reorders tasks the same
way, which means ensuring they all use the same input values. Obviously on
systems where the number of tasks per node is inconsistent the reordering will
also be inconsistent (some tasks may end up on the same node, or not as far
separated as desired, etc.) but at least this way you'll always end up with a
1:1 reordering.
- Renamed nodes/nodeCount to numNodes
- Renamed tasksPerNode to numTasksOnNode0
- Ensured that numTasksOnNode0 will always have the same value regardless of
which node you're on
- Removed inconsistently used globals numTasksWorld and tasksPerNode and
replaced with per-test params equivalents
- Added utility functions for setting these values:
- numNodes -> GetNumNodes
- numTasks -> GetNumTasks
- numTasksOnNode0 -> GetNumNodesOnTask0
- Improved MPI_VERSION < 3 logic for GetNumNodes so it works when numTasks is
not evenly divisible by numTasksOnNode0
- Left 'nodes' and 'tasksPerNode' in output alone to not break compatibility
- Allowed command-line params to override numTasks, numNodes, and
numTasksOnNode0 but default to using the MPI-calculated values
The parser now supports concurrent parsing of all plugin options.
Moved HDF5 collective_md option into the backend as an example.
Example: ./src/ior -a dummy --dummy.delay-xfer=50000
Context: write and read results from the same iteration
use the same length value in Bytes. When stonewalling is
used the size variates depending on the performance of
the access. This leads to wrong max bandwidths reported
for writes as shown in the following example:
write 10052 ...
read 9910 ...
write 10022 ...
read 9880 ...
write 10052 ...
read 9894 ...
Max Write: 9371.43 MiB/sec (9826.66 MB/sec)
Max Read: 9910.48 MiB/sec (10391.89 MB/sec)
This patch makes IOR separate variables used for read
and write tests.
Context: IOR initializes all available backends. If one
backend fails to initialize IOR cannot be used.
This patch makes IOR initialize only the backends
which will be used. The initialization is done after
that the parameters are checked so that the help message
can still be dispayed is something goes wrong.
inform aiori interface about RADOS backend
stubbed out aiori backend for rados
additions to get RADOS backend compiling/linking
first cut at rados create/open patha
make sure to return RADOS oid on open/create
implement rados xfer path for WRITE
refactor + implement getfilesize and close
remember to use read_op interface for stat
implement RADOS delete function
don't error in RADOS_Delete for now
implement RADOS set_version
handle open/create flags appropriately
cleanup RADOS error handling
implement read/readcheck/writecheck for RADOS
rados doesn't support directio
implement unsupported aiori ops for RADOS
implement RADOS access call
define rados types if no rados support
It shares the create/open/delete/set_version/get_file_size
functions with POSIX backend.
The mmap backend also supports fsync and fsyncPerWrite options,
and it will use msync() instead and fsync().
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Once a process hits the stonewall (timelimit), they all figure out the maximum pair read/written.
Each proces continues to read/write until the maximum number of pairs is reached, this simulates the wear out.
This makes it so that the buffers are only allocated once per test instead
of once per transfer. This also removes initial buffer set-up from the
timing window.
Added a new struct into ior.h IOR_io_buffers for the buffer, checkbuffer, and readcheckbuffer
so only one pointer needed to be passed to XferBuffersSetup(), XferBuffersFree(),
and WriteOrRead().
Changed the logic in XferBuffersSetup() and XferBuffersFree() to not be transfer
dependent. If a test includes a write check or read check the checkBuffer
and readcheckBuffer will be created once per test in TestIoSys(). The
argument now taken by both function has changed from the access type to
a pointer to IOR_param_t.
Changed WriteOrRead to take as an additional parameter
the IOR_io_buffers struct, since it was no longer creating those
buffers.
Changed how the -l option works. Now you choose the type of datapacket
-l i incompressible data packets
-l incompressible incompressible data packets
-l timestamp timestamped data packets
-l t timestamped data packets
-l offset offset data packets
-l o offset data packets
-G option now is either the seed for the incompressible random packets
or the timestamp, depending on the input to the -l option.
-G will no long timestamp packets on its own without the additon of -l timestamp or -l t
I kept shorter versions of the options for the sake of typing sanity.
These are variants on S3. S3 uses the "pure" S3 interface, e.g. using
Multi-Part-Upload. The "plus" variant enables EMC-extensions in the aws4c
library. This allows the N:N case to use "append", in the case where
"transfer_size" != "block_size" for IOR. In pure S3, the N:N case will
fail, because the EMC-extensions won't be enabled, and appending (which
attempts to use the EMC byte-range tricks to do this) will throw an error.
In the S3_EMC alg, N:1 uses EMCs other byte-range tricks to write different
parts of an N:1 file, and also uses append to write the parts of an N:N
file. Preliminary tests show these EMC extensions look to improve BW by
~20%.
I put all three algs in aiori-S3.c, because it seemed some code was getting
reused. Not sure if that's still going to make sense after the TBD, below.
TBD: Recently realized that the "pure' S3 shouldn't be trying to use
appends for anything. In the N:N case, it should just use MPU, within each
file. Then, there's no need for S3_plus. We just have S3, which does MPU
for all writes where transfer_size != block_size, and uses (standard)
byte-range reads for reading. Then S3_EMC uses "appends for N:N writes,
and byte-range writes for N:1 writes. This separates the code for the two
algs a little more, but we might still want them in the same file.
Testing on our EMC ViPR installation. Therefore, we also have available
some EMC extensions. For example, EMC supports a special "byte-range"
header-option ("Range: bytes=-1-") which allows appending to an object.
This is not needed for N:1 (where every write creates an independent part),
but is vital for N:N (where every write is considered an append, unless
"transfer-size" is the same as "block-size").
We also use a LANL-extended implementation of aws4c 0.5, which provides
some special features, and allows greater efficiency. That is included in
this commit as a tarball. Untar it somewhere else and build it, to produce
a library, which is linked with IOR. (configure with --with-S3).
TBD: EMC also supports a simpler alternative to Multi-Part Upload, which
appears to have several advantages. We'll add that in next, but wanted to
capture this as is, before I break it.
Julian M. Kunkel