Summary:
Upstream pull request #311 (https://github.com/hpc/ior/pull/311) added support
for bencmarking directory renames, inserting an extra entry into the list of
MDTEST operations, but missed updating some existing uses.
This change fixes the remaining uses, to correct misleading ops/sec messages.
Co-authored-by: Ragnar Kjorstad <ragnar@hudson-trading.com>
When running mdtest create + read, fileperproc is set properly so
the driver known it's not a single shared file. But when mdtest is
running with only read (-E) with a pre-existing dataset, fileperproc
is never set, and driver thinks it's a single shared file and can
do optimization to share the file handle.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
- better init/fini logic
- make errors non-fatal and update IO error to look like POSIX driver
- update dir lookup hash table to be more performant
- add dfs_rename
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
This patch adds some minor improvements to the error handling in S3-libS3 when using a single bucket for the deletion operation.
The new environment variable "S3LIB_DELETE_HEURISTICS" allows controlling at which file size of the initial fragment the list_bucket operation is executed. Otherwise, for each deleted file/directory the list_bucket operation is invoked which is very inefficient. Note that S3-libs3 intends to be an "optimal"/best case performance for S3 anyway.
* Support random data generation in utilities. Update first 8 byte element in each 4k block on updates to defy dedup.
* Incorporate different packet types into mdtest/md-workbench.
* Integrated utilities memory pattern tools into IOR. Now all tools use the same patterns.
* Added IOR long option for compatibility between IOR and other tools.
* Added new tests for random buffers.
Context: Options with values are not properly parsed.
If the value is not set, a bug makes appear that 0 was
provided.
This patch fixes options parsing (issue #337). An error
is reported if 'backend.option=' argument is provided.
* Basic support for memory allocation on GPU using CUDA unified memory. Partially addressing #284. IOR support completed.
* Support for GPU alloc in MDTest and MD-Workbench
* Option: support repeated parsing of same option (allows option sharing across modules).
* Checks for gpuDirect
* Integrate gpuDirect options and basic hooks, more testing to be done.
* POSIX: basic gpuDirect implementation working with fake-gpudirect library.
* CUDA allow setting of DeviceID for IOR (not yet MDTest).
* CUDA/GPUDirect Support --with-X=<path>
* Bugfix in option parser for flags that are part of an argument for an option, e.g., -O=1, if 1 is a flag it is wrongly assumed to be a flag.
* Improve and fix the reporting in MDTest. Reporting per rank now outputs the performance of the individual rank (before the barrier), the iteration throughput includes the barrier time (if not -B=0 is set). Before, it was the time after the barrier.
* Clarify the computation of the results.
* MDTest improve CSV output to include aggregated result.
* MDTest allow storing information per rank for later analysis when using the --saveRankPerformanceDetails=<FILE> option
* MDTest: refactored calculation of results, added time_before_executing a barrier.
IOR by default outputs the numbers of errors.
Reason: The amount of error messages can be overwhelming, particularly in a parallel program (Gigabytes...).
One -v increases the verbosity level to provide the extra details without adding too many other messages.
Vitaliy Filippov
ragnarkj