Commit Graph

822 Commits (master)

Author SHA1 Message Date
Vitaliy Filippov 2976361c8b Add fsync after each test in mdtest 2021-09-01 21:04:47 +03:00
ragnarkj fc909ccfc7
Fix off-by-one error in verbose ops/sec messages. (#382)
Upstream pull request #311 ( added support
for bencmarking directory renames, inserting an extra entry into the list of
MDTEST operations, but missed updating some existing uses.

This change fixes the remaining uses, to correct misleading ops/sec messages.

Co-authored-by: Ragnar Kjorstad <>
2021-07-14 12:19:11 +02:00
Mohamad Chaarawi 88e90d5bbf
mdtest: make sure fileperproc is set for mdtest read (#380)
When running mdtest create + read, fileperproc is set properly so
the driver known it's not a single shared file. But when mdtest is
running with only read (-E) with a pre-existing dataset, fileperproc
is never set, and driver thinks it's a single shared file and can
do optimization to share the file handle.

Signed-off-by: Mohamad Chaarawi <>
2021-07-13 17:30:24 +02:00
Mohamad Chaarawi 2727475158
aiori/DFS: several fixes and cleanup to DFS driver (#379)
- better init/fini logic
- make errors non-fatal and update IO error to look like POSIX driver
- update dir lookup hash table to be more performant
- add dfs_rename

Signed-off-by: Mohamad Chaarawi <>
2021-07-13 09:05:48 +02:00
Julian Kunkel 1d4c68a30c
Merge pull request #377 from madsys-dev/fix-offset-overflow
IOR: fix overflow in offset calculations
2021-05-30 15:35:21 +02:00
Runji Wang 2b5319b37c IOR: fix overflow in offset calculations
In expression 'i * test->numTasks', both operands have int type.
It will overflow 32 bits when the 'segmentCount' and 'numTasks' are large enough.
2021-05-30 18:56:58 +08:00
Julian Kunkel 03f72b4b7c
Merge pull request #376 from hpc/feat-dummy-test
Dummy plugin, add delay for close and sync.
2021-05-25 09:44:56 +02:00
Julian M. Kunkel 8d5bcf7a23 Dummy plugin, add delay for close and sync.
Example usage:
ior -a DUMMY --dummy.delay-create=10000 --dummy.delay-close=100000 --dummy.delay-xfer=20000
2021-05-25 09:40:50 +02:00
Julian Kunkel d7603c601f
Merge pull request #374 from mchaarawi/dfs_statfs
aiori-DFS: add statfs implementation for DFS driver
2021-05-18 17:31:14 +02:00
Mohamad Chaarawi b2223f1725 aiori-DFS: add statfs implementation for DFS driver 2021-05-18 13:51:05 +00:00
Julian Kunkel bff3202806
Merge pull request #373 from adrianjhpc/master
Add in collective options for script file
2021-05-18 14:08:26 +02:00
Adrian Jackson d212b4547e Adding back in collective option to the script file 2021-05-18 13:02:20 +01:00
Adrian Jackson 6445267e10
Merge pull request #4 from hpc/main
Merging in master
2021-05-18 13:00:22 +01:00
Julian Kunkel 8475c7d300
Merge pull request #367 from hpc/fix-s3collision
Bugfix in AIORI-S3-libs3 to prevent name collisions.
2021-05-03 18:49:09 +02:00
Julian M. Kunkel 81ac754c68 Bugfix in AIORI-S3-libs3 to prevent name collisions. Also allow to output verification errors. 2021-05-03 18:46:00 +02:00
Julian Kunkel 130f85462c
Merge pull request #366 from hpc/fix-tests
Bugfix advanced tests to work with new validation with stonewall.
2021-04-28 13:42:13 +02:00
Julian M. Kunkel e8eb77166a Bugfix advanced tests to work with new validation with stonewall. 2021-04-28 13:41:19 +02:00
Julian M. Kunkel 47f3709c68 MDTest bugfix options -f -l -s and add extra option check #365 2021-04-27 14:50:19 +02:00
Julian Kunkel a647f48b38
AIORI POSIX use open flag. #363 (#364)
Fixes #363
2021-04-26 09:10:47 -07:00
Julian Kunkel d19db1085a
Merge pull request #362 from hpc/s3-improve
This patch adds some minor improvements to the error handling in S3-libS3 when using a single bucket for the deletion operation.

The new environment variable "S3LIB_DELETE_HEURISTICS" allows controlling at which file size of the initial fragment the list_bucket operation is executed. Otherwise, for each deleted file/directory the list_bucket operation is invoked which is very inefficient. Note that S3-libs3 intends to be an "optimal"/best case performance for S3 anyway.
2021-04-23 12:24:21 +02:00
Julian M. Kunkel be80b4e618 S3-libs3 error handling improved, added a heuristic for testing as well. 2021-04-23 12:20:11 +02:00
Julian Kunkel 11a81e42ca
Merge pull request #360 from efajardo/main
Adding number of tasks to the CSV output
2021-04-21 08:47:38 +02:00
efajardo 39fcb5970d Adding number of tasks to the CSV output 2021-04-20 15:38:04 -07:00
Julian Kunkel 0a4d58576d
Merge pull request #359 from hpc/fix-347
IOR: add several sanity checks for stonewalling #345
2021-04-14 08:43:00 +02:00
Julian M. Kunkel 0410a38e98 IOR: add several sanity checks for stonewalling #345 2021-04-13 11:01:30 +02:00
Julian Kunkel a90d414a30
Merge pull request #358 from hpc/fix-357
MDTest, support flag to show per rank performance: #357
2021-04-07 18:28:20 +02:00
Julian M. Kunkel 8fc7ab5c9c MDTest, support flag to show per rank performance: #357
Also fix a potential issue for the calculation if a test is not run (number of items = 0).
2021-04-01 17:23:10 +02:00
Julian Kunkel 0ed2dc3187
Merge pull request #355 from hpc/fix-354
Bugfix IOR writecheck #354
2021-03-29 21:08:31 +02:00
Julian M. Kunkel a490ac8ba6 Bugfix IOR writecheck #354 2021-03-29 20:11:10 +02:00
Julian Kunkel e8409edc97
Merge pull request #351 from Kawashima-Azumi/prologues
Support prologue and epilogue for benchmark phases
2021-03-21 18:54:29 +01:00
Julian Kunkel 1b77361e09
Change short option to long option 2021-03-21 18:40:26 +01:00
GUAN Xin 2250e6c5f9 Prologue and epilogue support for benchmark phases 2021-03-20 23:50:04 +08:00
GUAN Xin 84a169c7c2 Clean up timer manipulation 2021-03-20 23:34:43 +08:00
Julian Kunkel a436395570
Support random data generation for memory pattern in utilities. (#348)
* Support random data generation in utilities. Update first 8 byte element in each 4k block on updates to defy dedup.
* Incorporate different packet types into mdtest/md-workbench.
* Integrated utilities memory pattern tools into IOR. Now all tools use the same patterns.
* Added IOR long option for compatibility between IOR and other tools.
* Added new tests for random buffers.
2021-03-18 21:42:50 +01:00
Julian Kunkel 3be3cfb274
Fix MPI-IO strided access. (#347)
* Remove MPI-IO malloc.
* Bugfix for MPI-IO segmented/view.
2021-03-17 00:24:23 +01:00
Mohamad Chaarawi df5fa556c8
reset testcomm to world_comm since it can be used in finalize callbacks. (#344)
Signed-off-by: Mohamad Chaarawi <>
2021-03-13 18:29:40 +01:00
Peter Steinbach 61f36f0c48
converting to sphinx code snippets (#343)
* converting to sphinx code snippets

* more rigorous code highlighting
2021-03-09 17:07:19 +00:00
Mohamad Chaarawi f239b74d83
free MPI comm and group for each iteration they are created in. (#342)
Signed-off-by: Mohamad Chaarawi <>
2021-03-08 18:46:58 +00:00
Rob Latham 4bad282932
Make noncontiguous I/O work again (#341) 2021-03-06 09:53:39 +00:00
Julian Kunkel ad3bf97304
Fixed some issues in build system detected using Clang. (#339) 2021-02-28 11:45:32 +00:00
Jean-Yves VET 0829ac5a18
Fix detection of option with empty value (#338)
Context: Options with values are not properly parsed.
If the value is not set, a bug makes appear that 0 was

This patch fixes options parsing (issue #337). An error
is reported if 'backend.option=' argument is provided.
2021-02-19 12:30:21 +00:00
Julian Kunkel 74df77430f
Feature ior gpu #284 (#323)
* Basic support for memory allocation on GPU using CUDA unified memory. Partially addressing #284. IOR support completed.
* Support for GPU alloc in MDTest and MD-Workbench
* Option: support repeated parsing of same option (allows option sharing across modules).
* Checks for gpuDirect
* Integrate gpuDirect options and basic hooks, more testing to be done.
* POSIX: basic gpuDirect implementation working with fake-gpudirect library.
* CUDA allow setting of DeviceID for IOR (not yet MDTest).
* CUDA/GPUDirect Support --with-X=<path>
* Bugfix in option parser for flags that are part of an argument for an option, e.g., -O=1, if 1 is a flag it is wrongly assumed to be a flag.
2021-02-18 10:40:42 +00:00
Julian Kunkel e78613d62d
Improve and fix the reporting in MDTest. (#336)
* Improve and fix the reporting in MDTest. Reporting per rank now outputs the performance of the individual rank (before the barrier), the iteration throughput includes the barrier time (if not -B=0 is set). Before, it was the time after the barrier.
* Clarify the computation of the results.
* MDTest improve CSV output to include aggregated result.
2021-02-18 09:53:07 +00:00
Glenn K. Lockwood 4038ebdb21 fix residual references to old main branch name 2021-02-17 12:36:20 -08:00
Julian M. Kunkel a198b0404c Bugfix MDTest: only rank 0 shall print errors. 2021-02-11 15:50:48 +00:00
Julian Kunkel 106eebecb3
MDTest allow storing information per rank for later analysis (#335)
* MDTest allow storing information per rank for later analysis when using the --saveRankPerformanceDetails=<FILE> option
* MDTest: refactored calculation of results, added time_before_executing a barrier.
2021-02-11 09:57:22 +00:00
Julian Kunkel b5963380ae
Feature: IOR rank details in CSV file (#334)
* IOR: Store individual rank results into a CSV file #333

Example usage: ior -O saveRankPerformanceDetailsCSV=test.csv
2021-02-09 17:54:14 +00:00
Julian Kunkel dcacb341e4
Merge pull request #330 from hpc/fix-329
Bugfix user docu for ranks #329. Bugfix missmatching barriers.
2021-02-08 13:54:06 +00:00
Julian M. Kunkel fa5b24f2aa Bugfix user docu for ranks. Bugfix missmatching barriers. 2021-02-08 13:28:13 +00:00
Julian Kunkel f15956247f
Merge pull request #327 from hpc/fix-testing
Testing improvements.
2021-02-05 11:36:11 +00:00