Julian Kunkel
1b77361e09
Change short option to long option
2021-03-21 18:40:26 +01:00
GUAN Xin
2250e6c5f9
Prologue and epilogue support for benchmark phases
2021-03-20 23:50:04 +08:00
GUAN Xin
84a169c7c2
Clean up timer manipulation
2021-03-20 23:34:43 +08:00
Julian Kunkel
a436395570
Support random data generation for memory pattern in utilities. ( #348 )
...
* Support random data generation in utilities. Update first 8 byte element in each 4k block on updates to defy dedup.
* Incorporate different packet types into mdtest/md-workbench.
* Integrated utilities memory pattern tools into IOR. Now all tools use the same patterns.
* Added IOR long option for compatibility between IOR and other tools.
* Added new tests for random buffers.
2021-03-18 21:42:50 +01:00
Mohamad Chaarawi
df5fa556c8
reset testcomm to world_comm since it can be used in finalize callbacks. ( #344 )
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2021-03-13 18:29:40 +01:00
Mohamad Chaarawi
f239b74d83
free MPI comm and group for each iteration they are created in. ( #342 )
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2021-03-08 18:46:58 +00:00
Julian Kunkel
74df77430f
Feature ior gpu #284 ( #323 )
...
* Basic support for memory allocation on GPU using CUDA unified memory. Partially addressing #284 . IOR support completed.
* Support for GPU alloc in MDTest and MD-Workbench
* Option: support repeated parsing of same option (allows option sharing across modules).
* Checks for gpuDirect
* Integrate gpuDirect options and basic hooks, more testing to be done.
* POSIX: basic gpuDirect implementation working with fake-gpudirect library.
* CUDA allow setting of DeviceID for IOR (not yet MDTest).
* CUDA/GPUDirect Support --with-X=<path>
* Bugfix in option parser for flags that are part of an argument for an option, e.g., -O=1, if 1 is a flag it is wrongly assumed to be a flag.
2021-02-18 10:40:42 +00:00
Julian Kunkel
e78613d62d
Improve and fix the reporting in MDTest. ( #336 )
...
* Improve and fix the reporting in MDTest. Reporting per rank now outputs the performance of the individual rank (before the barrier), the iteration throughput includes the barrier time (if not -B=0 is set). Before, it was the time after the barrier.
* Clarify the computation of the results.
* MDTest improve CSV output to include aggregated result.
2021-02-18 09:53:07 +00:00
Julian M. Kunkel
a198b0404c
Bugfix MDTest: only rank 0 shall print errors.
2021-02-11 15:50:48 +00:00
Julian Kunkel
106eebecb3
MDTest allow storing information per rank for later analysis ( #335 )
...
* MDTest allow storing information per rank for later analysis when using the --saveRankPerformanceDetails=<FILE> option
* MDTest: refactored calculation of results, added time_before_executing a barrier.
2021-02-11 09:57:22 +00:00
Julian M. Kunkel
7061b60ed8
Extracted memory pattern comparison, added mem check option to md-workbench.
2021-01-22 14:05:58 +00:00
Julian M. Kunkel
effcb4131c
MDTest: add randomness to buffers to defend dedup efforts. Allow to set random offset externally.
2021-01-22 12:24:33 +00:00
Julian M. Kunkel
3af915aae1
MDTest data verification. Fixed bug: added missing reduce. Add rank into I/O buffer to make individual files unique.
2021-01-22 12:10:09 +00:00
Julian Kunkel
310fd37427
Feature mdtest dir rename #306 ( #311 )
...
* MDTest: Support for directory renaming.
* Refactored some MDTest variables from hardcoded numbers to ENUM symbols.
2021-01-21 14:10:23 +00:00
Julian Kunkel
a2afcf7740
Merge pull request #319 from hpc/fix-309
...
MDTest replace fatals with warnings making the API more robust while …
2021-01-21 11:23:43 +00:00
Julian M. Kunkel
eb111f4279
MDTest replace fatals with warnings making the API more robust while it resolves #309 .
...
Example execution (can be done multiple times without cleanup):
$ mpiexec -np 2 ./src/mdtest -d test@test -u -n 100 -C
2021-01-20 15:19:38 +00:00
Julian M. Kunkel
970c5ef139
Adjust complex tests for changed -z behavior.
2021-01-20 14:57:21 +00:00
Julian M. Kunkel
e4120d600d
Reduce the dependency to global MPI Communicator
2021-01-20 14:06:05 +00:00
Julian Kunkel
3daf7a2d0f
Bugfix: valgrind memory issues ( #314 )
2021-01-19 21:23:30 +00:00
Julian Kunkel
b2089514e3
MDTest calculate min/max/mean across iterations #300 . ( #312 )
...
The calculation per iteration first computes the value of the slowest process, i.e., highest time or lowest rate. This is then the value for the iteration.
Secondly, calculate the min/max/mean across iterations.
For tree operations, the value is identical to previous as only Rank 0 is involved.
2021-01-18 21:30:16 +00:00
Julian Kunkel
33b70ecbc8
Fix MDTest multidir option to work with stonewall file. ( #308 )
2021-01-14 17:38:11 +00:00
Julian Kunkel
4a96436928
MDTest convert FATAL to WARN ( #307 )
2021-01-14 16:41:56 +00:00
Richard Mohr
41184c69e2
mdtest: fix reference to unique_dir_per_task for builds --with-lustre
2020-12-01 01:22:31 -05:00
Julian M. Kunkel
ae06908a93
Merge branch 'master' into fix-mdtest-iter
2020-11-30 14:16:41 +00:00
Julian M. Kunkel
11c784c8bd
Integrate review feedback.
2020-11-28 10:40:41 +00:00
Julian M. Kunkel
ae23523a70
Integrate review feedback
2020-11-28 10:34:20 +00:00
Julian M. Kunkel
5799e4ef3a
MDTest remove unnede variable.
2020-11-27 18:02:14 +00:00
Julian M. Kunkel
fd51654393
Readd check for mdtest for now.
2020-11-27 17:51:31 +00:00
Julian M. Kunkel
fbf976351a
MDTest refactoring: move all global static variables into the global static structure.
2020-11-27 17:49:45 +00:00
Julian M. Kunkel
ae8a11b42f
MDTest updated stonewall check.
2020-11-27 15:35:32 +00:00
Julian M. Kunkel
7542e75c82
MDTest ignore verbose potential format overflows to be able to spot real errors.
2020-11-27 15:23:32 +00:00
Julian M. Kunkel
4377aebcf8
Bugfix MDTest calculation of multiple iterations was incorrect.
2020-11-26 12:48:11 +00:00
Julian M. Kunkel
ad6dfc5e63
Allow MDTest to print per proc.
2020-11-25 09:50:26 +00:00
Julian M. Kunkel
b5891141d8
Move checks before inititalization.
...
Add simple validation for S3.
2020-11-22 11:43:42 +00:00
Julian M. Kunkel
277f380139
Remove duplicated functionality between mdtest and IOR.
...
Refactored the ShowFileSystemSize function.
2020-11-09 16:23:34 +00:00
Julian M. Kunkel
f275671cc9
AIORI POSIX AIO support. Collect ops until granularity is reached, then submit pending IOs. Synchronize latest on close. Doesn't work with data verification and reuses the existing buffer. The implementation shows the potential AIO may have.
...
Extract also the POSIX header from AIORI to allow better reuse. #240
2020-07-21 16:18:50 +01:00
Julian M. Kunkel
4258e14c11
mdtest: Bugfix of changed behavior, can only remove dir if all children were deleted.
2020-07-21 09:31:19 +01:00
Julian M. Kunkel
8ca388fc78
Merge branch 'feature-cleanup-aiori'
2020-07-21 09:26:03 +01:00
Karsten Weiss
f280123d0b
Spelling fixes (found by codespell)
2020-07-03 09:16:30 +02:00
Julian M. Kunkel
4f7350dc6a
MDTest remove testdir if created by MDTest.
2020-07-02 16:26:05 +01:00
Julian M. Kunkel
fb8f43fcfe
Reduce verbosity for verification
2020-07-01 10:05:09 +01:00
Julian M. Kunkel
5a5b90cdaa
Run existing check to allow only supported backends in mdtest.
2020-06-30 11:17:46 +01:00
Julian M. Kunkel
03dbb20594
MDTest changed verification pattern. Read now always checks the first byte/8 bytes for the signature (item number).
...
Added also --verify-write option which performs a read immediately after a write. Supports #206
2020-06-29 20:15:14 +01:00
Julian M. Kunkel
df3f7082d8
MDTest: Added warnings/errors for errorneous read pattern #206
2020-06-28 17:21:24 +01:00
Julian M. Kunkel
0bffd14de7
Added --warningAsErrors option to IOR and MDTest and refactored WARNINGs in IOR. #174
2020-06-24 11:13:12 +01:00
Mohamad Chaarawi
b4a54c34e8
Merge branch 'master' into fix-modules
...
Conflicts:
src/aiori-DAOS.c
src/aiori-DFS.c
2020-06-11 19:55:23 +00:00
Julian M. Kunkel
eb2cd27fac
Removed offset from IOR structure.
2020-06-10 17:47:07 +01:00
Sven Breuner
c828a0f32c
mdtest: use correct number of bytes for posix_memalign of read buffer
...
Fixes a typo in initial posix_memalign commit for DirectIO support.
2020-06-03 01:22:44 +03:00
Sven Breuner
3eb488939d
mdtest: allocate aligned buffers to support DirectIO
2020-06-03 00:30:38 +03:00
Julian M. Kunkel
5663593919
Moved generic IOR information to "hint" structure. Backends should only use the "hints" if set.
2020-05-31 12:50:03 +01:00