Commit Graph

794 Commits (a490ac8ba698c39a303e70bef584e98e79b49d3a)

Author SHA1 Message Date
Julian M. Kunkel a490ac8ba6 Bugfix IOR writecheck #354 2021-03-29 20:11:10 +02:00
Julian Kunkel e8409edc97
Merge pull request #351 from Kawashima-Azumi/prologues
Support prologue and epilogue for benchmark phases
2021-03-21 18:54:29 +01:00
Julian Kunkel 1b77361e09
Change short option to long option 2021-03-21 18:40:26 +01:00
GUAN Xin 2250e6c5f9 Prologue and epilogue support for benchmark phases 2021-03-20 23:50:04 +08:00
GUAN Xin 84a169c7c2 Clean up timer manipulation 2021-03-20 23:34:43 +08:00
Julian Kunkel a436395570
Support random data generation for memory pattern in utilities. (#348)
* Support random data generation in utilities. Update first 8 byte element in each 4k block on updates to defy dedup.
* Incorporate different packet types into mdtest/md-workbench.
* Integrated utilities memory pattern tools into IOR. Now all tools use the same patterns.
* Added IOR long option for compatibility between IOR and other tools.
* Added new tests for random buffers.
2021-03-18 21:42:50 +01:00
Julian Kunkel 3be3cfb274
Fix MPI-IO strided access. (#347)
* Remove MPI-IO malloc.
* Bugfix for MPI-IO segmented/view.
2021-03-17 00:24:23 +01:00
Mohamad Chaarawi df5fa556c8
reset testcomm to world_comm since it can be used in finalize callbacks. (#344)
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2021-03-13 18:29:40 +01:00
Peter Steinbach 61f36f0c48
converting to sphinx code snippets (#343)
* converting to sphinx code snippets

* more rigorous code highlighting
2021-03-09 17:07:19 +00:00
Mohamad Chaarawi f239b74d83
free MPI comm and group for each iteration they are created in. (#342)
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2021-03-08 18:46:58 +00:00
Rob Latham 4bad282932
Make noncontiguous I/O work again (#341) 2021-03-06 09:53:39 +00:00
Julian Kunkel ad3bf97304
Fixed some issues in build system detected using Clang. (#339) 2021-02-28 11:45:32 +00:00
Jean-Yves VET 0829ac5a18
Fix detection of option with empty value (#338)
Context: Options with values are not properly parsed.
If the value is not set, a bug makes appear that 0 was
provided.

This patch fixes options parsing (issue #337). An error
is reported if 'backend.option=' argument is provided.
2021-02-19 12:30:21 +00:00
Julian Kunkel 74df77430f
Feature ior gpu #284 (#323)
* Basic support for memory allocation on GPU using CUDA unified memory. Partially addressing #284. IOR support completed.
* Support for GPU alloc in MDTest and MD-Workbench
* Option: support repeated parsing of same option (allows option sharing across modules).
* Checks for gpuDirect
* Integrate gpuDirect options and basic hooks, more testing to be done.
* POSIX: basic gpuDirect implementation working with fake-gpudirect library.
* CUDA allow setting of DeviceID for IOR (not yet MDTest).
* CUDA/GPUDirect Support --with-X=<path>
* Bugfix in option parser for flags that are part of an argument for an option, e.g., -O=1, if 1 is a flag it is wrongly assumed to be a flag.
2021-02-18 10:40:42 +00:00
Julian Kunkel e78613d62d
Improve and fix the reporting in MDTest. (#336)
* Improve and fix the reporting in MDTest. Reporting per rank now outputs the performance of the individual rank (before the barrier), the iteration throughput includes the barrier time (if not -B=0 is set). Before, it was the time after the barrier.
* Clarify the computation of the results.
* MDTest improve CSV output to include aggregated result.
2021-02-18 09:53:07 +00:00
Glenn K. Lockwood 4038ebdb21 fix residual references to old main branch name 2021-02-17 12:36:20 -08:00
Julian M. Kunkel a198b0404c Bugfix MDTest: only rank 0 shall print errors. 2021-02-11 15:50:48 +00:00
Julian Kunkel 106eebecb3
MDTest allow storing information per rank for later analysis (#335)
* MDTest allow storing information per rank for later analysis when using the --saveRankPerformanceDetails=<FILE> option
* MDTest: refactored calculation of results, added time_before_executing a barrier.
2021-02-11 09:57:22 +00:00
Julian Kunkel b5963380ae
Feature: IOR rank details in CSV file (#334)
* IOR: Store individual rank results into a CSV file #333

Example usage: ior -O saveRankPerformanceDetailsCSV=test.csv
2021-02-09 17:54:14 +00:00
Julian Kunkel dcacb341e4
Merge pull request #330 from hpc/fix-329
Bugfix user docu for ranks #329. Bugfix missmatching barriers.
2021-02-08 13:54:06 +00:00
Julian M. Kunkel fa5b24f2aa Bugfix user docu for ranks. Bugfix missmatching barriers. 2021-02-08 13:28:13 +00:00
Julian Kunkel f15956247f
Merge pull request #327 from hpc/fix-testing
Testing improvements.
2021-02-05 11:36:11 +00:00
Julian M. Kunkel bdde24bd91 Save testlogs into directory per type. 2021-02-05 11:24:53 +00:00
Julian M. Kunkel b4db470459 Testing improvements.
Uploaded new mdtest patterns. Make them invariant to line nrs.
Added md-workbench tests.
Added --oversubscribe option if MPI understands it (needed for some systems)
2021-02-05 11:17:15 +00:00
Julian Kunkel e5e1053405
Merge pull request #326 from hpc/fix-mdworkbench-check
MDWorkbench: Fix new verification option
2021-02-05 11:14:26 +00:00
Julian M. Kunkel 265cdb2a42 MDWorkbench: Fix new verification option 2021-02-05 10:49:35 +00:00
Julian Kunkel 3f465c61ff
Merge pull request #325 from hpc/fix-init
Move initialization of TestComm before aiori->initialize()
2021-01-29 14:19:57 +00:00
Julian M. Kunkel 7b69fc1500 Free testcom after finalize. 2021-01-29 14:00:17 +00:00
Julian M. Kunkel 351fd4dfb9 Move initialization of TestComm before aiori->initialize() 2021-01-29 09:08:54 +00:00
Julian Kunkel 4c6f33f0f9
Merge pull request #322 from hpc/feature-mdtest-randnum
MDTest data verification improvements.
2021-01-22 15:38:44 +00:00
Julian M. Kunkel 0a066d8285 MD-Worbench: add -G option to set parameter. 2021-01-22 14:38:36 +00:00
Julian M. Kunkel 7061b60ed8 Extracted memory pattern comparison, added mem check option to md-workbench. 2021-01-22 14:05:58 +00:00
Julian M. Kunkel effcb4131c MDTest: add randomness to buffers to defend dedup efforts. Allow to set random offset externally. 2021-01-22 12:24:33 +00:00
Julian M. Kunkel 3af915aae1 MDTest data verification. Fixed bug: added missing reduce. Add rank into I/O buffer to make individual files unique. 2021-01-22 12:10:09 +00:00
Julian Kunkel 310fd37427
Feature mdtest dir rename #306 (#311)
* MDTest: Support for directory renaming.
* Refactored some MDTest variables from hardcoded numbers to ENUM symbols.
2021-01-21 14:10:23 +00:00
Julian Kunkel 4ba0eda38f
Merge pull request #321 from hpc/fix-posix-warn
AIORI POSIX use internal debug macros, include errno in msg.
2021-01-21 11:38:55 +00:00
Julian Kunkel 6e741054ac
Merge pull request #320 from hpc/fix-timer
Remove MPI timer in favor of gettimeofday() to prevent MPI timing issues.
2021-01-21 11:38:29 +00:00
Julian Kunkel a2afcf7740
Merge pull request #319 from hpc/fix-309
MDTest replace fatals with warnings making the API more robust while …
2021-01-21 11:23:43 +00:00
Julian M. Kunkel 852ee3e40e AIORI POSIX use internal debug macros, include errno in msg. 2021-01-21 11:06:13 +00:00
Julian M. Kunkel 4edb27b41a Remove MPI timer in favor of gettimeofday() to prevent MPI issues. Remove time adjustment as measurements are relative anyway. 2021-01-20 19:38:54 +00:00
Julian Kunkel d5f5cf974d
Merge pull request #318 from hpc/fix-comm
Refactoring: remove global comm_world communicator from utilities.c
2021-01-20 18:25:53 +00:00
Julian M. Kunkel eb111f4279 MDTest replace fatals with warnings making the API more robust while it resolves #309.
Example execution (can be done multiple times without cleanup):
$ mpiexec -np 2 ./src/mdtest -d test@test -u -n 100 -C
2021-01-20 15:19:38 +00:00
Julian Kunkel 9e0a8c44d5
IOR: move verbose output of detailed errors to verbosity level 1. (#315)
IOR by default outputs the numbers of errors.
Reason: The amount of error messages can be overwhelming, particularly in a parallel program (Gigabytes...).
One -v increases the verbosity level to provide the extra details without adding too many other messages.
2021-01-20 15:06:45 +00:00
Julian M. Kunkel f345a78046 Removed dependency. 2021-01-20 15:00:33 +00:00
Julian M. Kunkel 970c5ef139 Adjust complex tests for changed -z behavior. 2021-01-20 14:57:21 +00:00
Julian M. Kunkel 40c6d97e72 Replaced MPI_COMM_WORLD where needed with testComm. 2021-01-20 14:35:06 +00:00
Julian M. Kunkel e4120d600d Reduce the dependency to global MPI Communicator 2021-01-20 14:06:05 +00:00
Julian Kunkel 58fbefbd33
Bugfix IOR offsetting (fix before was incomplete). (#317) 2021-01-19 23:19:01 +00:00
Julian Kunkel 19ad73568a
Bugfix ior stonewalling case. (#316)
* Bugfix ior stonewalling case.
2021-01-19 22:28:34 +00:00
Julian Kunkel 3daf7a2d0f
Bugfix: valgrind memory issues (#314) 2021-01-19 21:23:30 +00:00