Commit Graph

587 Commits (1b77361e094a5e46d65982a3dce9880895ca4787)

Author SHA1 Message Date
Julian Kunkel 1b77361e09
Change short option to long option 2021-03-21 18:40:26 +01:00
GUAN Xin 2250e6c5f9 Prologue and epilogue support for benchmark phases 2021-03-20 23:50:04 +08:00
GUAN Xin 84a169c7c2 Clean up timer manipulation 2021-03-20 23:34:43 +08:00
Julian Kunkel a436395570
Support random data generation for memory pattern in utilities. (#348)
* Support random data generation in utilities. Update first 8 byte element in each 4k block on updates to defy dedup.
* Incorporate different packet types into mdtest/md-workbench.
* Integrated utilities memory pattern tools into IOR. Now all tools use the same patterns.
* Added IOR long option for compatibility between IOR and other tools.
* Added new tests for random buffers.
2021-03-18 21:42:50 +01:00
Julian Kunkel 3be3cfb274
Fix MPI-IO strided access. (#347)
* Remove MPI-IO malloc.
* Bugfix for MPI-IO segmented/view.
2021-03-17 00:24:23 +01:00
Mohamad Chaarawi df5fa556c8
reset testcomm to world_comm since it can be used in finalize callbacks. (#344)
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2021-03-13 18:29:40 +01:00
Mohamad Chaarawi f239b74d83
free MPI comm and group for each iteration they are created in. (#342)
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2021-03-08 18:46:58 +00:00
Rob Latham 4bad282932
Make noncontiguous I/O work again (#341) 2021-03-06 09:53:39 +00:00
Julian Kunkel ad3bf97304
Fixed some issues in build system detected using Clang. (#339) 2021-02-28 11:45:32 +00:00
Jean-Yves VET 0829ac5a18
Fix detection of option with empty value (#338)
Context: Options with values are not properly parsed.
If the value is not set, a bug makes appear that 0 was
provided.

This patch fixes options parsing (issue #337). An error
is reported if 'backend.option=' argument is provided.
2021-02-19 12:30:21 +00:00
Julian Kunkel 74df77430f
Feature ior gpu #284 (#323)
* Basic support for memory allocation on GPU using CUDA unified memory. Partially addressing #284. IOR support completed.
* Support for GPU alloc in MDTest and MD-Workbench
* Option: support repeated parsing of same option (allows option sharing across modules).
* Checks for gpuDirect
* Integrate gpuDirect options and basic hooks, more testing to be done.
* POSIX: basic gpuDirect implementation working with fake-gpudirect library.
* CUDA allow setting of DeviceID for IOR (not yet MDTest).
* CUDA/GPUDirect Support --with-X=<path>
* Bugfix in option parser for flags that are part of an argument for an option, e.g., -O=1, if 1 is a flag it is wrongly assumed to be a flag.
2021-02-18 10:40:42 +00:00
Julian Kunkel e78613d62d
Improve and fix the reporting in MDTest. (#336)
* Improve and fix the reporting in MDTest. Reporting per rank now outputs the performance of the individual rank (before the barrier), the iteration throughput includes the barrier time (if not -B=0 is set). Before, it was the time after the barrier.
* Clarify the computation of the results.
* MDTest improve CSV output to include aggregated result.
2021-02-18 09:53:07 +00:00
Julian M. Kunkel a198b0404c Bugfix MDTest: only rank 0 shall print errors. 2021-02-11 15:50:48 +00:00
Julian Kunkel 106eebecb3
MDTest allow storing information per rank for later analysis (#335)
* MDTest allow storing information per rank for later analysis when using the --saveRankPerformanceDetails=<FILE> option
* MDTest: refactored calculation of results, added time_before_executing a barrier.
2021-02-11 09:57:22 +00:00
Julian Kunkel b5963380ae
Feature: IOR rank details in CSV file (#334)
* IOR: Store individual rank results into a CSV file #333

Example usage: ior -O saveRankPerformanceDetailsCSV=test.csv
2021-02-09 17:54:14 +00:00
Julian M. Kunkel fa5b24f2aa Bugfix user docu for ranks. Bugfix missmatching barriers. 2021-02-08 13:28:13 +00:00
Julian Kunkel e5e1053405
Merge pull request #326 from hpc/fix-mdworkbench-check
MDWorkbench: Fix new verification option
2021-02-05 11:14:26 +00:00
Julian M. Kunkel 265cdb2a42 MDWorkbench: Fix new verification option 2021-02-05 10:49:35 +00:00
Julian M. Kunkel 7b69fc1500 Free testcom after finalize. 2021-01-29 14:00:17 +00:00
Julian M. Kunkel 351fd4dfb9 Move initialization of TestComm before aiori->initialize() 2021-01-29 09:08:54 +00:00
Julian M. Kunkel 0a066d8285 MD-Worbench: add -G option to set parameter. 2021-01-22 14:38:36 +00:00
Julian M. Kunkel 7061b60ed8 Extracted memory pattern comparison, added mem check option to md-workbench. 2021-01-22 14:05:58 +00:00
Julian M. Kunkel effcb4131c MDTest: add randomness to buffers to defend dedup efforts. Allow to set random offset externally. 2021-01-22 12:24:33 +00:00
Julian M. Kunkel 3af915aae1 MDTest data verification. Fixed bug: added missing reduce. Add rank into I/O buffer to make individual files unique. 2021-01-22 12:10:09 +00:00
Julian Kunkel 310fd37427
Feature mdtest dir rename #306 (#311)
* MDTest: Support for directory renaming.
* Refactored some MDTest variables from hardcoded numbers to ENUM symbols.
2021-01-21 14:10:23 +00:00
Julian Kunkel 4ba0eda38f
Merge pull request #321 from hpc/fix-posix-warn
AIORI POSIX use internal debug macros, include errno in msg.
2021-01-21 11:38:55 +00:00
Julian Kunkel 6e741054ac
Merge pull request #320 from hpc/fix-timer
Remove MPI timer in favor of gettimeofday() to prevent MPI timing issues.
2021-01-21 11:38:29 +00:00
Julian Kunkel a2afcf7740
Merge pull request #319 from hpc/fix-309
MDTest replace fatals with warnings making the API more robust while …
2021-01-21 11:23:43 +00:00
Julian M. Kunkel 852ee3e40e AIORI POSIX use internal debug macros, include errno in msg. 2021-01-21 11:06:13 +00:00
Julian M. Kunkel 4edb27b41a Remove MPI timer in favor of gettimeofday() to prevent MPI issues. Remove time adjustment as measurements are relative anyway. 2021-01-20 19:38:54 +00:00
Julian Kunkel d5f5cf974d
Merge pull request #318 from hpc/fix-comm
Refactoring: remove global comm_world communicator from utilities.c
2021-01-20 18:25:53 +00:00
Julian M. Kunkel eb111f4279 MDTest replace fatals with warnings making the API more robust while it resolves #309.
Example execution (can be done multiple times without cleanup):
$ mpiexec -np 2 ./src/mdtest -d test@test -u -n 100 -C
2021-01-20 15:19:38 +00:00
Julian Kunkel 9e0a8c44d5
IOR: move verbose output of detailed errors to verbosity level 1. (#315)
IOR by default outputs the numbers of errors.
Reason: The amount of error messages can be overwhelming, particularly in a parallel program (Gigabytes...).
One -v increases the verbosity level to provide the extra details without adding too many other messages.
2021-01-20 15:06:45 +00:00
Julian M. Kunkel f345a78046 Removed dependency. 2021-01-20 15:00:33 +00:00
Julian M. Kunkel 970c5ef139 Adjust complex tests for changed -z behavior. 2021-01-20 14:57:21 +00:00
Julian M. Kunkel 40c6d97e72 Replaced MPI_COMM_WORLD where needed with testComm. 2021-01-20 14:35:06 +00:00
Julian M. Kunkel e4120d600d Reduce the dependency to global MPI Communicator 2021-01-20 14:06:05 +00:00
Julian Kunkel 58fbefbd33
Bugfix IOR offsetting (fix before was incomplete). (#317) 2021-01-19 23:19:01 +00:00
Julian Kunkel 19ad73568a
Bugfix ior stonewalling case. (#316)
* Bugfix ior stonewalling case.
2021-01-19 22:28:34 +00:00
Julian Kunkel 3daf7a2d0f
Bugfix: valgrind memory issues (#314) 2021-01-19 21:23:30 +00:00
Mohamad Chaarawi fa316d5d24
dfs: conditionally compile out svcl setting (#313)
The SVCL argument is being removed from the DAOS API, so conditionally
compile out while maintaining backwards compatibility to versions where
 it's still required.
2021-01-19 13:43:01 +00:00
Julian Kunkel b2089514e3
MDTest calculate min/max/mean across iterations #300. (#312)
The calculation per iteration first computes the value of the slowest process, i.e., highest time or lowest rate. This is then the value for the iteration.
Secondly, calculate the min/max/mean across iterations.
For tree operations, the value is identical to previous as only Rank 0 is involved.
2021-01-18 21:30:16 +00:00
Julian Kunkel 33b70ecbc8
Fix MDTest multidir option to work with stonewall file. (#308) 2021-01-14 17:38:11 +00:00
Julian Kunkel 4a96436928
MDTest convert FATAL to WARN (#307) 2021-01-14 16:41:56 +00:00
Julian Kunkel d339caa501
Updated test (illustration of an empty test) and distclean for make distcheck #304 (#305)
Resolves #304
2021-01-04 10:50:38 -08:00
Julian M. Kunkel 21bf5a5a12 NCMPI ported to current ADIO. Fixed autoconf #303. 2020-12-25 16:49:04 +00:00
Julian Kunkel 8de13884a7
HDFS module ported to current AIORI API and improved (#302)
* HDFS module ported to current AIORI API and extended
* Added instructions how to build and run with a HDFS.
* Replaced read with pread to support offsets.
* Implemented HDFS MDTest API
* Improved sync semantics
2020-12-23 11:51:31 +00:00
Mohamad Chaarawi 6675cd50bf aiori-DFS: stat should not be fatal
ior now expects stat to not be fatal. update the DFS driver to not
exit if stat fails since the file can simply not exist.

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-12-16 15:41:58 +00:00
Frank 0bd52884e8
added region and location to libs3 (#295)
* added region and location to libs3
2020-12-11 13:05:24 +00:00
Julian Kunkel fbee4139d8
Merge pull request #292 from ofaaland/b-fix-mpi-checkf
Do not execute functions twice in MPI_CHECKF
2020-12-04 09:20:21 +00:00