Commit Graph

783 Commits (ad3bf97304ff1d623c83661521ef80e19e0d9c23)

Author SHA1 Message Date
Julian Kunkel ad3bf97304
Fixed some issues in build system detected using Clang. (#339) 2021-02-28 11:45:32 +00:00
Jean-Yves VET 0829ac5a18
Fix detection of option with empty value (#338)
Context: Options with values are not properly parsed.
If the value is not set, a bug makes appear that 0 was
provided.

This patch fixes options parsing (issue #337). An error
is reported if 'backend.option=' argument is provided.
2021-02-19 12:30:21 +00:00
Julian Kunkel 74df77430f
Feature ior gpu #284 (#323)
* Basic support for memory allocation on GPU using CUDA unified memory. Partially addressing #284. IOR support completed.
* Support for GPU alloc in MDTest and MD-Workbench
* Option: support repeated parsing of same option (allows option sharing across modules).
* Checks for gpuDirect
* Integrate gpuDirect options and basic hooks, more testing to be done.
* POSIX: basic gpuDirect implementation working with fake-gpudirect library.
* CUDA allow setting of DeviceID for IOR (not yet MDTest).
* CUDA/GPUDirect Support --with-X=<path>
* Bugfix in option parser for flags that are part of an argument for an option, e.g., -O=1, if 1 is a flag it is wrongly assumed to be a flag.
2021-02-18 10:40:42 +00:00
Julian Kunkel e78613d62d
Improve and fix the reporting in MDTest. (#336)
* Improve and fix the reporting in MDTest. Reporting per rank now outputs the performance of the individual rank (before the barrier), the iteration throughput includes the barrier time (if not -B=0 is set). Before, it was the time after the barrier.
* Clarify the computation of the results.
* MDTest improve CSV output to include aggregated result.
2021-02-18 09:53:07 +00:00
Glenn K. Lockwood 4038ebdb21 fix residual references to old main branch name 2021-02-17 12:36:20 -08:00
Julian M. Kunkel a198b0404c Bugfix MDTest: only rank 0 shall print errors. 2021-02-11 15:50:48 +00:00
Julian Kunkel 106eebecb3
MDTest allow storing information per rank for later analysis (#335)
* MDTest allow storing information per rank for later analysis when using the --saveRankPerformanceDetails=<FILE> option
* MDTest: refactored calculation of results, added time_before_executing a barrier.
2021-02-11 09:57:22 +00:00
Julian Kunkel b5963380ae
Feature: IOR rank details in CSV file (#334)
* IOR: Store individual rank results into a CSV file #333

Example usage: ior -O saveRankPerformanceDetailsCSV=test.csv
2021-02-09 17:54:14 +00:00
Julian Kunkel dcacb341e4
Merge pull request #330 from hpc/fix-329
Bugfix user docu for ranks #329. Bugfix missmatching barriers.
2021-02-08 13:54:06 +00:00
Julian M. Kunkel fa5b24f2aa Bugfix user docu for ranks. Bugfix missmatching barriers. 2021-02-08 13:28:13 +00:00
Julian Kunkel f15956247f
Merge pull request #327 from hpc/fix-testing
Testing improvements.
2021-02-05 11:36:11 +00:00
Julian M. Kunkel bdde24bd91 Save testlogs into directory per type. 2021-02-05 11:24:53 +00:00
Julian M. Kunkel b4db470459 Testing improvements.
Uploaded new mdtest patterns. Make them invariant to line nrs.
Added md-workbench tests.
Added --oversubscribe option if MPI understands it (needed for some systems)
2021-02-05 11:17:15 +00:00
Julian Kunkel e5e1053405
Merge pull request #326 from hpc/fix-mdworkbench-check
MDWorkbench: Fix new verification option
2021-02-05 11:14:26 +00:00
Julian M. Kunkel 265cdb2a42 MDWorkbench: Fix new verification option 2021-02-05 10:49:35 +00:00
Julian Kunkel 3f465c61ff
Merge pull request #325 from hpc/fix-init
Move initialization of TestComm before aiori->initialize()
2021-01-29 14:19:57 +00:00
Julian M. Kunkel 7b69fc1500 Free testcom after finalize. 2021-01-29 14:00:17 +00:00
Julian M. Kunkel 351fd4dfb9 Move initialization of TestComm before aiori->initialize() 2021-01-29 09:08:54 +00:00
Julian Kunkel 4c6f33f0f9
Merge pull request #322 from hpc/feature-mdtest-randnum
MDTest data verification improvements.
2021-01-22 15:38:44 +00:00
Julian M. Kunkel 0a066d8285 MD-Worbench: add -G option to set parameter. 2021-01-22 14:38:36 +00:00
Julian M. Kunkel 7061b60ed8 Extracted memory pattern comparison, added mem check option to md-workbench. 2021-01-22 14:05:58 +00:00
Julian M. Kunkel effcb4131c MDTest: add randomness to buffers to defend dedup efforts. Allow to set random offset externally. 2021-01-22 12:24:33 +00:00
Julian M. Kunkel 3af915aae1 MDTest data verification. Fixed bug: added missing reduce. Add rank into I/O buffer to make individual files unique. 2021-01-22 12:10:09 +00:00
Julian Kunkel 310fd37427
Feature mdtest dir rename #306 (#311)
* MDTest: Support for directory renaming.
* Refactored some MDTest variables from hardcoded numbers to ENUM symbols.
2021-01-21 14:10:23 +00:00
Julian Kunkel 4ba0eda38f
Merge pull request #321 from hpc/fix-posix-warn
AIORI POSIX use internal debug macros, include errno in msg.
2021-01-21 11:38:55 +00:00
Julian Kunkel 6e741054ac
Merge pull request #320 from hpc/fix-timer
Remove MPI timer in favor of gettimeofday() to prevent MPI timing issues.
2021-01-21 11:38:29 +00:00
Julian Kunkel a2afcf7740
Merge pull request #319 from hpc/fix-309
MDTest replace fatals with warnings making the API more robust while …
2021-01-21 11:23:43 +00:00
Julian M. Kunkel 852ee3e40e AIORI POSIX use internal debug macros, include errno in msg. 2021-01-21 11:06:13 +00:00
Julian M. Kunkel 4edb27b41a Remove MPI timer in favor of gettimeofday() to prevent MPI issues. Remove time adjustment as measurements are relative anyway. 2021-01-20 19:38:54 +00:00
Julian Kunkel d5f5cf974d
Merge pull request #318 from hpc/fix-comm
Refactoring: remove global comm_world communicator from utilities.c
2021-01-20 18:25:53 +00:00
Julian M. Kunkel eb111f4279 MDTest replace fatals with warnings making the API more robust while it resolves #309.
Example execution (can be done multiple times without cleanup):
$ mpiexec -np 2 ./src/mdtest -d test@test -u -n 100 -C
2021-01-20 15:19:38 +00:00
Julian Kunkel 9e0a8c44d5
IOR: move verbose output of detailed errors to verbosity level 1. (#315)
IOR by default outputs the numbers of errors.
Reason: The amount of error messages can be overwhelming, particularly in a parallel program (Gigabytes...).
One -v increases the verbosity level to provide the extra details without adding too many other messages.
2021-01-20 15:06:45 +00:00
Julian M. Kunkel f345a78046 Removed dependency. 2021-01-20 15:00:33 +00:00
Julian M. Kunkel 970c5ef139 Adjust complex tests for changed -z behavior. 2021-01-20 14:57:21 +00:00
Julian M. Kunkel 40c6d97e72 Replaced MPI_COMM_WORLD where needed with testComm. 2021-01-20 14:35:06 +00:00
Julian M. Kunkel e4120d600d Reduce the dependency to global MPI Communicator 2021-01-20 14:06:05 +00:00
Julian Kunkel 58fbefbd33
Bugfix IOR offsetting (fix before was incomplete). (#317) 2021-01-19 23:19:01 +00:00
Julian Kunkel 19ad73568a
Bugfix ior stonewalling case. (#316)
* Bugfix ior stonewalling case.
2021-01-19 22:28:34 +00:00
Julian Kunkel 3daf7a2d0f
Bugfix: valgrind memory issues (#314) 2021-01-19 21:23:30 +00:00
Mohamad Chaarawi fa316d5d24
dfs: conditionally compile out svcl setting (#313)
The SVCL argument is being removed from the DAOS API, so conditionally
compile out while maintaining backwards compatibility to versions where
 it's still required.
2021-01-19 13:43:01 +00:00
Julian Kunkel b2089514e3
MDTest calculate min/max/mean across iterations #300. (#312)
The calculation per iteration first computes the value of the slowest process, i.e., highest time or lowest rate. This is then the value for the iteration.
Secondly, calculate the min/max/mean across iterations.
For tree operations, the value is identical to previous as only Rank 0 is involved.
2021-01-18 21:30:16 +00:00
Julian Kunkel 33b70ecbc8
Fix MDTest multidir option to work with stonewall file. (#308) 2021-01-14 17:38:11 +00:00
Julian Kunkel 4a96436928
MDTest convert FATAL to WARN (#307) 2021-01-14 16:41:56 +00:00
Julian Kunkel d339caa501
Updated test (illustration of an empty test) and distclean for make distcheck #304 (#305)
Resolves #304
2021-01-04 10:50:38 -08:00
Julian M. Kunkel 21bf5a5a12 NCMPI ported to current ADIO. Fixed autoconf #303. 2020-12-25 16:49:04 +00:00
Glenn K. Lockwood eb883f533a update release docs based on 3.3.0 release experience 2020-12-23 18:17:47 -08:00
Glenn K. Lockwood e579531970 update NEWS for 3.3 release 2020-12-23 17:06:56 -08:00
Glenn K. Lockwood f467903cb9
draft of a new simplified release process (#297) 2020-12-23 11:56:49 +00:00
Julian Kunkel 8de13884a7
HDFS module ported to current AIORI API and improved (#302)
* HDFS module ported to current AIORI API and extended
* Added instructions how to build and run with a HDFS.
* Replaced read with pread to support offsets.
* Implemented HDFS MDTest API
* Improved sync semantics
2020-12-23 11:51:31 +00:00
Julian Kunkel 3e4a0f69d9
Merge pull request #296 from mchaarawi/mschaara/fix_dfs_stat_rc
aiori-DFS: stat should not be fatal
2020-12-16 15:53:43 +00:00