Julian M. Kunkel
f345a78046
Removed dependency.
2021-01-20 15:00:33 +00:00
Julian M. Kunkel
970c5ef139
Adjust complex tests for changed -z behavior.
2021-01-20 14:57:21 +00:00
Julian M. Kunkel
40c6d97e72
Replaced MPI_COMM_WORLD where needed with testComm.
2021-01-20 14:35:06 +00:00
Julian M. Kunkel
e4120d600d
Reduce the dependency to global MPI Communicator
2021-01-20 14:06:05 +00:00
Julian Kunkel
58fbefbd33
Bugfix IOR offsetting (fix before was incomplete). ( #317 )
2021-01-19 23:19:01 +00:00
Julian Kunkel
19ad73568a
Bugfix ior stonewalling case. ( #316 )
...
* Bugfix ior stonewalling case.
2021-01-19 22:28:34 +00:00
Julian Kunkel
3daf7a2d0f
Bugfix: valgrind memory issues ( #314 )
2021-01-19 21:23:30 +00:00
Mohamad Chaarawi
fa316d5d24
dfs: conditionally compile out svcl setting ( #313 )
...
The SVCL argument is being removed from the DAOS API, so conditionally
compile out while maintaining backwards compatibility to versions where
it's still required.
2021-01-19 13:43:01 +00:00
Julian Kunkel
b2089514e3
MDTest calculate min/max/mean across iterations #300 . ( #312 )
...
The calculation per iteration first computes the value of the slowest process, i.e., highest time or lowest rate. This is then the value for the iteration.
Secondly, calculate the min/max/mean across iterations.
For tree operations, the value is identical to previous as only Rank 0 is involved.
2021-01-18 21:30:16 +00:00
Julian Kunkel
33b70ecbc8
Fix MDTest multidir option to work with stonewall file. ( #308 )
2021-01-14 17:38:11 +00:00
Julian Kunkel
4a96436928
MDTest convert FATAL to WARN ( #307 )
2021-01-14 16:41:56 +00:00
Julian Kunkel
d339caa501
Updated test (illustration of an empty test) and distclean for make distcheck #304 ( #305 )
...
Resolves #304
2021-01-04 10:50:38 -08:00
Julian M. Kunkel
21bf5a5a12
NCMPI ported to current ADIO. Fixed autoconf #303 .
2020-12-25 16:49:04 +00:00
Julian Kunkel
8de13884a7
HDFS module ported to current AIORI API and improved ( #302 )
...
* HDFS module ported to current AIORI API and extended
* Added instructions how to build and run with a HDFS.
* Replaced read with pread to support offsets.
* Implemented HDFS MDTest API
* Improved sync semantics
2020-12-23 11:51:31 +00:00
Mohamad Chaarawi
6675cd50bf
aiori-DFS: stat should not be fatal
...
ior now expects stat to not be fatal. update the DFS driver to not
exit if stat fails since the file can simply not exist.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-12-16 15:41:58 +00:00
Frank
0bd52884e8
added region and location to libs3 ( #295 )
...
* added region and location to libs3
2020-12-11 13:05:24 +00:00
Julian Kunkel
fbee4139d8
Merge pull request #292 from ofaaland/b-fix-mpi-checkf
...
Do not execute functions twice in MPI_CHECKF
2020-12-04 09:20:21 +00:00
Julian M. Kunkel
69e006dd89
IOR clarify return API.
2020-12-03 21:07:45 +00:00
Julian M. Kunkel
073da08292
IOR: print overwrite check only if a write operation is requested.
2020-12-03 20:54:51 +00:00
Olaf Faaland
231868505d
Do not execute functions twice in MPI_CHECKF
...
Assigning MPI_STATUS to a local variable and then referring to the
local will ensure that the same value is used in both the conditional
expression and the call to MPI_Error_string.
Otherwise, when MPI_STATUS is a function call, like
MPI_CHECKF(fubar(), "%s", "error in fubar");
fubar() is called twice. If there are underlying intermittent errors,
the error code/message for the first call is lost, with confusing output
like this:
read 2206.18 17.27 145.93 262144 131072
0.272595 291.88 0.290829 292.41
ERROR: cannot access explicit, noncollective, MPI MPI_SUCCESS: no errors, (aiori-MPIIO.c:451)
2020-12-03 12:21:06 -08:00
Julian Kunkel
a8ab766483
Merge pull request #270 from hpc/fix-offset-integration
...
Fix offset integration
2020-12-03 12:08:53 +00:00
Julian Kunkel
09d2c3511b
Merge pull request #289 from hpc/fix-ior-no-warn
...
IOR add warning if file exists #273 . To stop on warnings, use existing flag --warningAsErrors
2020-12-02 10:41:44 +00:00
Julian Kunkel
b417505dfd
Merge branch 'master' into fix-modules
2020-12-02 10:21:59 +00:00
Julian M. Kunkel
f71a144b19
IOR raise warning for existing file once if not filePerProc.
2020-12-02 10:14:28 +00:00
Mohamad Chaarawi
938cf2771b
DFS: make use of collective open to share file handle
...
- remove un-needed libdaos_common lib since DAOS driver is removed
- use default container object class when it's not specified
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-12-01 21:36:25 +00:00
Julian M. Kunkel
da03ae5c9f
IOR add warning if file exists #273 . To stop on warnings, use existing flag --warningAsErrors
2020-12-01 18:24:09 +00:00
Julian Kunkel
691324f17b
Merge pull request #287 from hpc/fix-warning
...
Fix compiler issue with redefinition via typedef.
2020-12-01 17:44:14 +00:00
Julian M. Kunkel
9dcf9f79e4
Moved barriers to right location.
2020-12-01 14:47:57 +00:00
Julian M. Kunkel
546eb05d15
Rudimentary support for stonewall added, addresses #261 .
2020-12-01 14:45:07 +00:00
Julian M. Kunkel
5825dbae35
Provide new option randomPrefill for random (-z access only) that prefill the file with the specified blocksize, e.g., 2m. See issue #270 .
...
This option works so far only without stonewalling!
2020-12-01 13:58:40 +00:00
Richard Mohr
41184c69e2
mdtest: fix reference to unique_dir_per_task for builds --with-lustre
2020-12-01 01:22:31 -05:00
Julian M. Kunkel
ae06908a93
Merge branch 'master' into fix-mdtest-iter
2020-11-30 14:16:41 +00:00
Julian M. Kunkel
11c784c8bd
Integrate review feedback.
2020-11-28 10:40:41 +00:00
Julian M. Kunkel
ae23523a70
Integrate review feedback
2020-11-28 10:34:20 +00:00
Julian M. Kunkel
5799e4ef3a
MDTest remove unnede variable.
2020-11-27 18:02:14 +00:00
Julian M. Kunkel
fd51654393
Readd check for mdtest for now.
2020-11-27 17:51:31 +00:00
Julian M. Kunkel
fbf976351a
MDTest refactoring: move all global static variables into the global static structure.
2020-11-27 17:49:45 +00:00
Julian M. Kunkel
ae8a11b42f
MDTest updated stonewall check.
2020-11-27 15:35:32 +00:00
Julian M. Kunkel
7542e75c82
MDTest ignore verbose potential format overflows to be able to spot real errors.
2020-11-27 15:23:32 +00:00
Julian M. Kunkel
980ab1dc97
Lustre stripping: Fix default value such that Lustre striping is only set if anything is changed.
2020-11-26 15:56:34 +00:00
Julian M. Kunkel
4377aebcf8
Bugfix MDTest calculation of multiple iterations was incorrect.
2020-11-26 12:48:11 +00:00
Julian M. Kunkel
ad6dfc5e63
Allow MDTest to print per proc.
2020-11-25 09:50:26 +00:00
Julian M. Kunkel
02a47085ec
Bugfix: suboptimal name mapping was leading to accidential deletes in md-workbench.
2020-11-22 12:01:12 +00:00
Julian M. Kunkel
b5891141d8
Move checks before inititalization.
...
Add simple validation for S3.
2020-11-22 11:43:42 +00:00
Julian Kunkel
5d489064d8
Merge pull request #271 from hpc/fix-statvfs
...
Remove duplicated functionality between mdtest and IOR.
2020-11-13 09:24:24 +00:00
Julian Kunkel
13f9721dbc
Merge pull request #267 from hpc/feature-md-workbench
...
Feature: md-workbench
2020-11-10 19:05:20 +00:00
Julian M. Kunkel
277f380139
Remove duplicated functionality between mdtest and IOR.
...
Refactored the ShowFileSystemSize function.
2020-11-09 16:23:34 +00:00
Julian M. Kunkel
306598db67
Bugfix memory issue in global options.
2020-11-06 11:40:31 +00:00
Julian M. Kunkel
a59e98d7a6
Add error to indicate changed behavior to users
2020-11-06 10:39:16 +00:00
Julian M. Kunkel
af0753d8ad
Random: last modifications.
2020-11-06 10:27:11 +00:00
Julian M. Kunkel
a0e5e297d6
Create random for one block (1 segment) only to be repeated.
2020-11-06 09:52:35 +00:00
Julian M. Kunkel
ad985af763
Location problem on non Linux systems.
2020-11-06 09:30:59 +00:00
otatebe
0e91769328
aiori-Gfarm: sequel to #262 - sync interface ( #269 )
...
* aiori-Gfarm: update to the new aiori interface
* aiori-Gfarm: Gfarm_sync
2020-11-06 09:15:39 +00:00
Julian M. Kunkel
2d79efc0c5
Fix wait issue on MacOS
2020-11-05 19:13:08 +00:00
Julian M. Kunkel
3ee3e9ad5c
Integrated sequential code into the benchmark main loop.
...
A bit ugly but shows that the code logic hasn't changed.
2020-11-05 14:00:09 +00:00
Julian M. Kunkel
75c08386a4
Bugfix porting error for performance stats.
2020-11-04 20:47:30 +00:00
Julian M. Kunkel
c0ffdf44d0
Workbench API improved.
2020-11-04 20:25:27 +00:00
Julian M. Kunkel
d9c74af8f3
Fix (accidently rename of option)
2020-11-04 19:27:39 +00:00
Julian M. Kunkel
b5dfeea82a
Remove output for the API.
2020-11-04 18:04:46 +00:00
Julian Kunkel
fb66e77072
Enable random seed to be stored. ( #268 )
2020-11-04 13:47:35 +00:00
Julian M. Kunkel
4e452766b9
Enable random seed to be stored.
2020-11-04 12:48:46 +00:00
Julian M. Kunkel
e1dd3103cf
Fix make dist for md-workbench.
2020-11-03 10:52:45 +00:00
Mohamad Chaarawi
cb397242f9
DAOS backend cleanup ( #266 )
...
- remove legacy DAOS driver & update Readme.
- update configure options to remove cart requirement
- Optimize DFS file get_size
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-11-03 10:01:09 +00:00
Julian M. Kunkel
82d20f2744
Basic API converted.
2020-11-02 20:12:15 +00:00
Julian M. Kunkel
654b797788
Converted output to IOR output.
2020-11-02 19:34:15 +00:00
Julian M. Kunkel
dc89a59371
Workbench, adjusting the interface to IOR tool interfaces.
2020-11-02 19:23:13 +00:00
Julian M. Kunkel
d39ae556f0
Bugfixing md-workbench errors.
2020-11-02 19:04:27 +00:00
Julian M. Kunkel
348754c87a
md-workbench code ported.
2020-11-02 18:45:53 +00:00
Julian M. Kunkel
9ace15cce3
Workbench: further conversation.
2020-11-02 17:56:28 +00:00
Julian M. Kunkel
c28ed6dc72
Partial conversion of existing md-workbench to IOR APIs.
2020-11-02 17:34:11 +00:00
Julian M. Kunkel
4898badf48
Fix compiler issue with redefinition via typedef.
2020-10-29 10:38:33 +00:00
Julian Kunkel
d750d323e3
JSON output. Remove duplicated keys. ( #265 )
2020-10-28 09:47:38 +00:00
donko
dab62b57c8
fix JSON format, the end of Array object ( #264 )
2020-10-28 09:38:17 +00:00
Jean-Yves VET
077f4ef98d
aiori-IME: Fix init after finalize issue ( #263 )
2020-10-26 14:37:19 +00:00
otatebe
7bc9680b85
aiori-Gfarm: update to the new aiori interface ( #262 )
2020-10-26 11:26:10 +00:00
Sebastian Oeste
65ddc53950
CSV output for IOR results ( #260 )
...
This commit adds CSV output for IOR.
The output will just include the `Results` section.
All other sections are omitted, since it's hard to produce a useful csv
with all information included.
2020-10-09 09:35:01 +01:00
Julian Kunkel
bd76b45ef9
Integrated comparison checks into CompareData() which prevents additi… ( #254 )
...
* Integrated comparison checks into CompareData() which prevents additional buffer comparisons at the expense of some computation #249 .
* IOR: Added code documentation to warn people for changing the creation pattern but not the verification pattern.
2020-09-28 20:09:48 +01:00
Julian Kunkel
8f14166a79
S3: removed unneded check. ( #258 )
2020-09-13 09:20:46 +01:00
Julian Kunkel
e60801cf8e
Prefix S3 options correctly for S3-libs3 ( #257 )
...
* Option renamed to be consistent
2020-09-13 08:52:02 +01:00
Mohamad Chaarawi
7c201c0d9c
HDF5: fix segfault in HDF5 driver after latest AIORI changes ( #255 )
...
- the hints structure was not getting initialized in MPIIO when HDF5
backend is used. Since HDF5 utilizes the MPIIO backend, this causes
a segfault.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-09-02 18:12:17 +01:00
Julian Kunkel
cc0ac50086
POSIX GFPS adjusted to new API #248 ( #253 )
2020-09-02 10:08:52 +01:00
Julian Kunkel
af2429b47e
Fix s3-4c implementation ( #247 )
...
* Ported S3-4c version to current AIORI interface.
* S3-4c: add crypto dependency
* S3: Store username/host in options.
2020-08-13 16:25:36 +01:00
Julian Kunkel
4e2d179052
Detection of system features for number of sockets #245 ( #246 )
...
* Configure checks for architecture-specific functions to detect nr of sockets. #245
2020-08-03 12:30:21 +01:00
Julian Kunkel
47b63a8054
Merge pull request #243 from hpc/feature-aio
...
Feature aio
2020-08-03 10:06:07 +01:00
Mohamad Chaarawi
db3d06ec80
remove comm from get size cb in DAOS driver
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-07-22 19:39:11 +00:00
Julian M. Kunkel
e3f302ca53
AIORI AIO check params updated.
2020-07-21 16:50:13 +01:00
Julian M. Kunkel
e1741c188a
AIO plugin: missing reference to include.
2020-07-21 16:23:25 +01:00
Julian M. Kunkel
f275671cc9
AIORI POSIX AIO support. Collect ops until granularity is reached, then submit pending IOs. Synchronize latest on close. Doesn't work with data verification and reuses the existing buffer. The implementation shows the potential AIO may have.
...
Extract also the POSIX header from AIORI to allow better reuse. #240
2020-07-21 16:18:50 +01:00
Julian M. Kunkel
5e465ac8bf
Option: merge function to supported AIORI module reuse.
2020-07-21 13:54:50 +01:00
Julian M. Kunkel
4258e14c11
mdtest: Bugfix of changed behavior, can only remove dir if all children were deleted.
2020-07-21 09:31:19 +01:00
Julian M. Kunkel
8ca388fc78
Merge branch 'feature-cleanup-aiori'
2020-07-21 09:26:03 +01:00
Jean-Yves VET
f005147c4d
aiori-IME: Update to new aiori interface
...
This patch updates IME backend to support new aiori interface.
It also fixes some indentation issues.
2020-07-03 16:37:33 +02:00
Jean-Yves VET
49746b99d9
aiori-IME: Update to new aiori interface
...
This patch updates IME backend to support new aiori interface.
It also fixes some indentation issues.
2020-07-03 14:48:17 +02:00
Karsten Weiss
f280123d0b
Spelling fixes (found by codespell)
2020-07-03 09:16:30 +02:00
Julian M. Kunkel
a12ed015a6
S3: better testing, stat is semi-fake, other functions aren't.
2020-07-02 16:40:20 +01:00
Julian M. Kunkel
4f7350dc6a
MDTest remove testdir if created by MDTest.
2020-07-02 16:26:05 +01:00
Julian M. Kunkel
fb9fa7cc9f
S3: Better cleanup.
2020-07-02 15:57:46 +01:00
Julian M. Kunkel
fb8f43fcfe
Reduce verbosity for verification
2020-07-01 10:05:09 +01:00
Julian M. Kunkel
371335195b
Initialize logfile for warnings if not set.
2020-07-01 09:42:13 +01:00
Julian M. Kunkel
484cb420d2
Generate config file with build options
2020-07-01 09:19:10 +01:00