Julian M. Kunkel
306598db67
Bugfix memory issue in global options.
2020-11-06 11:40:31 +00:00
Julian M. Kunkel
a59e98d7a6
Add error to indicate changed behavior to users
2020-11-06 10:39:16 +00:00
Julian M. Kunkel
af0753d8ad
Random: last modifications.
2020-11-06 10:27:11 +00:00
Julian M. Kunkel
a0e5e297d6
Create random for one block (1 segment) only to be repeated.
2020-11-06 09:52:35 +00:00
Julian M. Kunkel
ad985af763
Location problem on non Linux systems.
2020-11-06 09:30:59 +00:00
otatebe
0e91769328
aiori-Gfarm: sequel to #262 - sync interface ( #269 )
...
* aiori-Gfarm: update to the new aiori interface
* aiori-Gfarm: Gfarm_sync
2020-11-06 09:15:39 +00:00
Julian M. Kunkel
2d79efc0c5
Fix wait issue on MacOS
2020-11-05 19:13:08 +00:00
Julian M. Kunkel
3ee3e9ad5c
Integrated sequential code into the benchmark main loop.
...
A bit ugly but shows that the code logic hasn't changed.
2020-11-05 14:00:09 +00:00
Julian M. Kunkel
75c08386a4
Bugfix porting error for performance stats.
2020-11-04 20:47:30 +00:00
Julian M. Kunkel
c0ffdf44d0
Workbench API improved.
2020-11-04 20:25:27 +00:00
Julian M. Kunkel
d9c74af8f3
Fix (accidently rename of option)
2020-11-04 19:27:39 +00:00
Julian M. Kunkel
b5dfeea82a
Remove output for the API.
2020-11-04 18:04:46 +00:00
Julian Kunkel
fb66e77072
Enable random seed to be stored. ( #268 )
2020-11-04 13:47:35 +00:00
Julian M. Kunkel
4e452766b9
Enable random seed to be stored.
2020-11-04 12:48:46 +00:00
Julian M. Kunkel
e1dd3103cf
Fix make dist for md-workbench.
2020-11-03 10:52:45 +00:00
Mohamad Chaarawi
cb397242f9
DAOS backend cleanup ( #266 )
...
- remove legacy DAOS driver & update Readme.
- update configure options to remove cart requirement
- Optimize DFS file get_size
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-11-03 10:01:09 +00:00
Julian M. Kunkel
82d20f2744
Basic API converted.
2020-11-02 20:12:15 +00:00
Julian M. Kunkel
654b797788
Converted output to IOR output.
2020-11-02 19:34:15 +00:00
Julian M. Kunkel
dc89a59371
Workbench, adjusting the interface to IOR tool interfaces.
2020-11-02 19:23:13 +00:00
Julian M. Kunkel
d39ae556f0
Bugfixing md-workbench errors.
2020-11-02 19:04:27 +00:00
Julian M. Kunkel
348754c87a
md-workbench code ported.
2020-11-02 18:45:53 +00:00
Julian M. Kunkel
9ace15cce3
Workbench: further conversation.
2020-11-02 17:56:28 +00:00
Julian M. Kunkel
c28ed6dc72
Partial conversion of existing md-workbench to IOR APIs.
2020-11-02 17:34:11 +00:00
Julian M. Kunkel
4898badf48
Fix compiler issue with redefinition via typedef.
2020-10-29 10:38:33 +00:00
Julian Kunkel
d750d323e3
JSON output. Remove duplicated keys. ( #265 )
2020-10-28 09:47:38 +00:00
donko
dab62b57c8
fix JSON format, the end of Array object ( #264 )
2020-10-28 09:38:17 +00:00
Jean-Yves VET
077f4ef98d
aiori-IME: Fix init after finalize issue ( #263 )
2020-10-26 14:37:19 +00:00
otatebe
7bc9680b85
aiori-Gfarm: update to the new aiori interface ( #262 )
2020-10-26 11:26:10 +00:00
Sebastian Oeste
65ddc53950
CSV output for IOR results ( #260 )
...
This commit adds CSV output for IOR.
The output will just include the `Results` section.
All other sections are omitted, since it's hard to produce a useful csv
with all information included.
2020-10-09 09:35:01 +01:00
Julian Kunkel
bd76b45ef9
Integrated comparison checks into CompareData() which prevents additi… ( #254 )
...
* Integrated comparison checks into CompareData() which prevents additional buffer comparisons at the expense of some computation #249 .
* IOR: Added code documentation to warn people for changing the creation pattern but not the verification pattern.
2020-09-28 20:09:48 +01:00
Julian Kunkel
8f14166a79
S3: removed unneded check. ( #258 )
2020-09-13 09:20:46 +01:00
Julian Kunkel
e60801cf8e
Prefix S3 options correctly for S3-libs3 ( #257 )
...
* Option renamed to be consistent
2020-09-13 08:52:02 +01:00
Mohamad Chaarawi
7c201c0d9c
HDF5: fix segfault in HDF5 driver after latest AIORI changes ( #255 )
...
- the hints structure was not getting initialized in MPIIO when HDF5
backend is used. Since HDF5 utilizes the MPIIO backend, this causes
a segfault.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-09-02 18:12:17 +01:00
Julian Kunkel
cc0ac50086
POSIX GFPS adjusted to new API #248 ( #253 )
2020-09-02 10:08:52 +01:00
Julian Kunkel
af2429b47e
Fix s3-4c implementation ( #247 )
...
* Ported S3-4c version to current AIORI interface.
* S3-4c: add crypto dependency
* S3: Store username/host in options.
2020-08-13 16:25:36 +01:00
Julian Kunkel
4e2d179052
Detection of system features for number of sockets #245 ( #246 )
...
* Configure checks for architecture-specific functions to detect nr of sockets. #245
2020-08-03 12:30:21 +01:00
Julian Kunkel
47b63a8054
Merge pull request #243 from hpc/feature-aio
...
Feature aio
2020-08-03 10:06:07 +01:00
Mohamad Chaarawi
db3d06ec80
remove comm from get size cb in DAOS driver
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-07-22 19:39:11 +00:00
Julian M. Kunkel
e3f302ca53
AIORI AIO check params updated.
2020-07-21 16:50:13 +01:00
Julian M. Kunkel
e1741c188a
AIO plugin: missing reference to include.
2020-07-21 16:23:25 +01:00
Julian M. Kunkel
f275671cc9
AIORI POSIX AIO support. Collect ops until granularity is reached, then submit pending IOs. Synchronize latest on close. Doesn't work with data verification and reuses the existing buffer. The implementation shows the potential AIO may have.
...
Extract also the POSIX header from AIORI to allow better reuse. #240
2020-07-21 16:18:50 +01:00
Julian M. Kunkel
5e465ac8bf
Option: merge function to supported AIORI module reuse.
2020-07-21 13:54:50 +01:00
Julian M. Kunkel
4258e14c11
mdtest: Bugfix of changed behavior, can only remove dir if all children were deleted.
2020-07-21 09:31:19 +01:00
Julian M. Kunkel
8ca388fc78
Merge branch 'feature-cleanup-aiori'
2020-07-21 09:26:03 +01:00
Jean-Yves VET
f005147c4d
aiori-IME: Update to new aiori interface
...
This patch updates IME backend to support new aiori interface.
It also fixes some indentation issues.
2020-07-03 16:37:33 +02:00
Jean-Yves VET
49746b99d9
aiori-IME: Update to new aiori interface
...
This patch updates IME backend to support new aiori interface.
It also fixes some indentation issues.
2020-07-03 14:48:17 +02:00
Karsten Weiss
f280123d0b
Spelling fixes (found by codespell)
2020-07-03 09:16:30 +02:00
Julian M. Kunkel
a12ed015a6
S3: better testing, stat is semi-fake, other functions aren't.
2020-07-02 16:40:20 +01:00
Julian M. Kunkel
4f7350dc6a
MDTest remove testdir if created by MDTest.
2020-07-02 16:26:05 +01:00
Julian M. Kunkel
fb9fa7cc9f
S3: Better cleanup.
2020-07-02 15:57:46 +01:00
Julian M. Kunkel
fb8f43fcfe
Reduce verbosity for verification
2020-07-01 10:05:09 +01:00
Julian M. Kunkel
371335195b
Initialize logfile for warnings if not set.
2020-07-01 09:42:13 +01:00
Julian M. Kunkel
484cb420d2
Generate config file with build options
2020-07-01 09:19:10 +01:00
Julian M. Kunkel
fcae6ff0b9
S3: Fine tuning, supporting bucket per file/dir and single bucket.
2020-06-30 20:02:36 +01:00
Julian M. Kunkel
87c9906d70
Error checking for NULL create/opens
2020-06-30 19:41:49 +01:00
Julian M. Kunkel
154cf2cde7
LibS3 version that stores fragments as one object each.
2020-06-30 18:43:44 +01:00
Julian M. Kunkel
2a3838c360
S3: Partial implementation.
2020-06-30 16:50:13 +01:00
Julian M. Kunkel
81a7a3ab81
Renamed existing S3 impl. which uses (apparently) outdated library. Added dummy support for libS3 with library search.
2020-06-30 13:41:59 +01:00
Julian M. Kunkel
3a9bd7828d
Ported PMDK API.
2020-06-30 13:12:44 +01:00
Julian M. Kunkel
82417128cd
Extracted file size routine that is replicated in each module into IOR removing the MPI dependency from AIORI.
2020-06-30 12:33:56 +01:00
Julian M. Kunkel
3395fc621c
Moved GetFileSize into CheckFileSize, starting to simplify get_file_size (too much replication in modules).
2020-06-30 12:03:05 +01:00
Julian M. Kunkel
5a5b90cdaa
Run existing check to allow only supported backends in mdtest.
2020-06-30 11:17:46 +01:00
Julian M. Kunkel
3e6bfd2db7
Added missing header info.
2020-06-29 20:58:45 +01:00
Julian M. Kunkel
03dbb20594
MDTest changed verification pattern. Read now always checks the first byte/8 bytes for the signature (item number).
...
Added also --verify-write option which performs a read immediately after a write. Supports #206
2020-06-29 20:15:14 +01:00
Julian M. Kunkel
df3f7082d8
MDTest: Added warnings/errors for errorneous read pattern #206
2020-06-28 17:21:24 +01:00
Julian M. Kunkel
145c71f7c3
Trivial cleanup: Extracted debug-related stuff into new header.
2020-06-28 17:16:56 +01:00
Julian M. Kunkel
38b1752bda
JSON issue with results. Fix for #233 .
2020-06-25 14:37:52 +01:00
Julian M. Kunkel
0bffd14de7
Added --warningAsErrors option to IOR and MDTest and refactored WARNINGs in IOR. #174
2020-06-24 11:13:12 +01:00
Julian M. Kunkel
c55848f14d
Bugfix based on the pull request for fix-189. Should fix #189 .
2020-06-24 10:15:31 +01:00
Julian Kunkel
d5de9bcc77
Merge branch 'master' into master
2020-06-24 09:50:03 +01:00
Julian Kunkel
6a88b37171
Merge branch 'master' into fix-194
2020-06-24 09:43:37 +01:00
Mohamad Chaarawi
b4a54c34e8
Merge branch 'master' into fix-modules
...
Conflicts:
src/aiori-DAOS.c
src/aiori-DFS.c
2020-06-11 19:55:23 +00:00
Mohamad Chaarawi
e08314f571
update DFS and DAOS drivers to new aiori module interface.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-06-11 17:26:58 +00:00
Julian M. Kunkel
eb2cd27fac
Removed offset from IOR structure.
2020-06-10 17:47:07 +01:00
Sven Breuner
c828a0f32c
mdtest: use correct number of bytes for posix_memalign of read buffer
...
Fixes a typo in initial posix_memalign commit for DirectIO support.
2020-06-03 01:22:44 +03:00
Sven Breuner
3eb488939d
mdtest: allocate aligned buffers to support DirectIO
2020-06-03 00:30:38 +03:00
Mohamad Chaarawi
3623f3b8bd
DFS: allow multiple IO phases when ior_run() is called
...
- init/fini can be made multiple times
- reset tunables on fini so they can change on a re-init
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-05-31 16:20:58 +00:00
Julian M. Kunkel
8fa8ef0c02
Call backend statfs() instead of only POSIX-specific variant.
2020-05-31 13:30:31 +01:00
Julian M. Kunkel
5663593919
Moved generic IOR information to "hint" structure. Backends should only use the "hints" if set.
2020-05-31 12:50:03 +01:00
Julian M. Kunkel
8098c3740a
Refactored fd into aiori_fd_t for type safety.
2020-05-31 12:12:49 +01:00
Julian M. Kunkel
c4c19d3245
Fixed type typo in name.
2020-05-31 11:58:34 +01:00
Julian M. Kunkel
294b8891e6
Created dummy option type to provide type safety. Fixed various call issues for options.
2020-05-31 11:50:15 +01:00
Julian M. Kunkel
e91b79cea0
Moved Lustre/GPFS/BeeGFS options into POSIX backend. Needs testing. Additional minor fixes.
2020-05-30 20:09:37 +01:00
Julian M. Kunkel
930ccdc68d
Updated HDF5 to new module specification.
2020-05-30 19:01:20 +01:00
Julian M. Kunkel
aa9e2ad815
MDTest updated for AIORI changes.
2020-05-30 18:30:26 +01:00
Julian M. Kunkel
1890aaaeb0
Started to move IOR specific knowledge out of AIORI.
...
For now, and compatibility, IOR options can still be set/internally accessed using the backends init_xfer_options.
This should be removed in the long run to strip away this dependency.
2020-05-30 18:19:48 +01:00
Julian M. Kunkel
2de4210311
Provide a string option parser function.
2020-05-29 16:05:42 +01:00
Mark Nelson
edba70e99c
aiori-CEPHFS: don't just unmount on close, release cmount too.
...
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-05-28 20:42:26 -05:00
Mark Nelson
efc6d49832
src/aiori-CEPHFS: Add option checks before init
...
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-05-28 19:52:45 -05:00
Mark Nelson
8abb7a62dc
src/aiori-CEPHFS: Add sync
...
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-05-28 22:28:16 +00:00
Julian M. Kunkel
153b7aa60f
Critical bugfix for ior API (used e.g. in IO500)
...
TODO: Check potential issue when using IOR scripts with different backends.
2020-05-28 21:18:44 +01:00
Julian M. Kunkel
4eb0ebec55
Add dummy sync to dummy backend.
2020-05-28 21:13:45 +01:00
Julian M. Kunkel
8c92a1b2ff
Merge branch 'master' of github.com:hpc/ior
2020-05-28 20:12:06 +01:00
Julian M. Kunkel
f97090a86b
Typo fix.
2020-05-28 20:11:52 +01:00
Mohamad Chaarawi
28fae361f4
dfs: add DFS sync call
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-05-28 18:34:31 +00:00
Julian M. Kunkel
4b417ef1f3
IOR shouldn't fail if it cannot determine file system sizes but warn. Functionality should presumably be moved into AIORI backends.
2020-05-28 19:05:41 +01:00
Julian M. Kunkel
a536649abb
Clarified some issues in the test framework and added example test. #219
2020-05-20 17:46:35 +01:00
Julian M. Kunkel
48eb1880e9
For better testing: Fixed fake tasks per node to work with MPI3, added environment variable to fake num nodes.
2020-04-21 10:35:53 +01:00
Julian Kunkel
8bc9a12f80
Merge pull request #221 from daos-stack/mschaara/dfs_l2g
...
DFS driver update to share DFS mount on init
2020-04-14 10:13:55 +01:00
Mohamad Chaarawi
1b8848ccbc
Use l2g-g2l to share the dfs mount instead of opening by each rank.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-04-13 19:59:56 +00:00