a0e5e297d6
Create random for one block (1 segment) only to be repeated.
2020-11-06 09:52:35 +00:00
ad985af763
Location problem on non Linux systems.
2020-11-06 09:30:59 +00:00
0e91769328
aiori-Gfarm: sequel to #262 - sync interface ( #269 )
...
* aiori-Gfarm: update to the new aiori interface
* aiori-Gfarm: Gfarm_sync
2020-11-06 09:15:39 +00:00
2d79efc0c5
Fix wait issue on MacOS
2020-11-05 19:13:08 +00:00
3ee3e9ad5c
Integrated sequential code into the benchmark main loop.
...
A bit ugly but shows that the code logic hasn't changed.
2020-11-05 14:00:09 +00:00
75c08386a4
Bugfix porting error for performance stats.
2020-11-04 20:47:30 +00:00
c0ffdf44d0
Workbench API improved.
2020-11-04 20:25:27 +00:00
d9c74af8f3
Fix (accidently rename of option)
2020-11-04 19:27:39 +00:00
b5dfeea82a
Remove output for the API.
2020-11-04 18:04:46 +00:00
fb66e77072
Enable random seed to be stored. ( #268 )
2020-11-04 13:47:35 +00:00
4e452766b9
Enable random seed to be stored.
2020-11-04 12:48:46 +00:00
e1dd3103cf
Fix make dist for md-workbench.
2020-11-03 10:52:45 +00:00
cb397242f9
DAOS backend cleanup ( #266 )
...
- remove legacy DAOS driver & update Readme.
- update configure options to remove cart requirement
- Optimize DFS file get_size
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-11-03 10:01:09 +00:00
82d20f2744
Basic API converted.
2020-11-02 20:12:15 +00:00
654b797788
Converted output to IOR output.
2020-11-02 19:34:15 +00:00
dc89a59371
Workbench, adjusting the interface to IOR tool interfaces.
2020-11-02 19:23:13 +00:00
d39ae556f0
Bugfixing md-workbench errors.
2020-11-02 19:04:27 +00:00
348754c87a
md-workbench code ported.
2020-11-02 18:45:53 +00:00
9ace15cce3
Workbench: further conversation.
2020-11-02 17:56:28 +00:00
c28ed6dc72
Partial conversion of existing md-workbench to IOR APIs.
2020-11-02 17:34:11 +00:00
4898badf48
Fix compiler issue with redefinition via typedef.
2020-10-29 10:38:33 +00:00
d750d323e3
JSON output. Remove duplicated keys. ( #265 )
2020-10-28 09:47:38 +00:00
dab62b57c8
fix JSON format, the end of Array object ( #264 )
2020-10-28 09:38:17 +00:00
077f4ef98d
aiori-IME: Fix init after finalize issue ( #263 )
2020-10-26 14:37:19 +00:00
7bc9680b85
aiori-Gfarm: update to the new aiori interface ( #262 )
2020-10-26 11:26:10 +00:00
65ddc53950
CSV output for IOR results ( #260 )
...
This commit adds CSV output for IOR.
The output will just include the `Results` section.
All other sections are omitted, since it's hard to produce a useful csv
with all information included.
2020-10-09 09:35:01 +01:00
bd76b45ef9
Integrated comparison checks into CompareData() which prevents additi… ( #254 )
...
* Integrated comparison checks into CompareData() which prevents additional buffer comparisons at the expense of some computation #249 .
* IOR: Added code documentation to warn people for changing the creation pattern but not the verification pattern.
2020-09-28 20:09:48 +01:00
8f14166a79
S3: removed unneded check. ( #258 )
2020-09-13 09:20:46 +01:00
e60801cf8e
Prefix S3 options correctly for S3-libs3 ( #257 )
...
* Option renamed to be consistent
2020-09-13 08:52:02 +01:00
7c201c0d9c
HDF5: fix segfault in HDF5 driver after latest AIORI changes ( #255 )
...
- the hints structure was not getting initialized in MPIIO when HDF5
backend is used. Since HDF5 utilizes the MPIIO backend, this causes
a segfault.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-09-02 18:12:17 +01:00
cc0ac50086
POSIX GFPS adjusted to new API #248 ( #253 )
2020-09-02 10:08:52 +01:00
af2429b47e
Fix s3-4c implementation ( #247 )
...
* Ported S3-4c version to current AIORI interface.
* S3-4c: add crypto dependency
* S3: Store username/host in options.
2020-08-13 16:25:36 +01:00
4e2d179052
Detection of system features for number of sockets #245 ( #246 )
...
* Configure checks for architecture-specific functions to detect nr of sockets. #245
2020-08-03 12:30:21 +01:00
47b63a8054
Merge pull request #243 from hpc/feature-aio
...
Feature aio
2020-08-03 10:06:07 +01:00
db3d06ec80
remove comm from get size cb in DAOS driver
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-07-22 19:39:11 +00:00
e3f302ca53
AIORI AIO check params updated.
2020-07-21 16:50:13 +01:00
e1741c188a
AIO plugin: missing reference to include.
2020-07-21 16:23:25 +01:00
f275671cc9
AIORI POSIX AIO support. Collect ops until granularity is reached, then submit pending IOs. Synchronize latest on close. Doesn't work with data verification and reuses the existing buffer. The implementation shows the potential AIO may have.
...
Extract also the POSIX header from AIORI to allow better reuse. #240
2020-07-21 16:18:50 +01:00
5e465ac8bf
Option: merge function to supported AIORI module reuse.
2020-07-21 13:54:50 +01:00
4258e14c11
mdtest: Bugfix of changed behavior, can only remove dir if all children were deleted.
2020-07-21 09:31:19 +01:00
8ca388fc78
Merge branch 'feature-cleanup-aiori'
2020-07-21 09:26:03 +01:00
f005147c4d
aiori-IME: Update to new aiori interface
...
This patch updates IME backend to support new aiori interface.
It also fixes some indentation issues.
2020-07-03 16:37:33 +02:00
49746b99d9
aiori-IME: Update to new aiori interface
...
This patch updates IME backend to support new aiori interface.
It also fixes some indentation issues.
2020-07-03 14:48:17 +02:00
f280123d0b
Spelling fixes (found by codespell)
2020-07-03 09:16:30 +02:00
a12ed015a6
S3: better testing, stat is semi-fake, other functions aren't.
2020-07-02 16:40:20 +01:00
4f7350dc6a
MDTest remove testdir if created by MDTest.
2020-07-02 16:26:05 +01:00
fb9fa7cc9f
S3: Better cleanup.
2020-07-02 15:57:46 +01:00
fb8f43fcfe
Reduce verbosity for verification
2020-07-01 10:05:09 +01:00
371335195b
Initialize logfile for warnings if not set.
2020-07-01 09:42:13 +01:00
484cb420d2
Generate config file with build options
2020-07-01 09:19:10 +01:00
fcae6ff0b9
S3: Fine tuning, supporting bucket per file/dir and single bucket.
2020-06-30 20:02:36 +01:00
87c9906d70
Error checking for NULL create/opens
2020-06-30 19:41:49 +01:00
154cf2cde7
LibS3 version that stores fragments as one object each.
2020-06-30 18:43:44 +01:00
2a3838c360
S3: Partial implementation.
2020-06-30 16:50:13 +01:00
81a7a3ab81
Renamed existing S3 impl. which uses (apparently) outdated library. Added dummy support for libS3 with library search.
2020-06-30 13:41:59 +01:00
3a9bd7828d
Ported PMDK API.
2020-06-30 13:12:44 +01:00
82417128cd
Extracted file size routine that is replicated in each module into IOR removing the MPI dependency from AIORI.
2020-06-30 12:33:56 +01:00
3395fc621c
Moved GetFileSize into CheckFileSize, starting to simplify get_file_size (too much replication in modules).
2020-06-30 12:03:05 +01:00
5a5b90cdaa
Run existing check to allow only supported backends in mdtest.
2020-06-30 11:17:46 +01:00
3e6bfd2db7
Added missing header info.
2020-06-29 20:58:45 +01:00
03dbb20594
MDTest changed verification pattern. Read now always checks the first byte/8 bytes for the signature (item number).
...
Added also --verify-write option which performs a read immediately after a write. Supports #206
2020-06-29 20:15:14 +01:00
df3f7082d8
MDTest: Added warnings/errors for errorneous read pattern #206
2020-06-28 17:21:24 +01:00
145c71f7c3
Trivial cleanup: Extracted debug-related stuff into new header.
2020-06-28 17:16:56 +01:00
38b1752bda
JSON issue with results. Fix for #233 .
2020-06-25 14:37:52 +01:00
0bffd14de7
Added --warningAsErrors option to IOR and MDTest and refactored WARNINGs in IOR. #174
2020-06-24 11:13:12 +01:00
c55848f14d
Bugfix based on the pull request for fix-189. Should fix #189 .
2020-06-24 10:15:31 +01:00
d5de9bcc77
Merge branch 'master' into master
2020-06-24 09:50:03 +01:00
6a88b37171
Merge branch 'master' into fix-194
2020-06-24 09:43:37 +01:00
b4a54c34e8
Merge branch 'master' into fix-modules
...
Conflicts:
src/aiori-DAOS.c
src/aiori-DFS.c
2020-06-11 19:55:23 +00:00
e08314f571
update DFS and DAOS drivers to new aiori module interface.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-06-11 17:26:58 +00:00
eb2cd27fac
Removed offset from IOR structure.
2020-06-10 17:47:07 +01:00
c828a0f32c
mdtest: use correct number of bytes for posix_memalign of read buffer
...
Fixes a typo in initial posix_memalign commit for DirectIO support.
2020-06-03 01:22:44 +03:00
3eb488939d
mdtest: allocate aligned buffers to support DirectIO
2020-06-03 00:30:38 +03:00
3623f3b8bd
DFS: allow multiple IO phases when ior_run() is called
...
- init/fini can be made multiple times
- reset tunables on fini so they can change on a re-init
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-05-31 16:20:58 +00:00
8fa8ef0c02
Call backend statfs() instead of only POSIX-specific variant.
2020-05-31 13:30:31 +01:00
5663593919
Moved generic IOR information to "hint" structure. Backends should only use the "hints" if set.
2020-05-31 12:50:03 +01:00
8098c3740a
Refactored fd into aiori_fd_t for type safety.
2020-05-31 12:12:49 +01:00
c4c19d3245
Fixed type typo in name.
2020-05-31 11:58:34 +01:00
294b8891e6
Created dummy option type to provide type safety. Fixed various call issues for options.
2020-05-31 11:50:15 +01:00
e91b79cea0
Moved Lustre/GPFS/BeeGFS options into POSIX backend. Needs testing. Additional minor fixes.
2020-05-30 20:09:37 +01:00
930ccdc68d
Updated HDF5 to new module specification.
2020-05-30 19:01:20 +01:00
aa9e2ad815
MDTest updated for AIORI changes.
2020-05-30 18:30:26 +01:00
1890aaaeb0
Started to move IOR specific knowledge out of AIORI.
...
For now, and compatibility, IOR options can still be set/internally accessed using the backends init_xfer_options.
This should be removed in the long run to strip away this dependency.
2020-05-30 18:19:48 +01:00
2de4210311
Provide a string option parser function.
2020-05-29 16:05:42 +01:00
edba70e99c
aiori-CEPHFS: don't just unmount on close, release cmount too.
...
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-05-28 20:42:26 -05:00
efc6d49832
src/aiori-CEPHFS: Add option checks before init
...
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-05-28 19:52:45 -05:00
8abb7a62dc
src/aiori-CEPHFS: Add sync
...
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-05-28 22:28:16 +00:00
153b7aa60f
Critical bugfix for ior API (used e.g. in IO500)
...
TODO: Check potential issue when using IOR scripts with different backends.
2020-05-28 21:18:44 +01:00
4eb0ebec55
Add dummy sync to dummy backend.
2020-05-28 21:13:45 +01:00
8c92a1b2ff
Merge branch 'master' of github.com:hpc/ior
2020-05-28 20:12:06 +01:00
f97090a86b
Typo fix.
2020-05-28 20:11:52 +01:00
28fae361f4
dfs: add DFS sync call
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-05-28 18:34:31 +00:00
4b417ef1f3
IOR shouldn't fail if it cannot determine file system sizes but warn. Functionality should presumably be moved into AIORI backends.
2020-05-28 19:05:41 +01:00
a536649abb
Clarified some issues in the test framework and added example test. #219
2020-05-20 17:46:35 +01:00
48eb1880e9
For better testing: Fixed fake tasks per node to work with MPI3, added environment variable to fake num nodes.
2020-04-21 10:35:53 +01:00
8bc9a12f80
Merge pull request #221 from daos-stack/mschaara/dfs_l2g
...
DFS driver update to share DFS mount on init
2020-04-14 10:13:55 +01:00
1b8848ccbc
Use l2g-g2l to share the dfs mount instead of opening by each rank.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-04-13 19:59:56 +00:00
a21137be3e
Display outlier host names
...
Display host names when using the -j argument (outlierThreshold) so
that IOR could help to better identify slow nodes:
WARNING: for client02, task 6, write elapsed transfer time is 65.738326
2020-03-17 16:31:56 +01:00
657ff8ad8f
Merge pull request #217 from markhpc/wip-aiori-cephfs
...
[RFC] src/aiori-CEPHFS: New libcephfs backend
2020-03-10 12:29:24 -07:00
9649a0c520
src/aiori-CEPHFS: New libcephfs backend
...
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-03-10 10:19:41 -05:00
9726aa0ce2
Bugfix API which doesn't set the AIORI backend.
2020-03-07 14:40:59 +00:00
f5ecb9d974
Merge pull request #215 from daos-stack/daos-devel
...
DAOS Driver: Update License & Align with 1.0 API changes
2020-02-26 19:13:48 +00:00
50a4ba0735
DAOS Driver: Update License & Align with 1.0 API changes
...
- License should inherit from IOR License.
- remove checksums from Array API
- add DFS dir oclass option
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-02-26 18:53:51 +00:00
ec8d246b33
Merge pull request #214 from markhpc/wip-ior-rados-mark
...
src/aiori-RADOS: Remove Direct IO option check.
2020-02-26 09:06:54 +00:00
e363713c07
src/aiori-RADOS: Remove Direct IO option check.
...
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-02-25 18:13:29 -06:00
c702a98376
Strict prototypes for AIORI
2019-12-22 11:21:40 +00:00
6de5cdc6f9
Fixed wrong usage of platform, fixed printf output.
2019-12-21 14:14:27 +00:00
240d5fae79
Trivial fix to ensure all functions are prototypes.
2019-12-21 11:19:02 +00:00
73c5c0efc2
Fix for collective operations by Wolfgang Szoecs #210 .
...
It does have performance implications but is only active if needed.
2019-12-19 13:43:38 +00:00
0666875df3
aiori-IME: Add support of sync and mknod in IME backend
...
This patch adds the support of sync (flush client page cache
for all opened files) and mknod in the IME backend.
2019-12-17 14:51:21 +01:00
7a2d307c74
Merge pull request #208 from jyvet/mknod
...
Fix confusing output of mknod
2019-12-16 12:38:13 -08:00
9da36abaca
Fix confusing output of mknod
...
Context: mknod does not return a file descriptor.
2019-12-16 14:55:26 +01:00
be48206e5f
Fix compilation warnings
2019-12-16 14:51:10 +01:00
3a34b2efb7
Merge pull request #200 from daos-stack/daos-devel
...
DAOS driver update
2019-11-13 10:38:28 +00:00
acfd4673cc
We only need to add dirs to the Hash in the DFS driver.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-11-13 00:09:03 +00:00
ddac5c8dbf
Merge pull request #3 from hpc/master
...
Bring fork up to date
2019-11-12 14:20:53 +00:00
47199e28dd
Not sure why iops and latency are left out of the JSON job output. It is really useful information.
...
Signed-off by: Robert LeBlanc <robert@leblancnet.us>
2019-11-11 23:16:01 +00:00
9c0926ef2a
Moving fsync pmdk functionalty from persist to drain on the advice of the PMDK developers
2019-11-07 20:17:55 +00:00
fbff9b6fb2
add DFS prefix option
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-11-07 18:01:07 +00:00
c24b2f156a
Merge pull request #2 from hpc/master
...
Merging in latest master changes
2019-11-05 15:00:47 +00:00
b4c85798bd
Tidying up PMDK backend
2019-11-05 14:39:31 +00:00
46c5d4e78d
Adding in dual mount functionality for NVRAM within nodes
2019-11-05 14:37:54 +00:00
817603f5d1
Updating copyright statement
2019-10-31 15:31:28 +00:00
fa25c31994
Adding PMDK functionality
2019-10-31 15:29:09 +00:00
3c7ac2df23
Merge pull request #196 from daos-stack/daos-devel
...
update DFS driver with new DAOS API changes.
2019-10-28 16:56:29 -07:00
4409082064
Merge pull request #188 from pkoutoupis/feature-global_default_dir_layout
...
Enable global default dir layout for subdirs in Lustre
2019-10-28 10:43:56 -07:00
92180e022d
update DFS read/write APIs with new DAOS API changes
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-10-28 16:42:44 +00:00
ccba3b11e5
Merge branch 'master' into fix-194
2019-10-23 16:46:00 -06:00
61333af822
Fix #194 .
...
On previous versions of IOR (and most Linux CLI applications in general)
multiple flags can be combined such as -vvv to get verbose=3 rather than
having to specify each separately as -v -v -v. This patch fixes this behavior.
It also fixes issues around handling of the verbose flag where the
output would never be printed as we were comparing verbose < VERBOSE_0
where verbose defaults to 0 and can only increase.
2019-10-23 16:17:37 -06:00
b334c3f6fd
Merge pull request #192 from daos-stack/daos-devel
...
fix -R option to do proper data verification
2019-10-16 16:20:43 +01:00
dc82a1bf2f
ior -R should memset the buffer being read, otherwise a read that does nothing
...
will report success in the data verification phase.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-10-16 14:54:00 +00:00
a69a5916cf
Hotfix for ior -F -u
2019-10-16 09:42:10 +01:00
f40b0744ff
Merge pull request #187 from daos-stack/daos-devel
...
IOR: Add latency and IOPS numbers to each iteration.
2019-10-02 13:09:26 -07:00
b3ecd756ce
Implemented AC_CHECK_HEADERS fix in style of #190
2019-09-30 04:25:59 -07:00
2aea04c8cb
fix some compiler warnings.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-21 22:43:20 +00:00
fca0a62ccb
update to new DAOS API for creating dfs containers.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-21 22:05:54 +00:00
604598ab2f
Merge pull request #182 from jschwartz-cray/fix-181
...
Fix #181 .
2019-09-19 16:53:41 +01:00
12284ae04a
Add latency and iops numbers to each iteration.
...
- Latency reported is computed by taking the average latency of
all ops from a single task, then taking the minimum of that
between all tasks.
- IOPS is computed by taking the total number of ops across all
tasks divided by the total access time to execute those ops.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-10 18:49:32 +00:00
c58ba8ffb5
add break that was accidentally removed
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-10 14:11:55 +00:00
fe9d76ddf3
fix segfault when no API is specified to mdtest
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 22:03:55 +00:00
73dbda09c6
remove printf added
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 20:19:42 +00:00
5622aabf46
missed Makefile.am update
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 20:14:12 +00:00
d332d586bf
remove uneeded code and doc
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 20:07:55 +00:00
e9dd5fe15e
Merge branch 'master' into daos-devel
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
Conflicts:
configure.ac
src/Makefile.am
src/aiori-MPIIO.c
src/aiori.c
src/aiori.h
src/ior.c
src/mdtest.c
src/option.c
2019-09-09 19:55:56 +00:00
57a16ddda8
- update debugging error checking in DFS.
...
- remove usleep() before daos_fini()
- fix README
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 19:14:35 +00:00
0921556456
Merge pull request #184 from hpc/feature-aiori
...
Feature aiori
2019-09-09 15:20:43 +01:00
36d13d3b97
Enable global default dir layout for subdirs in Lustre
2019-09-02 22:26:12 -07:00
c83edfe39b
Extracted check function into aiori. #24 . #177
2019-09-01 15:59:52 +01:00
ced9b0183f
Merge pull request #172 from hpc/fix-168
...
Fix #168
2019-09-01 15:51:44 +01:00
151102ff8f
Merge pull request #176 from otatebe/master
...
abstract I/O interface for Gfarm file system
2019-09-01 15:51:16 +01:00
Julian M. Kunkel