9726aa0ce2
Bugfix API which doesn't set the AIORI backend.
2020-03-07 14:40:59 +00:00
f5ecb9d974
Merge pull request #215 from daos-stack/daos-devel
...
DAOS Driver: Update License & Align with 1.0 API changes
2020-02-26 19:13:48 +00:00
50a4ba0735
DAOS Driver: Update License & Align with 1.0 API changes
...
- License should inherit from IOR License.
- remove checksums from Array API
- add DFS dir oclass option
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-02-26 18:53:51 +00:00
ec8d246b33
Merge pull request #214 from markhpc/wip-ior-rados-mark
...
src/aiori-RADOS: Remove Direct IO option check.
2020-02-26 09:06:54 +00:00
e363713c07
src/aiori-RADOS: Remove Direct IO option check.
...
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-02-25 18:13:29 -06:00
c702a98376
Strict prototypes for AIORI
2019-12-22 11:21:40 +00:00
6de5cdc6f9
Fixed wrong usage of platform, fixed printf output.
2019-12-21 14:14:27 +00:00
240d5fae79
Trivial fix to ensure all functions are prototypes.
2019-12-21 11:19:02 +00:00
73c5c0efc2
Fix for collective operations by Wolfgang Szoecs #210 .
...
It does have performance implications but is only active if needed.
2019-12-19 13:43:38 +00:00
0666875df3
aiori-IME: Add support of sync and mknod in IME backend
...
This patch adds the support of sync (flush client page cache
for all opened files) and mknod in the IME backend.
2019-12-17 14:51:21 +01:00
7a2d307c74
Merge pull request #208 from jyvet/mknod
...
Fix confusing output of mknod
2019-12-16 12:38:13 -08:00
9da36abaca
Fix confusing output of mknod
...
Context: mknod does not return a file descriptor.
2019-12-16 14:55:26 +01:00
be48206e5f
Fix compilation warnings
2019-12-16 14:51:10 +01:00
3a34b2efb7
Merge pull request #200 from daos-stack/daos-devel
...
DAOS driver update
2019-11-13 10:38:28 +00:00
acfd4673cc
We only need to add dirs to the Hash in the DFS driver.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-11-13 00:09:03 +00:00
ddac5c8dbf
Merge pull request #3 from hpc/master
...
Bring fork up to date
2019-11-12 14:20:53 +00:00
47199e28dd
Not sure why iops and latency are left out of the JSON job output. It is really useful information.
...
Signed-off by: Robert LeBlanc <robert@leblancnet.us>
2019-11-11 23:16:01 +00:00
9c0926ef2a
Moving fsync pmdk functionalty from persist to drain on the advice of the PMDK developers
2019-11-07 20:17:55 +00:00
fbff9b6fb2
add DFS prefix option
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-11-07 18:01:07 +00:00
c24b2f156a
Merge pull request #2 from hpc/master
...
Merging in latest master changes
2019-11-05 15:00:47 +00:00
b4c85798bd
Tidying up PMDK backend
2019-11-05 14:39:31 +00:00
46c5d4e78d
Adding in dual mount functionality for NVRAM within nodes
2019-11-05 14:37:54 +00:00
817603f5d1
Updating copyright statement
2019-10-31 15:31:28 +00:00
fa25c31994
Adding PMDK functionality
2019-10-31 15:29:09 +00:00
3c7ac2df23
Merge pull request #196 from daos-stack/daos-devel
...
update DFS driver with new DAOS API changes.
2019-10-28 16:56:29 -07:00
4409082064
Merge pull request #188 from pkoutoupis/feature-global_default_dir_layout
...
Enable global default dir layout for subdirs in Lustre
2019-10-28 10:43:56 -07:00
92180e022d
update DFS read/write APIs with new DAOS API changes
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-10-28 16:42:44 +00:00
ccba3b11e5
Merge branch 'master' into fix-194
2019-10-23 16:46:00 -06:00
61333af822
Fix #194 .
...
On previous versions of IOR (and most Linux CLI applications in general)
multiple flags can be combined such as -vvv to get verbose=3 rather than
having to specify each separately as -v -v -v. This patch fixes this behavior.
It also fixes issues around handling of the verbose flag where the
output would never be printed as we were comparing verbose < VERBOSE_0
where verbose defaults to 0 and can only increase.
2019-10-23 16:17:37 -06:00
b334c3f6fd
Merge pull request #192 from daos-stack/daos-devel
...
fix -R option to do proper data verification
2019-10-16 16:20:43 +01:00
dc82a1bf2f
ior -R should memset the buffer being read, otherwise a read that does nothing
...
will report success in the data verification phase.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-10-16 14:54:00 +00:00
a69a5916cf
Hotfix for ior -F -u
2019-10-16 09:42:10 +01:00
f40b0744ff
Merge pull request #187 from daos-stack/daos-devel
...
IOR: Add latency and IOPS numbers to each iteration.
2019-10-02 13:09:26 -07:00
b3ecd756ce
Implemented AC_CHECK_HEADERS fix in style of #190
2019-09-30 04:25:59 -07:00
2aea04c8cb
fix some compiler warnings.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-21 22:43:20 +00:00
fca0a62ccb
update to new DAOS API for creating dfs containers.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-21 22:05:54 +00:00
604598ab2f
Merge pull request #182 from jschwartz-cray/fix-181
...
Fix #181 .
2019-09-19 16:53:41 +01:00
12284ae04a
Add latency and iops numbers to each iteration.
...
- Latency reported is computed by taking the average latency of
all ops from a single task, then taking the minimum of that
between all tasks.
- IOPS is computed by taking the total number of ops across all
tasks divided by the total access time to execute those ops.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-10 18:49:32 +00:00
c58ba8ffb5
add break that was accidentally removed
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-10 14:11:55 +00:00
fe9d76ddf3
fix segfault when no API is specified to mdtest
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 22:03:55 +00:00
73dbda09c6
remove printf added
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 20:19:42 +00:00
5622aabf46
missed Makefile.am update
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 20:14:12 +00:00
d332d586bf
remove uneeded code and doc
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 20:07:55 +00:00
e9dd5fe15e
Merge branch 'master' into daos-devel
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
Conflicts:
configure.ac
src/Makefile.am
src/aiori-MPIIO.c
src/aiori.c
src/aiori.h
src/ior.c
src/mdtest.c
src/option.c
2019-09-09 19:55:56 +00:00
57a16ddda8
- update debugging error checking in DFS.
...
- remove usleep() before daos_fini()
- fix README
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 19:14:35 +00:00
0921556456
Merge pull request #184 from hpc/feature-aiori
...
Feature aiori
2019-09-09 15:20:43 +01:00
36d13d3b97
Enable global default dir layout for subdirs in Lustre
2019-09-02 22:26:12 -07:00
c83edfe39b
Extracted check function into aiori. #24 . #177
2019-09-01 15:59:52 +01:00
ced9b0183f
Merge pull request #172 from hpc/fix-168
...
Fix #168
2019-09-01 15:51:44 +01:00
151102ff8f
Merge pull request #176 from otatebe/master
...
abstract I/O interface for Gfarm file system
2019-09-01 15:51:16 +01:00
e3db1759b2
Moded sync() to aiori backend.
2019-09-01 15:47:42 +01:00
6de1ea3176
Merge branch 'feature-sync' into feature-aiori
2019-09-01 15:39:49 +01:00
60a641f911
sync help corrected.
2019-09-01 15:29:12 +01:00
10d3db1dc8
MDTest: fixing the memset() to account for the number of iterations.
2019-08-31 17:28:08 +01:00
47695aea49
Merge pull request #180 from jschwartz-cray/fix-179
...
Fix #179 .
2019-08-31 10:32:41 +01:00
0e952f0f8c
Fix #181 .
...
On systems where numTasks is not evenly divisible by 'tasksPerNode' we were
seeing some nodes reading multiple files while others read none after
reordering.
Commonly all nodes have the same number of tasks but there is nothing
requiring that to be the case. Imagine having 64 tasks running against 4
nodes which can run 20 tasks each. Here you get three groups of 20 and one
group of 4. On this sytem nodes running in the group of 4 were previously
getting tasksPerNode of 4 which meant they reordered tasks differently than
the nodes which got tasksPerNode of 20.
The key to fixing this is ensuring that every node reorders tasks the same
way, which means ensuring they all use the same input values. Obviously on
systems where the number of tasks per node is inconsistent the reordering will
also be inconsistent (some tasks may end up on the same node, or not as far
separated as desired, etc.) but at least this way you'll always end up with a
1:1 reordering.
- Renamed nodes/nodeCount to numNodes
- Renamed tasksPerNode to numTasksOnNode0
- Ensured that numTasksOnNode0 will always have the same value regardless of
which node you're on
- Removed inconsistently used globals numTasksWorld and tasksPerNode and
replaced with per-test params equivalents
- Added utility functions for setting these values:
- numNodes -> GetNumNodes
- numTasks -> GetNumTasks
- numTasksOnNode0 -> GetNumNodesOnTask0
- Improved MPI_VERSION < 3 logic for GetNumNodes so it works when numTasks is
not evenly divisible by numTasksOnNode0
- Left 'nodes' and 'tasksPerNode' in output alone to not break compatibility
- Allowed command-line params to override numTasks, numNodes, and
numTasksOnNode0 but default to using the MPI-calculated values
2019-08-30 16:45:03 -06:00
4c3d96bfed
Fix #179 .
...
-u (uniqueDir) will once again use the full file path specified by the
client instead of truncating it. This was caused by a broken sprintf
which was trying to read/write overlapping buffers.
From the glibc sprintf() documentation:
"The behavior of this function is undefined if copying takes place
between objects that overlap"
2019-08-30 15:31:23 -06:00
0bd979637e
Added some extra debug including ERRF, WARNF, and MPI_CHECKF format string macros.
2019-08-30 15:11:25 -06:00
32db1cd902
add timing for container close.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-29 03:24:48 +00:00
93730771fd
add some verbose messages on finalize routines for DAOS and DFS drivers.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-28 17:01:51 +00:00
4df051bf28
New option -Y to invoke the sync command.
2019-08-26 18:57:14 +01:00
a4068be551
Improved help for fsync.
2019-08-26 15:57:13 +01:00
0d9f46e980
MDTest re-added the -Z option for compatibility (for now) and switched back behavior.
2019-08-15 16:49:46 +01:00
de3baf8861
MDTest: Document choice of 42.
2019-08-15 16:21:30 +01:00
b3663bd29a
add sleep depending on MPI rank to avoid all ranks calling daos_fini()
...
at once (issue with PSM2).
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-14 13:32:51 +00:00
1320aa279c
add some barriers before cont close and destroy to make sure all
...
ranks are done.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-12 16:04:20 +00:00
8cb878507e
Add dfs chunk_size and oclass options.
...
update dfs_remove for API change.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-12 14:25:05 +00:00
70e8b13d1d
Fix Last Free
2019-08-04 18:27:20 -05:00
da5f091afc
Merge pull request #169 from ax3l/fix-singleRankHeapBufferOverflow
...
Fix: Heap Buffer Overflow
2019-08-03 09:20:15 +01:00
5e6a03442f
Merge pull request #170 from ax3l/fix-someMemleaks
...
Fix Some Memory Leaks; Thanks.
2019-08-03 09:19:08 +01:00
c4ff3d7c4e
Trivial fix for #168
2019-08-03 09:12:48 +01:00
f16ef9ace5
update dfs_lookup() call for extra parameter.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-03 05:07:13 +00:00
b72b51be48
Fix: Heap Buffer Overflow
...
Fix a memory violation when run in serial.
2019-08-02 23:34:13 -05:00
bfff0df8fd
Fix Some Memory Leaks
...
Fixing some memory leaks :)
2019-08-02 23:33:01 -05:00
b1b66962ac
incorrect warning
2019-08-02 13:03:59 +09:00
cf56715a5a
Make sure that each read buffer contains an invalid first byte.
2019-08-01 18:33:44 +01:00
ce1ae750f6
MDtest: Support to verify the read operation with a default pattern.
2019-08-01 18:29:32 +01:00
df8355a9bc
Added output of mdtest stonewall timer.
2019-08-01 17:57:45 +01:00
061b5a860f
Backmerged: New option: print rate AND time; improves debugging.
2019-08-01 17:54:11 +01:00
6c0fadc2a9
Include performance when stonewall is hit to output.
2019-08-01 17:20:01 +01:00
3890b71b54
Fixed issues and followed suggestions from Glenn's review of the PR
2019-08-01 09:42:03 +09:00
92939e4fbd
update for DAOS API changes
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-07-31 17:22:20 +00:00
a3c37808da
Made FAIL take variable args so we can pass printf like args to it
2019-07-28 11:17:11 -06:00
0ffec67d2b
Following Julian's suggestion about better naming
2019-07-28 10:25:42 -06:00
b2d486f749
Followed Andreas suggestion to replace escape double quotes within printf's with single quotes
2019-07-28 10:07:03 -06:00
168a407793
Fixed inconsistent spacing that Andreas commented upon
2019-07-28 09:55:00 -06:00
629ff810b7
Got IOR shifting to work regardless of whether node/task mapping is round-robin or contiguous
2019-07-27 15:27:20 -06:00
d69957e55b
Final changes cleaning up the output messages
2019-07-27 14:31:49 -06:00
e767ef3de9
Remove extraneous print_help function that was causing people to have to edit the same string in two different locations
2019-07-27 13:26:39 -06:00
f6491fcd37
Cleaned up the verbose messages by creating a macro and a function
2019-07-27 13:22:15 -06:00
9d6480d46e
Fixed bug in the nstride calculation where only 0 was computing it correctly. Had to bcast it out
2019-07-27 11:30:07 -06:00
945487f743
Better debug message when stat fails
2019-07-27 09:20:20 -06:00
524d053be1
Making shift work in mdtest as well as it works in IOR and on a per-node basis.
...
Also added printing the nodemap so we can check the allocation.
2019-07-26 08:55:24 -06:00
7981fc70b1
Changed the behavior of shift in mdtest to mimic that in IOR so that it shifts by _entire nodes_ not by ranks.
2019-07-25 19:18:52 -06:00
76f6bb6e42
Abort when invalid options are set.
2019-07-24 00:29:58 +01:00
1c640716c7
Use MPI Info object for all MPI_File_open calls.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-07-08 21:31:00 +00:00
ad61fdd89c
add proper check for DAOS object class and exit if unknown.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-07-08 18:18:24 +00:00
04e3df0d42
use the daos provided function for name to object class
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-07-08 17:02:30 +00:00
52ed766df4
update DAOS and DFS drivers to new DAOS API changes.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-07-08 15:33:06 +00:00
3eb0f86640
Function ncmpi_get_file_info was first introduced in
...
PnetCDF version 1.2.0. Check PnetCDF library version,
so printing MPI-IO hints can be enabled for NCMPI.
2019-06-28 17:40:35 -05:00
9f9730d6fd
should abort on daos init if pool/cont/svcl not specified.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-06-28 16:20:56 +00:00
2ae7c29de2
cache the parent dir in a hashtable so we don't have to do a lookup
...
on every access.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-06-27 21:31:07 +00:00
960005351d
update DAOS and DFS markings.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-06-25 22:41:30 +00:00
262d35d87e
- Merge branch 'master' into daos-devel
...
- fix bugs with cont destroy.
- add destroy option to DFS driver
- share pool and container handle with DFS driver, and allow multi rank access
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
Conflicts:
src/ior.c
src/mdtest.c
src/option.c
src/parse_options.c
2019-06-24 21:26:15 +00:00
a822363ee0
style fix
2019-06-22 10:48:41 +09:00
6cae44b512
abstract I/O interface for Gfarm file system
2019-06-21 21:05:41 +09:00
915283bae8
update array_g2l usage with DAOS API change
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-06-17 20:38:02 +00:00
af1aa627eb
use d_ version of function instead of daos_ since some of those are
...
going to be removed from the DAOS API.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-06-07 16:59:11 +00:00
cdaca2c591
Merge pull request #160 from hpc/fix-ime-157
...
Moving IME ODirect option into IME. #157
2019-05-28 08:41:26 +01:00
e4969b5af7
Fix for #159 (also slightly improved parsing for options, for portability between scripts also allow comment at the end of an option).
2019-05-23 12:47:43 +01:00
1b3182904d
Moving IME ODirect option into IME. #157
2019-05-23 09:54:10 +01:00
ed4c1d88d0
Fix missing initialization in MMAP options.
2019-05-23 09:31:05 +01:00
a6f8388f9f
add mknod support
...
Add new option '-k' to support file creation test with mknod,
which is widely used in lustre.
Signed-off-by: Gu Zheng <gzheng@ddn.com>
2019-05-23 13:41:46 +08:00
3d8893e507
Fix #155 . Fixup 3a873ea fixing #150 .
2019-05-22 15:25:27 +02:00
cf197d5e38
Moved broken collectiveMetadata option into HDF5 backend module. #154
2019-05-22 10:21:18 +01:00
c5c43c4f3c
Changed the version name to alpha to differentiate. Now mdtest has the version number from IOR.
2019-05-16 21:25:34 +01:00
3a873ea2b0
Fix #150 .
2019-05-16 19:35:00 +01:00
6abdcc363a
Trivial fix for #149
2019-05-16 12:56:56 +01:00
9ead56c24e
Merge pull request #148 from hpc/fix-146
...
Moving options to submodules + fixes #146 + fixes #144
2019-05-14 14:32:37 -07:00
30a1ca187b
Bugfix by rmn1 for #147 and #112 .
2019-05-14 14:55:11 +01:00
54522315ec
Fix #146 ; handling of hints. Why are GPFS hints special, remains unclear.
2019-05-08 14:02:40 +01:00
a9b0bf35bc
Support MDTest verbosity >=5 prints behavior for all ranks instead of just rank 0 (which is enabled with verbosity level 3 as before)
2019-05-02 09:58:35 +01:00
e3afdb5ed5
beegfs: don't fail if output exists
...
Use existing output file if it has the desired stripe settings
(number of targets, chunk size)
2019-04-29 15:52:26 +02:00
8fd755147a
Fix for #144
2019-04-19 10:20:04 +01:00
5797624d56
fix bug in file delete and access where only rank 0 is calling those functions.
...
remove the bcasts for the error code since the other ranks are not calling it.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-04-15 22:14:33 +00:00
27ac1b88af
Removed verbose output.
2019-04-02 12:34:35 +01:00
07217c93aa
Bugfix for the parsing of flags. Backwards compatibility for direct IO.
2019-04-02 12:08:31 +01:00
e295a4e45f
remove printf of chunk size.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-04-01 22:58:00 +00:00
4a788a0c23
re-write the DAOS IOR driver to use the Array API
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-30 19:11:44 +00:00
21405ed924
Updated MMap module to option parser.
2019-03-27 22:32:59 +00:00
b4fb80a69e
Allowed option parser to reset flags (needed in scriptfiles).
2019-03-27 22:32:38 +00:00
07ca5247d7
Checked proper IOR behavior for -f option.
2019-03-27 20:37:46 +00:00
20e960d020
Support the setting of module-specific options per test.
2019-03-27 20:04:48 +00:00
8b23c50389
update dfs_open for API change to expose chunk size selection
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-22 16:53:34 +00:00
39eca1bb08
add DAOS_Access routine to check if a container exists before deleting it.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-14 21:17:37 +00:00
d49d345007
Merge pull request #137 from DDNStorage/memoryPerNode
...
Fix segfault with option memoryPerNode
2019-03-14 18:33:24 +00:00
fabb1f4d3b
Fix segfault with option memoryPerNode
...
optarg contains a NULL address, which causes a segfault in
NodeMemoryStringToBytes().
2019-03-14 19:20:18 +01:00
c35f64237f
initialize data and akey checksum to NULL since they are not being used.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-14 16:55:54 +00:00
db19141c19
remove usage of GURT's D_GOTO from DFS plugin.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-01 20:41:01 +00:00
110153cf3a
mdtest: Enable IME native interface
...
This patch makes mdtest allow IME native interface.
It also defines a generic way to enable mdtest with each
backend.
2019-02-12 15:50:51 +01:00
c6abf85378
aiori-IME: Add support of statfs in IME backend
...
This patch adds the support of statfs in the IME backend.
It uses ime_native_statvfs call to fill the ior_aiori_statfs_t
structure.
2019-02-11 15:00:18 +01:00
29090df283
update DAOS and DFS drivers with new DAOS API changes.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-02-08 18:00:33 +00:00
2c87b5e0f5
Merge remote-tracking branch 'upstream/master' into daos
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
Conflicts:
src/aiori.c
src/aiori.h
src/ior.c
src/mdtest-main.c
src/mdtest.c
src/option.c
2019-01-24 00:31:12 +00:00
4c1c73ecae
add malloc check to fix
2019-01-23 16:02:58 -08:00
20bb99e1b7
Potential fix for #127 => Segmentation fault due to memory protection.
2019-01-19 17:00:49 +00:00
334bf73797
remove transaction commit and options to do that.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-01-17 20:58:58 +00:00
d2a0023765
remove epoch usage and rebase with latest DAOS API that switches to transactions.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-01-07 18:28:59 +00:00
e71cded6c3
Merge pull request #124 from hpc/fix-hdf5
...
Bugfix for HDF5 plugin; readded collective_md.
2019-01-02 10:01:56 -08:00
1dbca5c293
Bugfix for HDF5 plugin; readded collective_md.
2018-12-23 13:07:23 +00:00
ed421bcc7b
Fixes that issue. Hopefully works for all combinations.
2018-12-20 21:08:55 +00:00
7b603bc5a9
Reset stonewall timer to make it work again with running all phases in one execution; i.e. not using multiple runs specifying: -C, -r
2018-12-19 21:37:37 +00:00
4297ccd7f2
Bugfix for stonewall computation.
2018-12-19 20:29:47 +00:00
1f217ae48c
lustre: fix checks for lustre_user.h header
...
Fix the configure check if --with-lustre is specified, but
the linux/lustre/lustre_user.h header is not present. Only
one of the headers needs to be included if both are found.
In some cases, FASYNC is not defined, but forms part of the
O_LOV_DELAY_CREATE value, add a #define in that case.
Fixes #115 .
Fixes: cb88c4c19a831d94b864c49a162e2635730540e5
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
2018-12-14 16:45:43 -07:00
72c487013a
rebase with latest DAOS
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-12-04 22:59:15 +00:00
e1158daaf2
Merge pull request #105 from hpc/feature-test-scripts-85
...
Dry-run option to test the configuration / scripts.
2018-12-04 15:06:47 -06:00
5e2e1cc96f
Merge pull request #104 from hpc/feature-hdf5-backend
...
Changed the parser to fix #98 . Does _not_ yet have the HDF5 backend args updated. They were reverted to allow this to make the 3.2 release and should be re-applied as a separate commit.
2018-12-04 14:41:31 -06:00
5c847f552a
Merge pull request #110 from hpc/fix-gnu99-101
...
Moved _XOPEN_SOURCE to config.h; resolves #93
2018-12-04 14:13:16 -06:00
7db8ffd8ab
Reverted the change to HDF5 module back.
2018-12-04 19:16:41 +00:00
8902bc8722
mdtest: fix calculation for barriers
...
Currently, calculation for barriers case looks a bit
suprising, for example, Min stats is from min of
Max value of number of mpi proc, this makes Min
and Max very similar in our testing thus we get
a small Std value too.
I am not a MPI expert, but question is why we
don't use more nature calcuation which calculate
all results from different iterations and MPI proc?
Signed-off-by: Wang Shilong <wshilong@ddn.com>
2018-11-22 19:57:32 +08:00
5a8071fbd8
GNU99 extracted _XOPEN_SOURCE to config.h.
2018-10-28 18:34:22 +00:00
3a4f2ea33e
Merge pull request #101 from glennklockwood/issue93-squashed
...
compile correctly with -std=c99
2018-10-28 18:16:52 +00:00
52182afed6
Merge pull request #108 from hpc/fix-106
...
Fix 106
2018-10-19 11:06:05 +01:00
ea3aeea964
Bugfix initialize the variable to zero for proper API usage.
2018-10-17 16:16:46 +01:00
6b0b1a9830
Fix computation when using branching factor with -n and -i.
2018-10-17 16:04:49 +01:00
093113313d
mdtest: Use GetTimeStamp() instead of MPI_Wtime()
...
IOR uses the GetTimeStamp() wrapper to allow gettimeofday() to be used for timings instead of MPI_Wtime() to calculate throughput. This commits adds the same logic to mdtest timings.
2018-10-17 16:38:35 +02:00
f4afa63ebf
Bugfix #106 mostly.
2018-10-17 13:50:04 +01:00
f7dc084f32
Merge pull request #103 from DDNStorage/ime_mkdir_rmdir
...
IME: add support of mkdir/rmdir into the IME backend
2018-10-16 14:34:26 +01:00
6c5952906b
Dry-run option to test the configuration / scripts.
2018-10-11 19:58:30 +01:00
3b5f4fc002
Bugfix JSON output for multiple processes
2018-10-11 18:59:12 +01:00
fa38cb7992
Bugfix JSON header (allows parsing at line 0).
2018-10-11 18:49:25 +01:00
6f7576aa8a
Changed the parser.
...
The parser now supports concurrent parsing of all plugin options.
Moved HDF5 collective_md option into the backend as an example.
Example: ./src/ior -a dummy --dummy.delay-xfer=50000
2018-10-11 18:31:21 +01:00
912d93e94f
aiori-IME: add support of mkdir/rmdir to the backup
...
ime_native_mkdir() and ime_native_rmdir() are available starting
from IME native API version 1.3.
2018-10-09 17:56:08 +02:00
e6bd9ad3b3
common: call mkdir/access syscalls defined in backends
...
mkdir/access syscalls used in PrependDir() should call their equivalent
implemented by the backend.
2018-10-09 17:55:58 +02:00
87c5c9ef04
compile correctly with -std=c99
...
This patch enables correct compilation on both MacOS and Linux using only
POSIX.1-2008-compliant C with XSI extensions.
Specifically, POSIX.1-2008 is the minimum version because we use strdup(3);
explicit XSI is required to expose putenv from glibc.
2018-10-08 13:47:28 -07:00
7045295a86
Replaced usleep to nanosleep.
2018-10-06 17:30:00 +01:00
9f56d8b8dd
Merge pull request #89 from DDNStorage/fixwritecheck
...
Make write verification work even without read test
2018-10-06 09:03:32 -07:00
4f1be703b1
Merge pull request #97 from glennklockwood/issue96
...
fixes #96
2018-10-06 16:55:05 +01:00
c6580dc93b
Merge pull request #99 from glennklockwood/issue87
...
fixes #87
2018-10-06 16:54:23 +01:00
a0ef65e61f
stop the segfault when an invalid api is specified in configfile
2018-10-05 23:29:21 -07:00
d8d4742523
fixes #87
...
Leading whitespace is stripped from each line of the ior input file. This
allows indented comments to be treated as comments. However it does NOT allow
one to specify excessive whitespace inside of the `ior start` and `ior stop`
magic phrases.
Also added a test to catch regressions in this functionality.
2018-10-05 22:50:34 -07:00
3aa3ff10f5
fixes #96
...
Call MPI_Abort() if MPI is initialized; otherwise just exit()
2018-10-05 22:20:18 -07:00
acee84bedc
Merge pull request #88 from roblatham00/hdf5-coll-md
...
HDF5: option to enable collective metadata
2018-10-02 16:13:05 +01:00
60820da874
Merge pull request #91 from glennklockwood/makedist-fix
...
fix various issues related to release packages
2018-10-01 21:30:58 -07:00
1f1e56c363
Usleep feature added according to Nathan's suggestions.
2018-09-30 10:01:21 +01:00
0a103147ab
Merge pull request #92 from DDNStorage/backend-legacy
...
Support old legacy name in backends
2018-09-24 11:38:13 +01:00
f472162784
Support old legacy name in backends
...
Context: Some backends may have used different names in
the past (like IME backend use to be IM). Legacy scripts
may break.
This patch adds a legacy name option in the aiori structure.
Both name and legacy name work to select the interface.
But the following warning is printed if the legacy name is used:
ior WARNING: [legacy name] backend is deprecated use [name] instead.
2018-09-24 11:51:00 +02:00
e448ff266d
fix various issues related to release packages
...
- travis now tests the packaged source to detect missing source/headers
- basic tests are less sensitive to the directory from where they are run
- fixed some missing files from the `make dist` manifest
- updated the format of NEWS to work with `make dist`
2018-09-21 18:35:48 -04:00
7a7655e959
Fix erroneous write bandwidth with stonewalling
...
Context: write and read results from the same iteration
use the same length value in Bytes. When stonewalling is
used the size variates depending on the performance of
the access. This leads to wrong max bandwidths reported
for writes as shown in the following example:
write 10052 ...
read 9910 ...
write 10022 ...
read 9880 ...
write 10052 ...
read 9894 ...
Max Write: 9371.43 MiB/sec (9826.66 MB/sec)
Max Read: 9910.48 MiB/sec (10391.89 MB/sec)
This patch makes IOR separate variables used for read
and write tests.
2018-09-21 11:43:33 +02:00
8c727fa99c
Simplify and factorize a few functions
...
This patch factorizes some duplicated code. It also
simplifies the way times are stored to prepare next
patch.
2018-09-21 11:38:44 +02:00
f03e7c6778
Make write verification work even without read test
...
Context: IOR outputs errors when the '-w -W' flags are used
without '-r'. Write a file using with check write option
should be possible even without setting read.
This patch fixes a condition which was introduced for
HDFS to remove RDWR flag in some particular cases.
Write check was set with the write only flag but it
requieres the read flag.
2018-09-20 18:18:48 +02:00
7d2464f733
HDF5: option to enable collective metadata
...
In HDF5-1.10 the library has an option to read and write the metadata of a file collectively. One has to opt-in to this optimization with a property list but particularly at large scale it can have tremendous performance implications.
Further information:
- https://support.hdfgroup.org/HDF5/docNewFeatures/CollMDataIO/RFC-CollectiveMetadataReads.pdf
- https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetAllCollMetadataOps
2018-09-20 11:12:08 -05:00
e5c8a11769
Merge pull request #83 from DDNStorage/initbackends
...
Initialize only the required backends
2018-09-19 10:27:11 +01:00
2bcaea564e
Initialize only the required backends
...
Context: IOR initializes all available backends. If one
backend fails to initialize IOR cannot be used.
This patch makes IOR initialize only the backends
which will be used. The initialization is done after
that the parameters are checked so that the help message
can still be dispayed is something goes wrong.
2018-09-18 19:24:29 +02:00
44e8f03917
Fix bug in computing max write and read bandwidths
...
This patch fixes a bug when computing the maximum
bandwidths in short summary output.
2018-09-18 19:23:36 +02:00
fd34b08e05
Added patch fix O_DIRECT again.
2018-09-18 16:48:59 +01:00
0a7b1e38ea
fix verbose output in DAOS & DFS drivers
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-09-13 13:32:02 +00:00
2f713e947d
fix bug in driver name check for DAOS.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-09-12 22:53:42 +00:00
151a9daa52
Check provided IO API is valid
...
Context: IOR gets a segfault if an unsupported API string
is provided.
This patch checks that the I/O backend is supported, otherwise
IOR stops with an error.
2018-09-10 19:16:28 +02:00
1a4fd7d095
Fix -i option #81
2018-09-05 09:14:22 +01:00
2668363fb1
- update README for DAOS drivers.
...
- update flag options in DAOS API
- make svcl option in DFS API the same as DAOS API
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-08-30 14:47:17 +00:00
2d42c25f69
Fix issue #79 for the last run in a configuration file.
2018-08-30 15:10:38 +01:00
14d67c19d9
fix DAOS plugin options passing.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-08-29 22:25:48 +00:00
46ff4e26be
Merge remote branch 'daos-stack/daos'
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
Conflicts:
configure.ac
src/Makefile.am
src/aiori.c
src/aiori.h
src/ior.c
src/mdtest.c
src/parse_options.c
2018-08-29 21:14:19 +00:00
ba2f0330b7
Merge pull request #80 from roblatham00/more-fixes
...
We are getting there ^-^
2018-08-29 21:09:31 +01:00
c1f3f485ec
Bugfix #79
2018-08-29 21:07:34 +01:00
cb88c4c19a
including lustre_user now deprecated
...
put in a configure test (same one used in ROMIO) in case someone's using
old Lustre client headers
2018-08-28 16:27:21 -05:00
4a32a2cbb3
fix "number of arguments doesn’t match prototype" error
2018-08-28 16:24:34 -05:00
6b782226c7
fix pnetcdf build issues
...
It does not look like anyone has built the pnetcdf driver in a long time
2018-08-28 14:29:36 -05:00
982f266a73
clean up hints in HDF5 error cases
2018-08-28 13:30:51 -05:00
f4b03efd72
Add the DAOS ior driver.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-08-27 17:22:38 +00:00
5fb850c810
- update the DFS driver to latest DFS API.
...
- update cmd line options to add DAOS Pool and Container uuid and SVCL
- Add init/finalize backend functions.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-08-23 21:58:53 +00:00
4dcf36c1a4
Help text now outputs *compiled* APIs.
2018-08-15 17:07:17 +01:00
fefefa002d
make sure to allow fsyncs in RADOS
2018-08-14 17:08:04 -05:00
dfc86b60b8
add RADOS backend options for pool, config file
2018-08-14 16:56:19 -05:00
0ca5f26e84
cleanup RADOS func names and docs
2018-08-14 16:18:35 -05:00
dc7187794c
Merge pull request #70 from DDNStorage/fix-ime-backend
...
Fix compilation of some aiori backends and add new function calls in IME backend
2018-08-10 18:00:37 +01:00
13b885ce21
Merge pull request #71 from DDNStorage/fixwarnings
...
Fix warnings reported at compilation time
2018-08-09 09:46:24 -07:00
490a912e6e
Add support for new function calls in the IME backend
...
This patch adds initialize, finalize and stat calls to
the IME backend. statfs, mkdir and rmdir are currently
not supported. This patch also fixes the IME_GetVersion
call.
2018-08-09 13:08:43 +02:00
9596b4bbe0
Fix compilation of aiori backends requiring external libraries
...
This patchs makes a few adjustements in the automake file
in order to properly add include and library directories.
Those are required to be able to compile some aiori backends.
2018-08-09 13:08:36 +02:00
df34f024ba
Fix warnings reported at compilation time
...
This patch fixes most of the warnings caused
by unused variables and assignments from incompatible
type.
2018-08-09 13:04:52 +02:00
54fa3fa8e9
Fixed wrong output.
2018-08-07 23:28:19 +01:00
44ce8e5aaf
Add backend for DDN's Infinite Memory Engine (IME)
...
This patch adds the support of IME Native interface
as a new AIORI backend.
2018-08-07 11:41:07 +01:00
e708681f19
Option: check for missing arg.
2018-08-06 18:38:34 +01:00
0870ad78b3
Support shared large runs with limited number of files per directory by creating one directory for a bunch of files.
...
Therefore, it uses the -I argument (number of files per directory) and divides it by -n (items).
This will be a number of sub directories that are created on the top-level.
2018-07-26 10:38:06 +01:00
07ec65c0d5
remoce rank_list_parse as it is exposed by DAOS API now.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-07-19 21:36:29 +00:00
580fb8cdf7
Bugfix offsetof; thanks George for reporting.
2018-07-15 20:23:10 +01:00
f55761d5d2
Refactored results structure to AoS; allowed to keep results per repeat.
2018-07-15 19:38:17 +01:00
c7a598a435
New option parser: Fixed overflow for blocksize. Added warning.
2018-07-15 18:37:15 +01:00
bb6d4fb2a2
Update create for stonewall.
2018-07-15 10:02:45 +01:00
d4ecc76ee8
Stonewall bugfix for delete/stat.
2018-07-14 16:46:24 +01:00
c0657866fd
Minor fixes for stonewalling options.
2018-07-14 12:22:36 +01:00
5d4ac1a0a0
Bugfix parser: wrong datatype
2018-07-14 11:32:39 +01:00
04ba846da2
Fix for SetHints (after changing way strings are managed).
2018-07-14 11:11:47 +01:00
b613bf9396
AIORI update to initialization and fix HDF5 with changed API.
2018-07-14 10:47:25 +01:00
ac7600d662
Unit-test with automake for library version. Minor fix for library version.
2018-07-14 10:27:31 +01:00
5ef95675bf
AIORI initialize() and finalize() functions.
2018-07-14 09:29:08 +01:00
a0b8659cc8
IOR appears to work with the new option parser.
2018-07-14 09:09:48 +01:00
5e98fa0408
Dummy: option for xfer delay.
2018-07-14 08:50:27 +01:00
54e47cf729
IOR: use new option parser.
...
TODO: Parse "-O" options
2018-07-14 08:41:35 +01:00
0835191372
Option parser replaced.
...
Benefits: shows currently set options when used with -h
Supports options for child modules, example for aiori-dummy works, RADOS untested.
2018-07-12 18:09:13 +01:00
205b20f0f4
Bugfix merge
2018-07-12 16:13:40 +01:00
060c47e80a
Merge branch 'testing' into rados-hacks
2018-07-12 15:53:04 +01:00
7981691a63
add a new AIORI backend for RADOS
...
inform aiori interface about RADOS backend
stubbed out aiori backend for rados
additions to get RADOS backend compiling/linking
first cut at rados create/open patha
make sure to return RADOS oid on open/create
implement rados xfer path for WRITE
refactor + implement getfilesize and close
remember to use read_op interface for stat
implement RADOS delete function
don't error in RADOS_Delete for now
implement RADOS set_version
handle open/create flags appropriately
cleanup RADOS error handling
implement read/readcheck/writecheck for RADOS
rados doesn't support directio
implement unsupported aiori ops for RADOS
implement RADOS access call
define rados types if no rados support
2018-07-12 09:36:49 -05:00
831a5ef823
JSON: Slight output adaption.
2018-07-09 16:25:35 +01:00
0f17d4860f
Pretty printing for normal output.
2018-07-08 16:56:04 +01:00
2bc30410cc
JSON output works.
2018-07-08 16:47:23 +01:00
20ebeb71b8
Further adaptions to generic output, support of JSON output (partially complete).
2018-07-08 14:59:54 +01:00
8525f9734c
Extraction of output completed.
2018-07-08 13:50:20 +01:00
02883d4a63
Moved most output to ior-output.c
2018-07-08 13:38:05 +01:00
0f7a1f14b9
Support IO redirect from main()
2018-07-08 13:12:18 +01:00
Julian M. Kunkel