Mohamad Chaarawi
fe9d76ddf3
fix segfault when no API is specified to mdtest
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 22:03:55 +00:00
Mohamad Chaarawi
73dbda09c6
remove printf added
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 20:19:42 +00:00
Mohamad Chaarawi
5622aabf46
missed Makefile.am update
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 20:14:12 +00:00
Mohamad Chaarawi
d332d586bf
remove uneeded code and doc
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 20:07:55 +00:00
Mohamad Chaarawi
e9dd5fe15e
Merge branch 'master' into daos-devel
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
Conflicts:
configure.ac
src/Makefile.am
src/aiori-MPIIO.c
src/aiori.c
src/aiori.h
src/ior.c
src/mdtest.c
src/option.c
2019-09-09 19:55:56 +00:00
Mohamad Chaarawi
57a16ddda8
- update debugging error checking in DFS.
...
- remove usleep() before daos_fini()
- fix README
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-09 19:14:35 +00:00
Julian Kunkel
0921556456
Merge pull request #184 from hpc/feature-aiori
...
Feature aiori
2019-09-09 15:20:43 +01:00
Petros Koutoupis
36d13d3b97
Enable global default dir layout for subdirs in Lustre
2019-09-02 22:26:12 -07:00
Julian M. Kunkel
c83edfe39b
Extracted check function into aiori. #24 . #177
2019-09-01 15:59:52 +01:00
Julian Kunkel
ced9b0183f
Merge pull request #172 from hpc/fix-168
...
Fix #168
2019-09-01 15:51:44 +01:00
Julian Kunkel
151102ff8f
Merge pull request #176 from otatebe/master
...
abstract I/O interface for Gfarm file system
2019-09-01 15:51:16 +01:00
Julian M. Kunkel
e3db1759b2
Moded sync() to aiori backend.
2019-09-01 15:47:42 +01:00
Julian M. Kunkel
6de1ea3176
Merge branch 'feature-sync' into feature-aiori
2019-09-01 15:39:49 +01:00
Julian M. Kunkel
60a641f911
sync help corrected.
2019-09-01 15:29:12 +01:00
Julian M. Kunkel
10d3db1dc8
MDTest: fixing the memset() to account for the number of iterations.
2019-08-31 17:28:08 +01:00
Julian Kunkel
47695aea49
Merge pull request #180 from jschwartz-cray/fix-179
...
Fix #179 .
2019-08-31 10:32:41 +01:00
Josh Schwartz
0e952f0f8c
Fix #181 .
...
On systems where numTasks is not evenly divisible by 'tasksPerNode' we were
seeing some nodes reading multiple files while others read none after
reordering.
Commonly all nodes have the same number of tasks but there is nothing
requiring that to be the case. Imagine having 64 tasks running against 4
nodes which can run 20 tasks each. Here you get three groups of 20 and one
group of 4. On this sytem nodes running in the group of 4 were previously
getting tasksPerNode of 4 which meant they reordered tasks differently than
the nodes which got tasksPerNode of 20.
The key to fixing this is ensuring that every node reorders tasks the same
way, which means ensuring they all use the same input values. Obviously on
systems where the number of tasks per node is inconsistent the reordering will
also be inconsistent (some tasks may end up on the same node, or not as far
separated as desired, etc.) but at least this way you'll always end up with a
1:1 reordering.
- Renamed nodes/nodeCount to numNodes
- Renamed tasksPerNode to numTasksOnNode0
- Ensured that numTasksOnNode0 will always have the same value regardless of
which node you're on
- Removed inconsistently used globals numTasksWorld and tasksPerNode and
replaced with per-test params equivalents
- Added utility functions for setting these values:
- numNodes -> GetNumNodes
- numTasks -> GetNumTasks
- numTasksOnNode0 -> GetNumNodesOnTask0
- Improved MPI_VERSION < 3 logic for GetNumNodes so it works when numTasks is
not evenly divisible by numTasksOnNode0
- Left 'nodes' and 'tasksPerNode' in output alone to not break compatibility
- Allowed command-line params to override numTasks, numNodes, and
numTasksOnNode0 but default to using the MPI-calculated values
2019-08-30 16:45:03 -06:00
Josh Schwartz
4c3d96bfed
Fix #179 .
...
-u (uniqueDir) will once again use the full file path specified by the
client instead of truncating it. This was caused by a broken sprintf
which was trying to read/write overlapping buffers.
From the glibc sprintf() documentation:
"The behavior of this function is undefined if copying takes place
between objects that overlap"
2019-08-30 15:31:23 -06:00
Josh Schwartz
0bd979637e
Added some extra debug including ERRF, WARNF, and MPI_CHECKF format string macros.
2019-08-30 15:11:25 -06:00
Mohamad Chaarawi
32db1cd902
add timing for container close.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-29 03:24:48 +00:00
Mohamad Chaarawi
93730771fd
add some verbose messages on finalize routines for DAOS and DFS drivers.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-28 17:01:51 +00:00
Julian M. Kunkel
4df051bf28
New option -Y to invoke the sync command.
2019-08-26 18:57:14 +01:00
Julian M. Kunkel
a4068be551
Improved help for fsync.
2019-08-26 15:57:13 +01:00
Julian M. Kunkel
0d9f46e980
MDTest re-added the -Z option for compatibility (for now) and switched back behavior.
2019-08-15 16:49:46 +01:00
Julian M. Kunkel
de3baf8861
MDTest: Document choice of 42.
2019-08-15 16:21:30 +01:00
Mohamad Chaarawi
b3663bd29a
add sleep depending on MPI rank to avoid all ranks calling daos_fini()
...
at once (issue with PSM2).
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-14 13:32:51 +00:00
Mohamad Chaarawi
1320aa279c
add some barriers before cont close and destroy to make sure all
...
ranks are done.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-12 16:04:20 +00:00
Mohamad Chaarawi
8cb878507e
Add dfs chunk_size and oclass options.
...
update dfs_remove for API change.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-12 14:25:05 +00:00
Axel Huebl
70e8b13d1d
Fix Last Free
2019-08-04 18:27:20 -05:00
Julian Kunkel
da5f091afc
Merge pull request #169 from ax3l/fix-singleRankHeapBufferOverflow
...
Fix: Heap Buffer Overflow
2019-08-03 09:20:15 +01:00
Julian Kunkel
5e6a03442f
Merge pull request #170 from ax3l/fix-someMemleaks
...
Fix Some Memory Leaks; Thanks.
2019-08-03 09:19:08 +01:00
Julian M. Kunkel
c4ff3d7c4e
Trivial fix for #168
2019-08-03 09:12:48 +01:00
Mohamad Chaarawi
f16ef9ace5
update dfs_lookup() call for extra parameter.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-08-03 05:07:13 +00:00
Axel Huebl
b72b51be48
Fix: Heap Buffer Overflow
...
Fix a memory violation when run in serial.
2019-08-02 23:34:13 -05:00
Axel Huebl
bfff0df8fd
Fix Some Memory Leaks
...
Fixing some memory leaks :)
2019-08-02 23:33:01 -05:00
Osamu Tatebe
b1b66962ac
incorrect warning
2019-08-02 13:03:59 +09:00
Julian M. Kunkel
cf56715a5a
Make sure that each read buffer contains an invalid first byte.
2019-08-01 18:33:44 +01:00
Julian M. Kunkel
ce1ae750f6
MDtest: Support to verify the read operation with a default pattern.
2019-08-01 18:29:32 +01:00
Julian M. Kunkel
df8355a9bc
Added output of mdtest stonewall timer.
2019-08-01 17:57:45 +01:00
Julian M. Kunkel
061b5a860f
Backmerged: New option: print rate AND time; improves debugging.
2019-08-01 17:54:11 +01:00
Julian M. Kunkel
6c0fadc2a9
Include performance when stonewall is hit to output.
2019-08-01 17:20:01 +01:00
John Bent
3890b71b54
Fixed issues and followed suggestions from Glenn's review of the PR
2019-08-01 09:42:03 +09:00
Mohamad Chaarawi
92939e4fbd
update for DAOS API changes
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-07-31 17:22:20 +00:00
John Bent
a3c37808da
Made FAIL take variable args so we can pass printf like args to it
2019-07-28 11:17:11 -06:00
John Bent
0ffec67d2b
Following Julian's suggestion about better naming
2019-07-28 10:25:42 -06:00
John Bent
b2d486f749
Followed Andreas suggestion to replace escape double quotes within printf's with single quotes
2019-07-28 10:07:03 -06:00
John Bent
168a407793
Fixed inconsistent spacing that Andreas commented upon
2019-07-28 09:55:00 -06:00
John Bent
629ff810b7
Got IOR shifting to work regardless of whether node/task mapping is round-robin or contiguous
2019-07-27 15:27:20 -06:00
John Bent
d69957e55b
Final changes cleaning up the output messages
2019-07-27 14:31:49 -06:00
John Bent
e767ef3de9
Remove extraneous print_help function that was causing people to have to edit the same string in two different locations
2019-07-27 13:26:39 -06:00
John Bent
f6491fcd37
Cleaned up the verbose messages by creating a macro and a function
2019-07-27 13:22:15 -06:00
John Bent
9d6480d46e
Fixed bug in the nstride calculation where only 0 was computing it correctly. Had to bcast it out
2019-07-27 11:30:07 -06:00
John Bent
945487f743
Better debug message when stat fails
2019-07-27 09:20:20 -06:00
John Bent
524d053be1
Making shift work in mdtest as well as it works in IOR and on a per-node basis.
...
Also added printing the nodemap so we can check the allocation.
2019-07-26 08:55:24 -06:00
John Bent
7981fc70b1
Changed the behavior of shift in mdtest to mimic that in IOR so that it shifts by _entire nodes_ not by ranks.
2019-07-25 19:18:52 -06:00
Julian M. Kunkel
76f6bb6e42
Abort when invalid options are set.
2019-07-24 00:29:58 +01:00
Mohamad Chaarawi
1c640716c7
Use MPI Info object for all MPI_File_open calls.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-07-08 21:31:00 +00:00
Mohamad Chaarawi
ad61fdd89c
add proper check for DAOS object class and exit if unknown.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-07-08 18:18:24 +00:00
Mohamad Chaarawi
04e3df0d42
use the daos provided function for name to object class
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-07-08 17:02:30 +00:00
Mohamad Chaarawi
52ed766df4
update DAOS and DFS drivers to new DAOS API changes.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-07-08 15:33:06 +00:00
Wei-keng Liao
3eb0f86640
Function ncmpi_get_file_info was first introduced in
...
PnetCDF version 1.2.0. Check PnetCDF library version,
so printing MPI-IO hints can be enabled for NCMPI.
2019-06-28 17:40:35 -05:00
Mohamad Chaarawi
9f9730d6fd
should abort on daos init if pool/cont/svcl not specified.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-06-28 16:20:56 +00:00
Mohamad Chaarawi
2ae7c29de2
cache the parent dir in a hashtable so we don't have to do a lookup
...
on every access.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-06-27 21:31:07 +00:00
Mohamad Chaarawi
960005351d
update DAOS and DFS markings.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-06-25 22:41:30 +00:00
Mohamad Chaarawi
262d35d87e
- Merge branch 'master' into daos-devel
...
- fix bugs with cont destroy.
- add destroy option to DFS driver
- share pool and container handle with DFS driver, and allow multi rank access
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
Conflicts:
src/ior.c
src/mdtest.c
src/option.c
src/parse_options.c
2019-06-24 21:26:15 +00:00
Osamu Tatebe
a822363ee0
style fix
2019-06-22 10:48:41 +09:00
Osamu Tatebe
6cae44b512
abstract I/O interface for Gfarm file system
2019-06-21 21:05:41 +09:00
Mohamad Chaarawi
915283bae8
update array_g2l usage with DAOS API change
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-06-17 20:38:02 +00:00
Mohamad Chaarawi
af1aa627eb
use d_ version of function instead of daos_ since some of those are
...
going to be removed from the DAOS API.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-06-07 16:59:11 +00:00
Julian Kunkel
cdaca2c591
Merge pull request #160 from hpc/fix-ime-157
...
Moving IME ODirect option into IME. #157
2019-05-28 08:41:26 +01:00
Julian M. Kunkel
e4969b5af7
Fix for #159 (also slightly improved parsing for options, for portability between scripts also allow comment at the end of an option).
2019-05-23 12:47:43 +01:00
Julian M. Kunkel
1b3182904d
Moving IME ODirect option into IME. #157
2019-05-23 09:54:10 +01:00
Julian M. Kunkel
ed4c1d88d0
Fix missing initialization in MMAP options.
2019-05-23 09:31:05 +01:00
Gu Zheng
a6f8388f9f
add mknod support
...
Add new option '-k' to support file creation test with mknod,
which is widely used in lustre.
Signed-off-by: Gu Zheng <gzheng@ddn.com>
2019-05-23 13:41:46 +08:00
Vaclav Hapla
3d8893e507
Fix #155 . Fixup 3a873ea
fixing #150 .
2019-05-22 15:25:27 +02:00
Julian M. Kunkel
cf197d5e38
Moved broken collectiveMetadata option into HDF5 backend module. #154
2019-05-22 10:21:18 +01:00
Julian M. Kunkel
c5c43c4f3c
Changed the version name to alpha to differentiate. Now mdtest has the version number from IOR.
2019-05-16 21:25:34 +01:00
Julian M. Kunkel
3a873ea2b0
Fix #150 .
2019-05-16 19:35:00 +01:00
Julian M. Kunkel
6abdcc363a
Trivial fix for #149
2019-05-16 12:56:56 +01:00
Glenn K. Lockwood
9ead56c24e
Merge pull request #148 from hpc/fix-146
...
Moving options to submodules + fixes #146 + fixes #144
2019-05-14 14:32:37 -07:00
Julian M. Kunkel
30a1ca187b
Bugfix by rmn1 for #147 and #112 .
2019-05-14 14:55:11 +01:00
Julian M. Kunkel
54522315ec
Fix #146 ; handling of hints. Why are GPFS hints special, remains unclear.
2019-05-08 14:02:40 +01:00
Julian M. Kunkel
a9b0bf35bc
Support MDTest verbosity >=5 prints behavior for all ranks instead of just rank 0 (which is enabled with verbosity level 3 as before)
2019-05-02 09:58:35 +01:00
Oliver Steffen
e3afdb5ed5
beegfs: don't fail if output exists
...
Use existing output file if it has the desired stripe settings
(number of targets, chunk size)
2019-04-29 15:52:26 +02:00
Julian M. Kunkel
8fd755147a
Fix for #144
2019-04-19 10:20:04 +01:00
Mohamad Chaarawi
5797624d56
fix bug in file delete and access where only rank 0 is calling those functions.
...
remove the bcasts for the error code since the other ranks are not calling it.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-04-15 22:14:33 +00:00
Julian M. Kunkel
27ac1b88af
Removed verbose output.
2019-04-02 12:34:35 +01:00
Julian M. Kunkel
07217c93aa
Bugfix for the parsing of flags. Backwards compatibility for direct IO.
2019-04-02 12:08:31 +01:00
Mohamad Chaarawi
e295a4e45f
remove printf of chunk size.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-04-01 22:58:00 +00:00
Mohamad Chaarawi
4a788a0c23
re-write the DAOS IOR driver to use the Array API
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-30 19:11:44 +00:00
Julian M. Kunkel
21405ed924
Updated MMap module to option parser.
2019-03-27 22:32:59 +00:00
Julian M. Kunkel
b4fb80a69e
Allowed option parser to reset flags (needed in scriptfiles).
2019-03-27 22:32:38 +00:00
Julian M. Kunkel
07ca5247d7
Checked proper IOR behavior for -f option.
2019-03-27 20:37:46 +00:00
Julian M. Kunkel
20e960d020
Support the setting of module-specific options per test.
2019-03-27 20:04:48 +00:00
Mohamad Chaarawi
8b23c50389
update dfs_open for API change to expose chunk size selection
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-22 16:53:34 +00:00
Mohamad Chaarawi
39eca1bb08
add DAOS_Access routine to check if a container exists before deleting it.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-14 21:17:37 +00:00
Julian Kunkel
d49d345007
Merge pull request #137 from DDNStorage/memoryPerNode
...
Fix segfault with option memoryPerNode
2019-03-14 18:33:24 +00:00
Sylvain Didelot
fabb1f4d3b
Fix segfault with option memoryPerNode
...
optarg contains a NULL address, which causes a segfault in
NodeMemoryStringToBytes().
2019-03-14 19:20:18 +01:00
Mohamad Chaarawi
c35f64237f
initialize data and akey checksum to NULL since they are not being used.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-14 16:55:54 +00:00
Mohamad Chaarawi
db19141c19
remove usage of GURT's D_GOTO from DFS plugin.
...
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-01 20:41:01 +00:00