vitalif
/
mdtest
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity
623 Commits
1 Branch
0 Tags
2.5 MiB
Commit Graph

447 Commits (f280123d0b08c7f54987a33f3209c20d8e288b1f)

Author SHA1 Message Date
Julian M. Kunkel 240d5fae79 Trivial fix to ensure all functions are prototypes. 2019-12-21 11:19:02 +00:00
Julian M. Kunkel 73c5c0efc2 Fix for collective operations by Wolfgang Szoecs #210. 
It does have performance implications but is only active if needed.
 2019-12-19 13:43:38 +00:00
Jean-Yves VET 0666875df3 aiori-IME: Add support of sync and mknod in IME backend 
This patch adds the support of sync (flush client page cache
for all opened files) and mknod in the IME backend.
 2019-12-17 14:51:21 +01:00
Glenn K. Lockwood 7a2d307c74
Merge pull request #208 from jyvet/mknod 
Fix confusing output of mknod
 2019-12-16 12:38:13 -08:00
Jean-Yves VET 9da36abaca Fix confusing output of mknod 
Context: mknod does not return a file descriptor.
 2019-12-16 14:55:26 +01:00
Jean-Yves VET be48206e5f Fix compilation warnings 2019-12-16 14:51:10 +01:00
Julian Kunkel 3a34b2efb7
Merge pull request #200 from daos-stack/daos-devel 
DAOS driver update
 2019-11-13 10:38:28 +00:00
Mohamad Chaarawi acfd4673cc We only need to add dirs to the Hash in the DFS driver. 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-11-13 00:09:03 +00:00
Adrian Jackson ddac5c8dbf
Merge pull request #3 from hpc/master 
Bring fork up to date
 2019-11-12 14:20:53 +00:00
Robert LeBlanc 47199e28dd Not sure why iops and latency are left out of the JSON job output. It is really useful information. 
Signed-off by: Robert LeBlanc <robert@leblancnet.us>
 2019-11-11 23:16:01 +00:00
Afrian Jackson 9c0926ef2a Moving fsync pmdk functionalty from persist to drain on the advice of the PMDK developers 2019-11-07 20:17:55 +00:00
Mohamad Chaarawi fbff9b6fb2 add DFS prefix option 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-11-07 18:01:07 +00:00
Adrian Jackson c24b2f156a
Merge pull request #2 from hpc/master 
Merging in latest master changes
 2019-11-05 15:00:47 +00:00
Afrian Jackson b4c85798bd Tidying up PMDK backend 2019-11-05 14:39:31 +00:00
Afrian Jackson 46c5d4e78d Adding in dual mount functionality for NVRAM within nodes 2019-11-05 14:37:54 +00:00
Afrian Jackson 817603f5d1 Updating copyright statement 2019-10-31 15:31:28 +00:00
Afrian Jackson fa25c31994 Adding PMDK functionality 2019-10-31 15:29:09 +00:00
Glenn K. Lockwood 3c7ac2df23
Merge pull request #196 from daos-stack/daos-devel 
update DFS driver with new DAOS API changes.
 2019-10-28 16:56:29 -07:00
Glenn K. Lockwood 4409082064
Merge pull request #188 from pkoutoupis/feature-global_default_dir_layout 
Enable global default dir layout for subdirs in Lustre
 2019-10-28 10:43:56 -07:00
Mohamad Chaarawi 92180e022d update DFS read/write APIs with new DAOS API changes 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-10-28 16:42:44 +00:00
jschwartz-cray ccba3b11e5
Merge branch 'master' into fix-194 2019-10-23 16:46:00 -06:00
Josh Schwartz 61333af822 Fix #194. 
On previous versions of IOR (and most Linux CLI applications in general)
multiple flags can be combined such as -vvv to get verbose=3 rather than
having to specify each separately as -v -v -v.  This patch fixes this behavior.

It also fixes issues around handling of the verbose flag where the
output would never be printed as we were comparing verbose < VERBOSE_0
where verbose defaults to 0 and can only increase.
 2019-10-23 16:17:37 -06:00
Julian Kunkel b334c3f6fd
Merge pull request #192 from daos-stack/daos-devel 
fix -R option to do proper data verification
 2019-10-16 16:20:43 +01:00
Mohamad Chaarawi dc82a1bf2f ior -R should memset the buffer being read, otherwise a read that does nothing 
will report success in the data verification phase.

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-10-16 14:54:00 +00:00
Julian M. Kunkel a69a5916cf Hotfix for ior -F -u 2019-10-16 09:42:10 +01:00
Glenn K. Lockwood f40b0744ff
Merge pull request #187 from daos-stack/daos-devel 
IOR: Add latency and IOPS numbers to each iteration.
 2019-10-02 13:09:26 -07:00
Petros Koutoupis b3ecd756ce Implemented AC_CHECK_HEADERS fix in style of #190 2019-09-30 04:25:59 -07:00
Mohamad Chaarawi 2aea04c8cb fix some compiler warnings. 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-09-21 22:43:20 +00:00
Mohamad Chaarawi fca0a62ccb update to new DAOS API for creating dfs containers. 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-09-21 22:05:54 +00:00
Julian Kunkel 604598ab2f
Merge pull request #182 from jschwartz-cray/fix-181 
Fix #181.
 2019-09-19 16:53:41 +01:00
Mohamad Chaarawi 12284ae04a Add latency and iops numbers to each iteration. 
- Latency reported is computed by taking the average latency of
  all ops from a single task, then taking the minimum of that
  between all tasks.

- IOPS is computed by taking the total number of ops across all
  tasks divided by the total access time to execute those ops.

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-09-10 18:49:32 +00:00
Mohamad Chaarawi c58ba8ffb5 add break that was accidentally removed 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-09-10 14:11:55 +00:00
Mohamad Chaarawi fe9d76ddf3 fix segfault when no API is specified to mdtest 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-09-09 22:03:55 +00:00
Mohamad Chaarawi 73dbda09c6 remove printf added 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-09-09 20:19:42 +00:00
Mohamad Chaarawi 5622aabf46 missed Makefile.am update 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-09-09 20:14:12 +00:00
Mohamad Chaarawi d332d586bf remove uneeded code and doc 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-09-09 20:07:55 +00:00
Mohamad Chaarawi e9dd5fe15e Merge branch 'master' into daos-devel 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	configure.ac
	src/Makefile.am
	src/aiori-MPIIO.c
	src/aiori.c
	src/aiori.h
	src/ior.c
	src/mdtest.c
	src/option.c
 2019-09-09 19:55:56 +00:00
Mohamad Chaarawi 57a16ddda8 - update debugging error checking in DFS. 
- remove usleep() before daos_fini()
- fix README

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-09-09 19:14:35 +00:00
Julian Kunkel 0921556456
Merge pull request #184 from hpc/feature-aiori 
Feature aiori
 2019-09-09 15:20:43 +01:00
Petros Koutoupis 36d13d3b97 Enable global default dir layout for subdirs in Lustre 2019-09-02 22:26:12 -07:00
Julian M. Kunkel c83edfe39b Extracted check function into aiori. #24. #177 2019-09-01 15:59:52 +01:00
Julian Kunkel ced9b0183f
Merge pull request #172 from hpc/fix-168 
Fix #168
 2019-09-01 15:51:44 +01:00
Julian Kunkel 151102ff8f
Merge pull request #176 from otatebe/master 
abstract I/O interface for Gfarm file system
 2019-09-01 15:51:16 +01:00
Julian M. Kunkel e3db1759b2 Moded sync() to aiori backend. 2019-09-01 15:47:42 +01:00
Julian M. Kunkel 6de1ea3176 Merge branch 'feature-sync' into feature-aiori 2019-09-01 15:39:49 +01:00
Julian M. Kunkel 60a641f911 sync help corrected. 2019-09-01 15:29:12 +01:00
Julian M. Kunkel 10d3db1dc8 MDTest: fixing the memset() to account for the number of iterations. 2019-08-31 17:28:08 +01:00
Julian Kunkel 47695aea49
Merge pull request #180 from jschwartz-cray/fix-179 
Fix #179.
 2019-08-31 10:32:41 +01:00
Josh Schwartz 0e952f0f8c Fix #181. 
On systems where numTasks is not evenly divisible by 'tasksPerNode' we were
seeing some nodes reading multiple files while others read none after
reordering.

Commonly all nodes have the same number of tasks but there is nothing
requiring that to be the case.  Imagine having 64 tasks running against 4
nodes which can run 20 tasks each.  Here you get three groups of 20 and one
group of 4.  On this sytem nodes running in the group of 4 were previously
getting tasksPerNode of 4 which meant they reordered tasks differently than
the nodes which got tasksPerNode of 20.

The key to fixing this is ensuring that every node reorders tasks the same
way, which means ensuring they all use the same input values.  Obviously on
systems where the number of tasks per node is inconsistent the reordering will
also be inconsistent (some tasks may end up on the same node, or not as far
separated as desired, etc.) but at least this way you'll always end up with a
1:1 reordering.

- Renamed nodes/nodeCount to numNodes
- Renamed tasksPerNode to numTasksOnNode0
- Ensured that numTasksOnNode0 will always have the same value regardless of
  which node you're on
- Removed inconsistently used globals numTasksWorld and tasksPerNode and
  replaced with per-test params equivalents
- Added utility functions for setting these values:
  - numNodes -> GetNumNodes
  - numTasks -> GetNumTasks
  - numTasksOnNode0 -> GetNumNodesOnTask0
- Improved MPI_VERSION < 3 logic for GetNumNodes so it works when numTasks is
  not evenly divisible by numTasksOnNode0
- Left 'nodes' and 'tasksPerNode' in output alone to not break compatibility
- Allowed command-line params to override numTasks, numNodes, and
  numTasksOnNode0 but default to using the MPI-calculated values
 2019-08-30 16:45:03 -06:00
Josh Schwartz 4c3d96bfed Fix #179. 
-u (uniqueDir) will once again use the full file path specified by the
client instead of truncating it.  This was caused by a broken sprintf
which was trying to read/write overlapping buffers.

From the glibc sprintf() documentation:
"The behavior of this function is undefined if copying takes place
between objects that overlap"
 2019-08-30 15:31:23 -06:00
Josh Schwartz 0bd979637e Added some extra debug including ERRF, WARNF, and MPI_CHECKF format string macros. 2019-08-30 15:11:25 -06:00
Mohamad Chaarawi 32db1cd902 add timing for container close. 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-08-29 03:24:48 +00:00
Mohamad Chaarawi 93730771fd add some verbose messages on finalize routines for DAOS and DFS drivers. 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-08-28 17:01:51 +00:00
Julian M. Kunkel 4df051bf28 New option -Y to invoke the sync command. 2019-08-26 18:57:14 +01:00
Julian M. Kunkel a4068be551 Improved help for fsync. 2019-08-26 15:57:13 +01:00
Julian M. Kunkel 0d9f46e980 MDTest re-added the -Z option for compatibility (for now) and switched back behavior. 2019-08-15 16:49:46 +01:00
Julian M. Kunkel de3baf8861 MDTest: Document choice of 42. 2019-08-15 16:21:30 +01:00
Mohamad Chaarawi b3663bd29a add sleep depending on MPI rank to avoid all ranks calling daos_fini() 
at once (issue with PSM2).

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-08-14 13:32:51 +00:00
Mohamad Chaarawi 1320aa279c add some barriers before cont close and destroy to make sure all 
ranks are done.

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-08-12 16:04:20 +00:00
Mohamad Chaarawi 8cb878507e Add dfs chunk_size and oclass options. 
update dfs_remove for API change.

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-08-12 14:25:05 +00:00
Axel Huebl 70e8b13d1d
Fix Last Free 2019-08-04 18:27:20 -05:00
Julian Kunkel da5f091afc
Merge pull request #169 from ax3l/fix-singleRankHeapBufferOverflow 
Fix: Heap Buffer Overflow
 2019-08-03 09:20:15 +01:00
Julian Kunkel 5e6a03442f
Merge pull request #170 from ax3l/fix-someMemleaks 
Fix Some Memory Leaks; Thanks.
 2019-08-03 09:19:08 +01:00
Julian M. Kunkel c4ff3d7c4e Trivial fix for #168 2019-08-03 09:12:48 +01:00
Mohamad Chaarawi f16ef9ace5 update dfs_lookup() call for extra parameter. 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-08-03 05:07:13 +00:00
Axel Huebl b72b51be48
Fix: Heap Buffer Overflow 
Fix a memory violation when run in serial.
 2019-08-02 23:34:13 -05:00
Axel Huebl bfff0df8fd
Fix Some Memory Leaks 
Fixing some memory leaks :)
 2019-08-02 23:33:01 -05:00
Osamu Tatebe b1b66962ac incorrect warning 2019-08-02 13:03:59 +09:00
Julian M. Kunkel cf56715a5a Make sure that each read buffer contains an invalid first byte. 2019-08-01 18:33:44 +01:00
Julian M. Kunkel ce1ae750f6 MDtest: Support to verify the read operation with a default pattern. 2019-08-01 18:29:32 +01:00
Julian M. Kunkel df8355a9bc Added output of mdtest stonewall timer. 2019-08-01 17:57:45 +01:00
Julian M. Kunkel 061b5a860f Backmerged: New option: print rate AND time; improves debugging. 2019-08-01 17:54:11 +01:00
Julian M. Kunkel 6c0fadc2a9 Include performance when stonewall is hit to output. 2019-08-01 17:20:01 +01:00
John Bent 3890b71b54 Fixed issues and followed suggestions from Glenn's review of the PR 2019-08-01 09:42:03 +09:00
Mohamad Chaarawi 92939e4fbd update for DAOS API changes 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-07-31 17:22:20 +00:00
John Bent a3c37808da Made FAIL take variable args so we can pass printf like args to it 2019-07-28 11:17:11 -06:00
John Bent 0ffec67d2b Following Julian's suggestion about better naming 2019-07-28 10:25:42 -06:00
John Bent b2d486f749 Followed Andreas suggestion to replace escape double quotes within printf's with single quotes 2019-07-28 10:07:03 -06:00
John Bent 168a407793 Fixed inconsistent spacing that Andreas commented upon 2019-07-28 09:55:00 -06:00
John Bent 629ff810b7 Got IOR shifting to work regardless of whether node/task mapping is round-robin or contiguous 2019-07-27 15:27:20 -06:00
John Bent d69957e55b Final changes cleaning up the output messages 2019-07-27 14:31:49 -06:00
John Bent e767ef3de9 Remove extraneous print_help function that was causing people to have to edit the same string in two different locations 2019-07-27 13:26:39 -06:00
John Bent f6491fcd37 Cleaned up the verbose messages by creating a macro and a function 2019-07-27 13:22:15 -06:00
John Bent 9d6480d46e Fixed bug in the nstride calculation where only 0 was computing it correctly. Had to bcast it out 2019-07-27 11:30:07 -06:00
John Bent 945487f743 Better debug message when stat fails 2019-07-27 09:20:20 -06:00
John Bent 524d053be1 Making shift work in mdtest as well as it works in IOR and on a per-node basis. 
Also added printing the nodemap so we can check the allocation.
 2019-07-26 08:55:24 -06:00
John Bent 7981fc70b1 Changed the behavior of shift in mdtest to mimic that in IOR so that it shifts by _entire nodes_ not by ranks. 2019-07-25 19:18:52 -06:00
Julian M. Kunkel 76f6bb6e42 Abort when invalid options are set. 2019-07-24 00:29:58 +01:00
Mohamad Chaarawi 1c640716c7 Use MPI Info object for all MPI_File_open calls. 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-07-08 21:31:00 +00:00
Mohamad Chaarawi ad61fdd89c add proper check for DAOS object class and exit if unknown. 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-07-08 18:18:24 +00:00
Mohamad Chaarawi 04e3df0d42 use the daos provided function for name to object class 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-07-08 17:02:30 +00:00
Mohamad Chaarawi 52ed766df4 update DAOS and DFS drivers to new DAOS API changes. 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-07-08 15:33:06 +00:00
Wei-keng Liao 3eb0f86640 Function ncmpi_get_file_info was first introduced in 
PnetCDF version 1.2.0. Check PnetCDF library version,
so printing MPI-IO hints can be enabled for NCMPI.
 2019-06-28 17:40:35 -05:00
Mohamad Chaarawi 9f9730d6fd should abort on daos init if pool/cont/svcl not specified. 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-06-28 16:20:56 +00:00
Mohamad Chaarawi 2ae7c29de2 cache the parent dir in a hashtable so we don't have to do a lookup 
on every access.

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-06-27 21:31:07 +00:00
Mohamad Chaarawi 960005351d update DAOS and DFS markings. 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-06-25 22:41:30 +00:00
Mohamad Chaarawi 262d35d87e - Merge branch 'master' into daos-devel 
- fix bugs with cont destroy.
- add destroy option to DFS driver
- share pool and container handle with DFS driver, and allow multi rank access

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	src/ior.c
	src/mdtest.c
	src/option.c
	src/parse_options.c
 2019-06-24 21:26:15 +00:00
Osamu Tatebe a822363ee0 style fix 2019-06-22 10:48:41 +09:00
Osamu Tatebe 6cae44b512 abstract I/O interface for Gfarm file system 2019-06-21 21:05:41 +09:00
Mohamad Chaarawi 915283bae8 update array_g2l usage with DAOS API change 
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
 2019-06-17 20:38:02 +00:00