Commit Graph

179 Commits (f280123d0b08c7f54987a33f3209c20d8e288b1f)

Author SHA1 Message Date
Karsten Weiss f280123d0b Spelling fixes (found by codespell) 2020-07-03 09:16:30 +02:00
Julian M. Kunkel 38b1752bda JSON issue with results. Fix for #233. 2020-06-25 14:37:52 +01:00
Julian M. Kunkel 0bffd14de7 Added --warningAsErrors option to IOR and MDTest and refactored WARNINGs in IOR. #174 2020-06-24 11:13:12 +01:00
Julian Kunkel d5de9bcc77
Merge branch 'master' into master 2020-06-24 09:50:03 +01:00
Julian Kunkel 6a88b37171
Merge branch 'master' into fix-194 2020-06-24 09:43:37 +01:00
Mohamad Chaarawi b4a54c34e8 Merge branch 'master' into fix-modules
Conflicts:
	src/aiori-DAOS.c
	src/aiori-DFS.c
2020-06-11 19:55:23 +00:00
Julian M. Kunkel eb2cd27fac Removed offset from IOR structure. 2020-06-10 17:47:07 +01:00
Mohamad Chaarawi 3623f3b8bd DFS: allow multiple IO phases when ior_run() is called
- init/fini can be made multiple times
- reset tunables on fini so they can change on a re-init

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-05-31 16:20:58 +00:00
Julian M. Kunkel 8fa8ef0c02 Call backend statfs() instead of only POSIX-specific variant. 2020-05-31 13:30:31 +01:00
Julian M. Kunkel 5663593919 Moved generic IOR information to "hint" structure. Backends should only use the "hints" if set. 2020-05-31 12:50:03 +01:00
Julian M. Kunkel 8098c3740a Refactored fd into aiori_fd_t for type safety. 2020-05-31 12:12:49 +01:00
Julian M. Kunkel 294b8891e6 Created dummy option type to provide type safety. Fixed various call issues for options. 2020-05-31 11:50:15 +01:00
Julian M. Kunkel e91b79cea0 Moved Lustre/GPFS/BeeGFS options into POSIX backend. Needs testing. Additional minor fixes. 2020-05-30 20:09:37 +01:00
Julian M. Kunkel 930ccdc68d Updated HDF5 to new module specification. 2020-05-30 19:01:20 +01:00
Julian M. Kunkel 1890aaaeb0 Started to move IOR specific knowledge out of AIORI.
For now, and compatibility, IOR options can still be set/internally accessed using the backends init_xfer_options.
This should be removed in the long run to strip away this dependency.
2020-05-30 18:19:48 +01:00
Julian M. Kunkel 153b7aa60f Critical bugfix for ior API (used e.g. in IO500)
TODO: Check potential issue when using IOR scripts with different backends.
2020-05-28 21:18:44 +01:00
Julian M. Kunkel a536649abb Clarified some issues in the test framework and added example test. #219 2020-05-20 17:46:35 +01:00
Jean-Yves VET a21137be3e Display outlier host names
Display host names when using the -j argument (outlierThreshold) so
that IOR could help to better identify slow nodes:

WARNING: for client02, task 6, write elapsed transfer time is 65.738326
2020-03-17 16:31:56 +01:00
Glenn K. Lockwood 657ff8ad8f
Merge pull request #217 from markhpc/wip-aiori-cephfs
[RFC] src/aiori-CEPHFS: New libcephfs backend
2020-03-10 12:29:24 -07:00
Mark Nelson 9649a0c520 src/aiori-CEPHFS: New libcephfs backend
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-03-10 10:19:41 -05:00
Julian M. Kunkel 9726aa0ce2 Bugfix API which doesn't set the AIORI backend. 2020-03-07 14:40:59 +00:00
Julian M. Kunkel c702a98376 Strict prototypes for AIORI 2019-12-22 11:21:40 +00:00
Julian M. Kunkel 73c5c0efc2 Fix for collective operations by Wolfgang Szoecs #210.
It does have performance implications but is only active if needed.
2019-12-19 13:43:38 +00:00
Jean-Yves VET be48206e5f Fix compilation warnings 2019-12-16 14:51:10 +01:00
Afrian Jackson 46c5d4e78d Adding in dual mount functionality for NVRAM within nodes 2019-11-05 14:37:54 +00:00
jschwartz-cray ccba3b11e5
Merge branch 'master' into fix-194 2019-10-23 16:46:00 -06:00
Josh Schwartz 61333af822 Fix #194.
On previous versions of IOR (and most Linux CLI applications in general)
multiple flags can be combined such as -vvv to get verbose=3 rather than
having to specify each separately as -v -v -v.  This patch fixes this behavior.

It also fixes issues around handling of the verbose flag where the
output would never be printed as we were comparing verbose < VERBOSE_0
where verbose defaults to 0 and can only increase.
2019-10-23 16:17:37 -06:00
Julian Kunkel b334c3f6fd
Merge pull request #192 from daos-stack/daos-devel
fix -R option to do proper data verification
2019-10-16 16:20:43 +01:00
Mohamad Chaarawi dc82a1bf2f ior -R should memset the buffer being read, otherwise a read that does nothing
will report success in the data verification phase.

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-10-16 14:54:00 +00:00
Julian M. Kunkel a69a5916cf Hotfix for ior -F -u 2019-10-16 09:42:10 +01:00
Glenn K. Lockwood f40b0744ff
Merge pull request #187 from daos-stack/daos-devel
IOR: Add latency and IOPS numbers to each iteration.
2019-10-02 13:09:26 -07:00
Julian Kunkel 604598ab2f
Merge pull request #182 from jschwartz-cray/fix-181
Fix #181.
2019-09-19 16:53:41 +01:00
Mohamad Chaarawi 12284ae04a Add latency and iops numbers to each iteration.
- Latency reported is computed by taking the average latency of
  all ops from a single task, then taking the minimum of that
  between all tasks.

- IOPS is computed by taking the total number of ops across all
  tasks divided by the total access time to execute those ops.

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-09-10 18:49:32 +00:00
Mohamad Chaarawi e9dd5fe15e Merge branch 'master' into daos-devel
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	configure.ac
	src/Makefile.am
	src/aiori-MPIIO.c
	src/aiori.c
	src/aiori.h
	src/ior.c
	src/mdtest.c
	src/option.c
2019-09-09 19:55:56 +00:00
Julian Kunkel 0921556456
Merge pull request #184 from hpc/feature-aiori
Feature aiori
2019-09-09 15:20:43 +01:00
Julian M. Kunkel c83edfe39b Extracted check function into aiori. #24. #177 2019-09-01 15:59:52 +01:00
Julian Kunkel 151102ff8f
Merge pull request #176 from otatebe/master
abstract I/O interface for Gfarm file system
2019-09-01 15:51:16 +01:00
Julian Kunkel 47695aea49
Merge pull request #180 from jschwartz-cray/fix-179
Fix #179.
2019-08-31 10:32:41 +01:00
Josh Schwartz 0e952f0f8c Fix #181.
On systems where numTasks is not evenly divisible by 'tasksPerNode' we were
seeing some nodes reading multiple files while others read none after
reordering.

Commonly all nodes have the same number of tasks but there is nothing
requiring that to be the case.  Imagine having 64 tasks running against 4
nodes which can run 20 tasks each.  Here you get three groups of 20 and one
group of 4.  On this sytem nodes running in the group of 4 were previously
getting tasksPerNode of 4 which meant they reordered tasks differently than
the nodes which got tasksPerNode of 20.

The key to fixing this is ensuring that every node reorders tasks the same
way, which means ensuring they all use the same input values.  Obviously on
systems where the number of tasks per node is inconsistent the reordering will
also be inconsistent (some tasks may end up on the same node, or not as far
separated as desired, etc.) but at least this way you'll always end up with a
1:1 reordering.

- Renamed nodes/nodeCount to numNodes
- Renamed tasksPerNode to numTasksOnNode0
- Ensured that numTasksOnNode0 will always have the same value regardless of
  which node you're on
- Removed inconsistently used globals numTasksWorld and tasksPerNode and
  replaced with per-test params equivalents
- Added utility functions for setting these values:
  - numNodes -> GetNumNodes
  - numTasks -> GetNumTasks
  - numTasksOnNode0 -> GetNumNodesOnTask0
- Improved MPI_VERSION < 3 logic for GetNumNodes so it works when numTasks is
  not evenly divisible by numTasksOnNode0
- Left 'nodes' and 'tasksPerNode' in output alone to not break compatibility
- Allowed command-line params to override numTasks, numNodes, and
  numTasksOnNode0 but default to using the MPI-calculated values
2019-08-30 16:45:03 -06:00
Josh Schwartz 4c3d96bfed Fix #179.
-u (uniqueDir) will once again use the full file path specified by the
client instead of truncating it.  This was caused by a broken sprintf
which was trying to read/write overlapping buffers.

From the glibc sprintf() documentation:
"The behavior of this function is undefined if copying takes place
between objects that overlap"
2019-08-30 15:31:23 -06:00
Josh Schwartz 0bd979637e Added some extra debug including ERRF, WARNF, and MPI_CHECKF format string macros. 2019-08-30 15:11:25 -06:00
Osamu Tatebe b1b66962ac incorrect warning 2019-08-02 13:03:59 +09:00
Julian M. Kunkel 6c0fadc2a9 Include performance when stonewall is hit to output. 2019-08-01 17:20:01 +01:00
John Bent 3890b71b54 Fixed issues and followed suggestions from Glenn's review of the PR 2019-08-01 09:42:03 +09:00
John Bent 0ffec67d2b Following Julian's suggestion about better naming 2019-07-28 10:25:42 -06:00
John Bent 629ff810b7 Got IOR shifting to work regardless of whether node/task mapping is round-robin or contiguous 2019-07-27 15:27:20 -06:00
Mohamad Chaarawi 262d35d87e - Merge branch 'master' into daos-devel
- fix bugs with cont destroy.
- add destroy option to DFS driver
- share pool and container handle with DFS driver, and allow multi rank access

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	src/ior.c
	src/mdtest.c
	src/option.c
	src/parse_options.c
2019-06-24 21:26:15 +00:00
Julian M. Kunkel 21405ed924 Updated MMap module to option parser. 2019-03-27 22:32:59 +00:00
Julian M. Kunkel 20e960d020 Support the setting of module-specific options per test. 2019-03-27 20:04:48 +00:00
Mohamad Chaarawi 39eca1bb08 add DAOS_Access routine to check if a container exists before deleting it.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-14 21:17:37 +00:00