Commit Graph

309 Commits (8cb878507e73f57ebe0809ee623dcf95a5417221)

Author SHA1 Message Date
Mohamad Chaarawi 334bf73797 remove transaction commit and options to do that.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-01-17 20:58:58 +00:00
Mohamad Chaarawi d2a0023765 remove epoch usage and rebase with latest DAOS API that switches to transactions.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-01-07 18:28:59 +00:00
Glenn K. Lockwood e71cded6c3
Merge pull request #124 from hpc/fix-hdf5
Bugfix for HDF5 plugin; readded collective_md.
2019-01-02 10:01:56 -08:00
Julian M. Kunkel 1dbca5c293 Bugfix for HDF5 plugin; readded collective_md. 2018-12-23 13:07:23 +00:00
Julian M. Kunkel ed421bcc7b Fixes that issue. Hopefully works for all combinations. 2018-12-20 21:08:55 +00:00
Julian M. Kunkel 7b603bc5a9 Reset stonewall timer to make it work again with running all phases in one execution; i.e. not using multiple runs specifying: -C, -r 2018-12-19 21:37:37 +00:00
Julian M. Kunkel 4297ccd7f2 Bugfix for stonewall computation. 2018-12-19 20:29:47 +00:00
Andreas Dilger 1f217ae48c lustre: fix checks for lustre_user.h header
Fix the configure check if --with-lustre is specified, but
the linux/lustre/lustre_user.h header is not present.  Only
one of the headers needs to be included if both are found.

In some cases, FASYNC is not defined, but forms part of the
O_LOV_DELAY_CREATE value, add a #define in that case.

Fixes #115.

Fixes: cb88c4c19a831d94b864c49a162e2635730540e5
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
2018-12-14 16:45:43 -07:00
Mohamad Chaarawi 72c487013a rebase with latest DAOS
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-12-04 22:59:15 +00:00
Glenn K. Lockwood e1158daaf2
Merge pull request #105 from hpc/feature-test-scripts-85
Dry-run option to test the configuration / scripts.
2018-12-04 15:06:47 -06:00
Glenn K. Lockwood 5e2e1cc96f
Merge pull request #104 from hpc/feature-hdf5-backend
Changed the parser to fix #98.  Does _not_ yet have the HDF5 backend args updated.  They were reverted to allow this to make the 3.2 release and should be re-applied as a separate commit.
2018-12-04 14:41:31 -06:00
Glenn K. Lockwood 5c847f552a
Merge pull request #110 from hpc/fix-gnu99-101
Moved _XOPEN_SOURCE to config.h; resolves #93
2018-12-04 14:13:16 -06:00
Julian M. Kunkel 7db8ffd8ab Reverted the change to HDF5 module back. 2018-12-04 19:16:41 +00:00
Wang Shilong 8902bc8722 mdtest: fix calculation for barriers
Currently, calculation for barriers case looks a bit
suprising, for example, Min stats is from min of
Max value of number of mpi proc, this makes Min
and Max very similar in our testing thus we get
a small Std value too.

I am not a MPI expert, but question is why we
don't use more nature calcuation which calculate
all results from different iterations and MPI proc?

Signed-off-by: Wang Shilong <wshilong@ddn.com>
2018-11-22 19:57:32 +08:00
Julian M. Kunkel 5a8071fbd8 GNU99 extracted _XOPEN_SOURCE to config.h. 2018-10-28 18:34:22 +00:00
Julian Kunkel 3a4f2ea33e
Merge pull request #101 from glennklockwood/issue93-squashed
compile correctly with -std=c99
2018-10-28 18:16:52 +00:00
Julian Kunkel 52182afed6
Merge pull request #108 from hpc/fix-106
Fix 106
2018-10-19 11:06:05 +01:00
Julian M. Kunkel ea3aeea964 Bugfix initialize the variable to zero for proper API usage. 2018-10-17 16:16:46 +01:00
Julian M. Kunkel 6b0b1a9830 Fix computation when using branching factor with -n and -i. 2018-10-17 16:04:49 +01:00
Marc Vef 093113313d mdtest: Use GetTimeStamp() instead of MPI_Wtime()
IOR uses the GetTimeStamp() wrapper to allow gettimeofday() to be used for timings instead of MPI_Wtime() to calculate throughput. This commits adds the same logic to mdtest timings.
2018-10-17 16:38:35 +02:00
Julian M. Kunkel f4afa63ebf Bugfix #106 mostly. 2018-10-17 13:50:04 +01:00
Julian Kunkel f7dc084f32
Merge pull request #103 from DDNStorage/ime_mkdir_rmdir
IME: add support of mkdir/rmdir into the IME backend
2018-10-16 14:34:26 +01:00
Julian M. Kunkel 6c5952906b Dry-run option to test the configuration / scripts. 2018-10-11 19:58:30 +01:00
Julian M. Kunkel 3b5f4fc002 Bugfix JSON output for multiple processes 2018-10-11 18:59:12 +01:00
Julian M. Kunkel fa38cb7992 Bugfix JSON header (allows parsing at line 0). 2018-10-11 18:49:25 +01:00
Julian M. Kunkel 6f7576aa8a Changed the parser.
The parser now supports concurrent parsing of all plugin options.
Moved HDF5 collective_md option into the backend as an example.
Example: ./src/ior -a dummy --dummy.delay-xfer=50000
2018-10-11 18:31:21 +01:00
Sylvain Didelot 912d93e94f aiori-IME: add support of mkdir/rmdir to the backup
ime_native_mkdir() and ime_native_rmdir() are available starting
from IME native API version 1.3.
2018-10-09 17:56:08 +02:00
Sylvain Didelot e6bd9ad3b3 common: call mkdir/access syscalls defined in backends
mkdir/access syscalls used in PrependDir() should call their equivalent
implemented by the backend.
2018-10-09 17:55:58 +02:00
Glenn K. Lockwood 87c5c9ef04 compile correctly with -std=c99
This patch enables correct compilation on both MacOS and Linux using only
POSIX.1-2008-compliant C with XSI extensions.

Specifically, POSIX.1-2008 is the minimum version because we use strdup(3);
explicit XSI is required to expose putenv from glibc.
2018-10-08 13:47:28 -07:00
Julian M. Kunkel 7045295a86 Replaced usleep to nanosleep. 2018-10-06 17:30:00 +01:00
Glenn K. Lockwood 9f56d8b8dd
Merge pull request #89 from DDNStorage/fixwritecheck
Make write verification work even without read test
2018-10-06 09:03:32 -07:00
Julian Kunkel 4f1be703b1
Merge pull request #97 from glennklockwood/issue96
fixes #96
2018-10-06 16:55:05 +01:00
Julian Kunkel c6580dc93b
Merge pull request #99 from glennklockwood/issue87
fixes #87
2018-10-06 16:54:23 +01:00
Glenn K. Lockwood a0ef65e61f stop the segfault when an invalid api is specified in configfile 2018-10-05 23:29:21 -07:00
Glenn K. Lockwood d8d4742523 fixes #87
Leading whitespace is stripped from each line of the ior input file.  This
allows indented comments to be treated as comments.  However it does NOT allow
one to specify excessive whitespace inside of the `ior start` and `ior stop`
magic phrases.

Also added a test to catch regressions in this functionality.
2018-10-05 22:50:34 -07:00
Glenn K. Lockwood 3aa3ff10f5 fixes #96
Call MPI_Abort() if MPI is initialized; otherwise just exit()
2018-10-05 22:20:18 -07:00
Julian Kunkel acee84bedc
Merge pull request #88 from roblatham00/hdf5-coll-md
HDF5: option to enable collective metadata
2018-10-02 16:13:05 +01:00
Glenn K. Lockwood 60820da874
Merge pull request #91 from glennklockwood/makedist-fix
fix various issues related to release packages
2018-10-01 21:30:58 -07:00
Julian M. Kunkel 1f1e56c363 Usleep feature added according to Nathan's suggestions. 2018-09-30 10:01:21 +01:00
Julian Kunkel 0a103147ab
Merge pull request #92 from DDNStorage/backend-legacy
Support old legacy name in backends
2018-09-24 11:38:13 +01:00
Jean-Yves VET f472162784 Support old legacy name in backends
Context: Some backends may have used different names in
the past (like IME backend use to be IM). Legacy scripts
may break.

This patch adds a legacy name option in the aiori structure.
Both name and legacy name work to select the interface.
But the following warning is printed if the legacy name is used:

ior WARNING: [legacy name] backend is deprecated use [name] instead.
2018-09-24 11:51:00 +02:00
Glenn K. Lockwood e448ff266d fix various issues related to release packages
- travis now tests the packaged source to detect missing source/headers
- basic tests are less sensitive to the directory from where they are run
- fixed some missing files from the `make dist` manifest
- updated the format of NEWS to work with `make dist`
2018-09-21 18:35:48 -04:00
Jean-Yves VET 7a7655e959 Fix erroneous write bandwidth with stonewalling
Context: write and read results from the same iteration
use the same length value in Bytes. When stonewalling is
used the size variates depending on the performance of
the access. This leads to wrong max bandwidths reported
for writes as shown in the following example:

    write     10052      ...
    read      9910       ...
    write     10022      ...
    read      9880       ...
    write     10052      ...
    read      9894       ...
    Max Write: 9371.43 MiB/sec (9826.66 MB/sec)
    Max Read:  9910.48 MiB/sec (10391.89 MB/sec)

This patch makes IOR separate variables used for read
and write tests.
2018-09-21 11:43:33 +02:00
Jean-Yves VET 8c727fa99c Simplify and factorize a few functions
This patch factorizes some duplicated code. It also
simplifies the way times are stored to prepare next
patch.
2018-09-21 11:38:44 +02:00
Jean-Yves VET f03e7c6778 Make write verification work even without read test
Context: IOR outputs errors when the '-w -W' flags are used
without '-r'. Write a file using with check write option
should be possible even without setting read.

This patch fixes a condition which was introduced for
HDFS to remove RDWR flag in some particular cases.
Write check was set with the write only flag but it
requieres the read flag.
2018-09-20 18:18:48 +02:00
Rob Latham 7d2464f733 HDF5: option to enable collective metadata
In HDF5-1.10 the library has an option to read and write the metadata of a file collectively.  One has to opt-in to this optimization with a property list but particularly at large scale it can have tremendous performance implications.

Further information:
- https://support.hdfgroup.org/HDF5/docNewFeatures/CollMDataIO/RFC-CollectiveMetadataReads.pdf
- https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetAllCollMetadataOps
2018-09-20 11:12:08 -05:00
Julian Kunkel e5c8a11769
Merge pull request #83 from DDNStorage/initbackends
Initialize only the required backends
2018-09-19 10:27:11 +01:00
Jean-Yves VET 2bcaea564e Initialize only the required backends
Context: IOR initializes all available backends. If one
backend fails to initialize IOR cannot be used.

This patch makes IOR initialize only the backends
which will be used. The initialization is done after
that the parameters are checked so that the help message
can still be dispayed is something goes wrong.
2018-09-18 19:24:29 +02:00
Jean-Yves VET 44e8f03917 Fix bug in computing max write and read bandwidths
This patch fixes a bug when computing the maximum
bandwidths in short summary output.
2018-09-18 19:23:36 +02:00
Julian M. Kunkel fd34b08e05 Added patch fix O_DIRECT again. 2018-09-18 16:48:59 +01:00
Mohamad Chaarawi 0a7b1e38ea fix verbose output in DAOS & DFS drivers
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-09-13 13:32:02 +00:00
Mohamad Chaarawi 2f713e947d fix bug in driver name check for DAOS.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-09-12 22:53:42 +00:00
Jean-Yves VET 151a9daa52 Check provided IO API is valid
Context: IOR gets a segfault if an unsupported API string
is provided.

This patch checks that the I/O backend is supported, otherwise
IOR stops with an error.
2018-09-10 19:16:28 +02:00
Julian M. Kunkel 1a4fd7d095 Fix -i option #81 2018-09-05 09:14:22 +01:00
Mohamad Chaarawi 2668363fb1 - update README for DAOS drivers.
- update flag options in DAOS API
- make svcl option in DFS API the same as DAOS API

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-08-30 14:47:17 +00:00
Julian M. Kunkel 2d42c25f69 Fix issue #79 for the last run in a configuration file. 2018-08-30 15:10:38 +01:00
Mohamad Chaarawi 14d67c19d9 fix DAOS plugin options passing.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-08-29 22:25:48 +00:00
Mohamad Chaarawi 46ff4e26be Merge remote branch 'daos-stack/daos'
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	configure.ac
	src/Makefile.am
	src/aiori.c
	src/aiori.h
	src/ior.c
	src/mdtest.c
	src/parse_options.c
2018-08-29 21:14:19 +00:00
Julian Kunkel ba2f0330b7
Merge pull request #80 from roblatham00/more-fixes
We are getting there ^-^
2018-08-29 21:09:31 +01:00
Julian M. Kunkel c1f3f485ec Bugfix #79 2018-08-29 21:07:34 +01:00
Rob Latham cb88c4c19a including lustre_user now deprecated
put in a configure test (same one used in ROMIO) in case someone's using
old Lustre client headers
2018-08-28 16:27:21 -05:00
Rob Latham 4a32a2cbb3 fix "number of arguments doesn’t match prototype" error 2018-08-28 16:24:34 -05:00
Rob Latham 6b782226c7 fix pnetcdf build issues
It does not look like anyone has built the pnetcdf driver in a long time
2018-08-28 14:29:36 -05:00
Rob Latham 982f266a73 clean up hints in HDF5 error cases 2018-08-28 13:30:51 -05:00
Mohamad Chaarawi f4b03efd72 Add the DAOS ior driver.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-08-27 17:22:38 +00:00
Mohamad Chaarawi 5fb850c810 - update the DFS driver to latest DFS API.
- update cmd line options to add DAOS Pool and Container uuid and SVCL
- Add init/finalize backend functions.

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-08-23 21:58:53 +00:00
Julian M. Kunkel 4dcf36c1a4 Help text now outputs *compiled* APIs. 2018-08-15 17:07:17 +01:00
Shane Snyder fefefa002d make sure to allow fsyncs in RADOS 2018-08-14 17:08:04 -05:00
Shane Snyder dfc86b60b8 add RADOS backend options for pool, config file 2018-08-14 16:56:19 -05:00
Shane Snyder 0ca5f26e84 cleanup RADOS func names and docs 2018-08-14 16:18:35 -05:00
Julian Kunkel dc7187794c
Merge pull request #70 from DDNStorage/fix-ime-backend
Fix compilation of some aiori backends and add new function calls in IME backend
2018-08-10 18:00:37 +01:00
Glenn K. Lockwood 13b885ce21
Merge pull request #71 from DDNStorage/fixwarnings
Fix warnings reported at compilation time
2018-08-09 09:46:24 -07:00
Jean-Yves VET 490a912e6e Add support for new function calls in the IME backend
This patch adds initialize, finalize and stat calls to
the IME backend. statfs, mkdir and rmdir are currently
not supported. This patch also fixes the IME_GetVersion
call.
2018-08-09 13:08:43 +02:00
Jean-Yves VET 9596b4bbe0 Fix compilation of aiori backends requiring external libraries
This patchs makes a few adjustements in the automake file
in order to properly add include and library directories.
Those are required to be able to compile some aiori backends.
2018-08-09 13:08:36 +02:00
Jean-Yves VET df34f024ba Fix warnings reported at compilation time
This patch fixes most of the warnings caused
by unused variables and assignments from incompatible
type.
2018-08-09 13:04:52 +02:00
Julian M. Kunkel 54fa3fa8e9 Fixed wrong output. 2018-08-07 23:28:19 +01:00
Jean-Yves VET 44ce8e5aaf Add backend for DDN's Infinite Memory Engine (IME)
This patch adds the support of IME Native interface
as a new AIORI backend.
2018-08-07 11:41:07 +01:00
Julian M. Kunkel e708681f19 Option: check for missing arg. 2018-08-06 18:38:34 +01:00
Julian M. Kunkel 0870ad78b3 Support shared large runs with limited number of files per directory by creating one directory for a bunch of files.
Therefore, it uses the -I argument (number of files per directory) and divides it by -n (items).
This will be a number of sub directories that are created on the top-level.
2018-07-26 10:38:06 +01:00
Mohamad Chaarawi 07ec65c0d5 remoce rank_list_parse as it is exposed by DAOS API now.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-07-19 21:36:29 +00:00
Julian M. Kunkel 580fb8cdf7 Bugfix offsetof; thanks George for reporting. 2018-07-15 20:23:10 +01:00
Julian M. Kunkel f55761d5d2 Refactored results structure to AoS; allowed to keep results per repeat. 2018-07-15 19:38:17 +01:00
Julian M. Kunkel c7a598a435 New option parser: Fixed overflow for blocksize. Added warning. 2018-07-15 18:37:15 +01:00
Julian M. Kunkel bb6d4fb2a2 Update create for stonewall. 2018-07-15 10:02:45 +01:00
Julian M. Kunkel d4ecc76ee8 Stonewall bugfix for delete/stat. 2018-07-14 16:46:24 +01:00
Julian M. Kunkel c0657866fd Minor fixes for stonewalling options. 2018-07-14 12:22:36 +01:00
Julian M. Kunkel 5d4ac1a0a0 Bugfix parser: wrong datatype 2018-07-14 11:32:39 +01:00
Julian M. Kunkel 04ba846da2 Fix for SetHints (after changing way strings are managed). 2018-07-14 11:11:47 +01:00
Julian M. Kunkel b613bf9396 AIORI update to initialization and fix HDF5 with changed API. 2018-07-14 10:47:25 +01:00
Julian M. Kunkel ac7600d662 Unit-test with automake for library version. Minor fix for library version. 2018-07-14 10:27:31 +01:00
Julian M. Kunkel 5ef95675bf AIORI initialize() and finalize() functions. 2018-07-14 09:29:08 +01:00
Julian M. Kunkel a0b8659cc8 IOR appears to work with the new option parser. 2018-07-14 09:09:48 +01:00
Julian M. Kunkel 5e98fa0408 Dummy: option for xfer delay. 2018-07-14 08:50:27 +01:00
Julian M. Kunkel 54e47cf729 IOR: use new option parser.
TODO: Parse "-O" options
2018-07-14 08:41:35 +01:00
Julian M. Kunkel 0835191372 Option parser replaced.
Benefits: shows currently set options when used with -h
Supports options for child modules, example for aiori-dummy works, RADOS untested.
2018-07-12 18:09:13 +01:00
Julian M. Kunkel 205b20f0f4 Bugfix merge 2018-07-12 16:13:40 +01:00
Julian Kunkel 060c47e80a
Merge branch 'testing' into rados-hacks 2018-07-12 15:53:04 +01:00
Shane Snyder 7981691a63 add a new AIORI backend for RADOS
inform aiori interface about RADOS backend

stubbed out aiori backend for rados

additions to get RADOS backend compiling/linking

first cut at rados create/open patha

make sure to return RADOS oid on open/create

implement rados xfer path for WRITE

refactor + implement getfilesize and close

remember to use read_op interface for stat

implement RADOS delete function

don't error in RADOS_Delete for now

implement RADOS set_version

handle open/create flags appropriately

cleanup RADOS error handling

implement read/readcheck/writecheck for RADOS

rados doesn't support directio

implement unsupported aiori ops for RADOS

implement RADOS access call

define rados types if no rados support
2018-07-12 09:36:49 -05:00
Julian M. Kunkel 831a5ef823 JSON: Slight output adaption. 2018-07-09 16:25:35 +01:00
Julian M. Kunkel 0f17d4860f Pretty printing for normal output. 2018-07-08 16:56:04 +01:00