Commit Graph

43 Commits (master)

Author SHA1 Message Date
Julian M. Kunkel 306598db67 Bugfix memory issue in global options. 2020-11-06 11:40:31 +00:00
Mohamad Chaarawi cb397242f9
DAOS backend cleanup (#266)
- remove legacy DAOS driver & update Readme.
- update configure options to remove cart requirement
- Optimize DFS file get_size

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-11-03 10:01:09 +00:00
Julian M. Kunkel f275671cc9 AIORI POSIX AIO support. Collect ops until granularity is reached, then submit pending IOs. Synchronize latest on close. Doesn't work with data verification and reuses the existing buffer. The implementation shows the potential AIO may have.
Extract also the POSIX header from AIORI to allow better reuse. #240
2020-07-21 16:18:50 +01:00
Julian M. Kunkel 371335195b Initialize logfile for warnings if not set. 2020-07-01 09:42:13 +01:00
Julian M. Kunkel 81a7a3ab81 Renamed existing S3 impl. which uses (apparently) outdated library. Added dummy support for libS3 with library search. 2020-06-30 13:41:59 +01:00
Julian Kunkel d5de9bcc77
Merge branch 'master' into master 2020-06-24 09:50:03 +01:00
Julian M. Kunkel 8fa8ef0c02 Call backend statfs() instead of only POSIX-specific variant. 2020-05-31 13:30:31 +01:00
Julian M. Kunkel c4c19d3245 Fixed type typo in name. 2020-05-31 11:58:34 +01:00
Julian M. Kunkel 294b8891e6 Created dummy option type to provide type safety. Fixed various call issues for options. 2020-05-31 11:50:15 +01:00
Julian M. Kunkel 1890aaaeb0 Started to move IOR specific knowledge out of AIORI.
For now, and compatibility, IOR options can still be set/internally accessed using the backends init_xfer_options.
This should be removed in the long run to strip away this dependency.
2020-05-30 18:19:48 +01:00
Mark Nelson 9649a0c520 src/aiori-CEPHFS: New libcephfs backend
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-03-10 10:19:41 -05:00
Afrian Jackson fa25c31994 Adding PMDK functionality 2019-10-31 15:29:09 +00:00
Mohamad Chaarawi e9dd5fe15e Merge branch 'master' into daos-devel
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	configure.ac
	src/Makefile.am
	src/aiori-MPIIO.c
	src/aiori.c
	src/aiori.h
	src/ior.c
	src/mdtest.c
	src/option.c
2019-09-09 19:55:56 +00:00
Mohamad Chaarawi 262d35d87e - Merge branch 'master' into daos-devel
- fix bugs with cont destroy.
- add destroy option to DFS driver
- share pool and container handle with DFS driver, and allow multi rank access

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	src/ior.c
	src/mdtest.c
	src/option.c
	src/parse_options.c
2019-06-24 21:26:15 +00:00
Osamu Tatebe 6cae44b512 abstract I/O interface for Gfarm file system 2019-06-21 21:05:41 +09:00
Julian M. Kunkel 20e960d020 Support the setting of module-specific options per test. 2019-03-27 20:04:48 +00:00
Jean-Yves VET 110153cf3a mdtest: Enable IME native interface
This patch makes mdtest allow IME native interface.
It also defines a generic way to enable mdtest with each
backend.
2019-02-12 15:50:51 +01:00
Mohamad Chaarawi 2c87b5e0f5 Merge remote-tracking branch 'upstream/master' into daos
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	src/aiori.c
	src/aiori.h
	src/ior.c
	src/mdtest-main.c
	src/mdtest.c
	src/option.c
2019-01-24 00:31:12 +00:00
Glenn K. Lockwood 5e2e1cc96f
Merge pull request #104 from hpc/feature-hdf5-backend
Changed the parser to fix #98.  Does _not_ yet have the HDF5 backend args updated.  They were reverted to allow this to make the 3.2 release and should be re-applied as a separate commit.
2018-12-04 14:41:31 -06:00
Julian M. Kunkel 5a8071fbd8 GNU99 extracted _XOPEN_SOURCE to config.h. 2018-10-28 18:34:22 +00:00
Julian M. Kunkel 6f7576aa8a Changed the parser.
The parser now supports concurrent parsing of all plugin options.
Moved HDF5 collective_md option into the backend as an example.
Example: ./src/ior -a dummy --dummy.delay-xfer=50000
2018-10-11 18:31:21 +01:00
Glenn K. Lockwood 87c5c9ef04 compile correctly with -std=c99
This patch enables correct compilation on both MacOS and Linux using only
POSIX.1-2008-compliant C with XSI extensions.

Specifically, POSIX.1-2008 is the minimum version because we use strdup(3);
explicit XSI is required to expose putenv from glibc.
2018-10-08 13:47:28 -07:00
Jean-Yves VET f472162784 Support old legacy name in backends
Context: Some backends may have used different names in
the past (like IME backend use to be IM). Legacy scripts
may break.

This patch adds a legacy name option in the aiori structure.
Both name and legacy name work to select the interface.
But the following warning is printed if the legacy name is used:

ior WARNING: [legacy name] backend is deprecated use [name] instead.
2018-09-24 11:51:00 +02:00
Jean-Yves VET 2bcaea564e Initialize only the required backends
Context: IOR initializes all available backends. If one
backend fails to initialize IOR cannot be used.

This patch makes IOR initialize only the backends
which will be used. The initialization is done after
that the parameters are checked so that the help message
can still be dispayed is something goes wrong.
2018-09-18 19:24:29 +02:00
Mohamad Chaarawi 46ff4e26be Merge remote branch 'daos-stack/daos'
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	configure.ac
	src/Makefile.am
	src/aiori.c
	src/aiori.h
	src/ior.c
	src/mdtest.c
	src/parse_options.c
2018-08-29 21:14:19 +00:00
Mohamad Chaarawi f4b03efd72 Add the DAOS ior driver.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-08-27 17:22:38 +00:00
Julian M. Kunkel 4dcf36c1a4 Help text now outputs *compiled* APIs. 2018-08-15 17:07:17 +01:00
Jean-Yves VET 44ce8e5aaf Add backend for DDN's Infinite Memory Engine (IME)
This patch adds the support of IME Native interface
as a new AIORI backend.
2018-08-07 11:41:07 +01:00
Julian M. Kunkel b613bf9396 AIORI update to initialization and fix HDF5 with changed API. 2018-07-14 10:47:25 +01:00
Julian M. Kunkel 5ef95675bf AIORI initialize() and finalize() functions. 2018-07-14 09:29:08 +01:00
Julian M. Kunkel 54e47cf729 IOR: use new option parser.
TODO: Parse "-O" options
2018-07-14 08:41:35 +01:00
Julian M. Kunkel 205b20f0f4 Bugfix merge 2018-07-12 16:13:40 +01:00
Julian Kunkel 060c47e80a
Merge branch 'testing' into rados-hacks 2018-07-12 15:53:04 +01:00
Shane Snyder 7981691a63 add a new AIORI backend for RADOS
inform aiori interface about RADOS backend

stubbed out aiori backend for rados

additions to get RADOS backend compiling/linking

first cut at rados create/open patha

make sure to return RADOS oid on open/create

implement rados xfer path for WRITE

refactor + implement getfilesize and close

remember to use read_op interface for stat

implement RADOS delete function

don't error in RADOS_Delete for now

implement RADOS set_version

handle open/create flags appropriately

cleanup RADOS error handling

implement read/readcheck/writecheck for RADOS

rados doesn't support directio

implement unsupported aiori ops for RADOS

implement RADOS access call

define rados types if no rados support
2018-07-12 09:36:49 -05:00
Julian M. Kunkel 02883d4a63 Moved most output to ior-output.c 2018-07-08 13:38:05 +01:00
Shane Snyder 7d3c7e901e expose generic aiori_ calls for access, etc.
Signed-off-by: Julian M. Kunkel <juliankunkel@googlemail.com>
2018-07-08 12:09:44 +01:00
Julian M. Kunkel 378789737f Dummy AIORI driver 2018-07-07 13:25:17 +01:00
Mohamad Chaarawi 8d0cddd21e Add DFS ior/mdtest driver
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-06-13 18:37:37 +00:00
Li Dongyang 6ba22a19e4 Add mmap as a new AIORI backend
It shares the create/open/delete/set_version/get_file_size
functions with POSIX backend.
The mmap backend also supports fsync and fsyncPerWrite options,
and it will use msync() instead and fsync().

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
2018-05-11 12:13:42 +10:00
Shane Snyder d7fc07163e expose generic aiori_ calls for access, etc. 2018-04-25 17:21:48 -05:00
Nathan Hjelm 6ba323bb69 aiori: accept any case when selecting backend
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
2017-10-19 15:29:27 -06:00
Nathan Hjelm 0c9921dc0d aiori: remove support for plfs
Remove support for plfs by recommendation of its original author.

Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
2017-10-19 15:29:18 -06:00
Nathan Hjelm 4dc0081628 First cut at using the IOR backends for mdtest
This commit makes changes to the AIORI backends to add support for
abstacting statfs, mkdir, rmdir, stat, and access. These new
abstractions are used by a modified mdtest. Some changes:

 - Require C99. Its 2017 and most compilers now support C11. The
   benefits of using C99 include subobject naming (for aiori backend
   structs), and fixed size integers (uint64_t). There is no reason to
   use the non-standard long long type.

 - Moved some of the aiori code into aiori.c so it can be used by both
   mdtest and ior.

 - Code cleanup of mdtest. This is mostly due to the usage of the IOR
   backends rather than a mess of #if code.

Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
2017-10-19 15:29:12 -06:00