Commit Graph

64 Commits (master)

Author SHA1 Message Date
Julian Kunkel a647f48b38
AIORI POSIX use open flag. #363 (#364)
Fixes #363
2021-04-26 09:10:47 -07:00
Julian Kunkel 74df77430f
Feature ior gpu #284 (#323)
* Basic support for memory allocation on GPU using CUDA unified memory. Partially addressing #284. IOR support completed.
* Support for GPU alloc in MDTest and MD-Workbench
* Option: support repeated parsing of same option (allows option sharing across modules).
* Checks for gpuDirect
* Integrate gpuDirect options and basic hooks, more testing to be done.
* POSIX: basic gpuDirect implementation working with fake-gpudirect library.
* CUDA allow setting of DeviceID for IOR (not yet MDTest).
* CUDA/GPUDirect Support --with-X=<path>
* Bugfix in option parser for flags that are part of an argument for an option, e.g., -O=1, if 1 is a flag it is wrongly assumed to be a flag.
2021-02-18 10:40:42 +00:00
Julian Kunkel 310fd37427
Feature mdtest dir rename #306 (#311)
* MDTest: Support for directory renaming.
* Refactored some MDTest variables from hardcoded numbers to ENUM symbols.
2021-01-21 14:10:23 +00:00
Julian M. Kunkel 852ee3e40e AIORI POSIX use internal debug macros, include errno in msg. 2021-01-21 11:06:13 +00:00
Julian M. Kunkel 980ab1dc97 Lustre stripping: Fix default value such that Lustre striping is only set if anything is changed. 2020-11-26 15:56:34 +00:00
Julian M. Kunkel d9c74af8f3 Fix (accidently rename of option) 2020-11-04 19:27:39 +00:00
Julian M. Kunkel 82d20f2744 Basic API converted. 2020-11-02 20:12:15 +00:00
Julian M. Kunkel dc89a59371 Workbench, adjusting the interface to IOR tool interfaces. 2020-11-02 19:23:13 +00:00
Julian Kunkel cc0ac50086
POSIX GFPS adjusted to new API #248 (#253) 2020-09-02 10:08:52 +01:00
Julian M. Kunkel f275671cc9 AIORI POSIX AIO support. Collect ops until granularity is reached, then submit pending IOs. Synchronize latest on close. Doesn't work with data verification and reuses the existing buffer. The implementation shows the potential AIO may have.
Extract also the POSIX header from AIORI to allow better reuse. #240
2020-07-21 16:18:50 +01:00
Julian M. Kunkel 8ca388fc78 Merge branch 'feature-cleanup-aiori' 2020-07-21 09:26:03 +01:00
Karsten Weiss f280123d0b Spelling fixes (found by codespell) 2020-07-03 09:16:30 +02:00
Julian M. Kunkel 82417128cd Extracted file size routine that is replicated in each module into IOR removing the MPI dependency from AIORI. 2020-06-30 12:33:56 +01:00
Julian M. Kunkel 0bffd14de7 Added --warningAsErrors option to IOR and MDTest and refactored WARNINGs in IOR. #174 2020-06-24 11:13:12 +01:00
Julian M. Kunkel c55848f14d Bugfix based on the pull request for fix-189. Should fix #189. 2020-06-24 10:15:31 +01:00
Julian M. Kunkel eb2cd27fac Removed offset from IOR structure. 2020-06-10 17:47:07 +01:00
Julian M. Kunkel 5663593919 Moved generic IOR information to "hint" structure. Backends should only use the "hints" if set. 2020-05-31 12:50:03 +01:00
Julian M. Kunkel 8098c3740a Refactored fd into aiori_fd_t for type safety. 2020-05-31 12:12:49 +01:00
Julian M. Kunkel c4c19d3245 Fixed type typo in name. 2020-05-31 11:58:34 +01:00
Julian M. Kunkel 294b8891e6 Created dummy option type to provide type safety. Fixed various call issues for options. 2020-05-31 11:50:15 +01:00
Julian M. Kunkel e91b79cea0 Moved Lustre/GPFS/BeeGFS options into POSIX backend. Needs testing. Additional minor fixes. 2020-05-30 20:09:37 +01:00
Julian M. Kunkel 1890aaaeb0 Started to move IOR specific knowledge out of AIORI.
For now, and compatibility, IOR options can still be set/internally accessed using the backends init_xfer_options.
This should be removed in the long run to strip away this dependency.
2020-05-30 18:19:48 +01:00
Glenn K. Lockwood 7a2d307c74
Merge pull request #208 from jyvet/mknod
Fix confusing output of mknod
2019-12-16 12:38:13 -08:00
Jean-Yves VET 9da36abaca Fix confusing output of mknod
Context: mknod does not return a file descriptor.
2019-12-16 14:55:26 +01:00
Jean-Yves VET be48206e5f Fix compilation warnings 2019-12-16 14:51:10 +01:00
Julian M. Kunkel e3db1759b2 Moded sync() to aiori backend. 2019-09-01 15:47:42 +01:00
Josh Schwartz 0bd979637e Added some extra debug including ERRF, WARNF, and MPI_CHECKF format string macros. 2019-08-30 15:11:25 -06:00
Axel Huebl 70e8b13d1d
Fix Last Free 2019-08-04 18:27:20 -05:00
Axel Huebl bfff0df8fd
Fix Some Memory Leaks
Fixing some memory leaks :)
2019-08-02 23:33:01 -05:00
Gu Zheng a6f8388f9f add mknod support
Add new option '-k' to support file creation test with mknod,
which is widely used in lustre.

Signed-off-by: Gu Zheng <gzheng@ddn.com>
2019-05-23 13:41:46 +08:00
Glenn K. Lockwood 9ead56c24e
Merge pull request #148 from hpc/fix-146
Moving options to submodules + fixes #146 + fixes #144
2019-05-14 14:32:37 -07:00
Oliver Steffen e3afdb5ed5 beegfs: don't fail if output exists
Use existing output file if it has the desired stripe settings
(number of targets, chunk size)
2019-04-29 15:52:26 +02:00
Julian M. Kunkel 27ac1b88af Removed verbose output. 2019-04-02 12:34:35 +01:00
Julian M. Kunkel 07217c93aa Bugfix for the parsing of flags. Backwards compatibility for direct IO. 2019-04-02 12:08:31 +01:00
Julian M. Kunkel 21405ed924 Updated MMap module to option parser. 2019-03-27 22:32:59 +00:00
Julian M. Kunkel 07ca5247d7 Checked proper IOR behavior for -f option. 2019-03-27 20:37:46 +00:00
Julian M. Kunkel 20e960d020 Support the setting of module-specific options per test. 2019-03-27 20:04:48 +00:00
Jean-Yves VET 110153cf3a mdtest: Enable IME native interface
This patch makes mdtest allow IME native interface.
It also defines a generic way to enable mdtest with each
backend.
2019-02-12 15:50:51 +01:00
Andreas Dilger 1f217ae48c lustre: fix checks for lustre_user.h header
Fix the configure check if --with-lustre is specified, but
the linux/lustre/lustre_user.h header is not present.  Only
one of the headers needs to be included if both are found.

In some cases, FASYNC is not defined, but forms part of the
O_LOV_DELAY_CREATE value, add a #define in that case.

Fixes #115.

Fixes: cb88c4c19a831d94b864c49a162e2635730540e5
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
2018-12-14 16:45:43 -07:00
Julian M. Kunkel 6c5952906b Dry-run option to test the configuration / scripts. 2018-10-11 19:58:30 +01:00
Jean-Yves VET f472162784 Support old legacy name in backends
Context: Some backends may have used different names in
the past (like IME backend use to be IM). Legacy scripts
may break.

This patch adds a legacy name option in the aiori structure.
Both name and legacy name work to select the interface.
But the following warning is printed if the legacy name is used:

ior WARNING: [legacy name] backend is deprecated use [name] instead.
2018-09-24 11:51:00 +02:00
Rob Latham cb88c4c19a including lustre_user now deprecated
put in a configure test (same one used in ROMIO) in case someone's using
old Lustre client headers
2018-08-28 16:27:21 -05:00
Julian M. Kunkel 54e47cf729 IOR: use new option parser.
TODO: Parse "-O" options
2018-07-14 08:41:35 +01:00
Julian M. Kunkel 205b20f0f4 Bugfix merge 2018-07-12 16:13:40 +01:00
Shane Snyder 8069e018e3 make ncmpi/hdf5 use mpiio access/delete
Signed-off-by: Julian M. Kunkel <juliankunkel@googlemail.com>
2018-07-08 12:13:38 +01:00
Li Dongyang 6ba22a19e4 Add mmap as a new AIORI backend
It shares the create/open/delete/set_version/get_file_size
functions with POSIX backend.
The mmap backend also supports fsync and fsyncPerWrite options,
and it will use msync() instead and fsync().

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
2018-05-11 12:13:42 +10:00
Nathan Hjelm 4dc0081628 First cut at using the IOR backends for mdtest
This commit makes changes to the AIORI backends to add support for
abstacting statfs, mkdir, rmdir, stat, and access. These new
abstractions are used by a modified mdtest. Some changes:

 - Require C99. Its 2017 and most compilers now support C11. The
   benefits of using C99 include subobject naming (for aiori backend
   structs), and fixed size integers (uint64_t). There is no reason to
   use the non-standard long long type.

 - Moved some of the aiori code into aiori.c so it can be used by both
   mdtest and ior.

 - Code cleanup of mdtest. This is mostly due to the usage of the IOR
   backends rather than a mess of #if code.

Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
2017-10-19 15:29:12 -06:00
Oliver Steffen 738190bd79 add support for tuning BeeGFS parameters 2017-10-04 14:14:36 +02:00
Jeff Inman 37738dab26 Numerous changes to file-modes, small build-tweaks, and a tweak to aiori-S3.c
(Only rank-0 should create the bucket, if it doesn't already exist.)

Prepping this for a push (as an experimental S3 build) to github.
2015-05-19 09:36:28 -06:00
Jeffrey Thornton Inman ca801fe0ec First-cut at PLFS support, integrating Brett K's code.
Builds, but not running yet.  Requires 'configure ... --with-plfs', so it seems okay to leave this in as an optional build, pending further work.
2014-09-18 11:20:37 -06:00