Commit Graph

72 Commits (ad3bf97304ff1d623c83661521ef80e19e0d9c23)

Author SHA1 Message Date
Julian Kunkel ad3bf97304
Fixed some issues in build system detected using Clang. (#339) 2021-02-28 11:45:32 +00:00
Julian Kunkel 74df77430f
Feature ior gpu #284 (#323)
* Basic support for memory allocation on GPU using CUDA unified memory. Partially addressing #284. IOR support completed.
* Support for GPU alloc in MDTest and MD-Workbench
* Option: support repeated parsing of same option (allows option sharing across modules).
* Checks for gpuDirect
* Integrate gpuDirect options and basic hooks, more testing to be done.
* POSIX: basic gpuDirect implementation working with fake-gpudirect library.
* CUDA allow setting of DeviceID for IOR (not yet MDTest).
* CUDA/GPUDirect Support --with-X=<path>
* Bugfix in option parser for flags that are part of an argument for an option, e.g., -O=1, if 1 is a flag it is wrongly assumed to be a flag.
2021-02-18 10:40:42 +00:00
Julian M. Kunkel 21bf5a5a12 NCMPI ported to current ADIO. Fixed autoconf #303. 2020-12-25 16:49:04 +00:00
Mohamad Chaarawi 938cf2771b DFS: make use of collective open to share file handle
- remove un-needed libdaos_common lib since DAOS driver is removed
- use default container object class when it's not specified

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-12-01 21:36:25 +00:00
Mohamad Chaarawi cb397242f9
DAOS backend cleanup (#266)
- remove legacy DAOS driver & update Readme.
- update configure options to remove cart requirement
- Optimize DFS file get_size

Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-11-03 10:01:09 +00:00
Julian Kunkel 4e2d179052
Detection of system features for number of sockets #245 (#246)
* Configure checks for architecture-specific functions to detect nr of sockets. #245
2020-08-03 12:30:21 +01:00
Julian M. Kunkel f275671cc9 AIORI POSIX AIO support. Collect ops until granularity is reached, then submit pending IOs. Synchronize latest on close. Doesn't work with data verification and reuses the existing buffer. The implementation shows the potential AIO may have.
Extract also the POSIX header from AIORI to allow better reuse. #240
2020-07-21 16:18:50 +01:00
Julian M. Kunkel 81a7a3ab81 Renamed existing S3 impl. which uses (apparently) outdated library. Added dummy support for libS3 with library search. 2020-06-30 13:41:59 +01:00
Julian M. Kunkel c55848f14d Bugfix based on the pull request for fix-189. Should fix #189. 2020-06-24 10:15:31 +01:00
Julian Kunkel d5de9bcc77
Merge branch 'master' into master 2020-06-24 09:50:03 +01:00
Julian M. Kunkel a536649abb Clarified some issues in the test framework and added example test. #219 2020-05-20 17:46:35 +01:00
Julian Kunkel 8bc9a12f80
Merge pull request #221 from daos-stack/mschaara/dfs_l2g
DFS driver update to share DFS mount on init
2020-04-14 10:13:55 +01:00
Mohamad Chaarawi 1b8848ccbc Use l2g-g2l to share the dfs mount instead of opening by each rank.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-04-13 19:59:56 +00:00
Mohamad Chaarawi eed22a4aaf Look into lib64 for cart libraries too.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2020-04-06 21:42:51 +00:00
Mark Nelson 9649a0c520 src/aiori-CEPHFS: New libcephfs backend
Signed-off-by: Mark Nelson <mnelson@redhat.com>
2020-03-10 10:19:41 -05:00
Jeff Olivier 2a4c21e457 Add runpath to configure script for daos and cart
Enables running without setting LD_LIBRARY_PATH
Uses DT_RUNPATH (--enable-new-dtags) so that it can
be overwritten by setting LD_LIBRARY_PATH should
libraries be moved.

Signed-off-by: Jeff Olivier <jeffrey.v.olivier@intel.com>
2020-01-08 16:45:50 -07:00
Julian Kunkel 3a34b2efb7
Merge pull request #200 from daos-stack/daos-devel
DAOS driver update
2019-11-13 10:38:28 +00:00
Mohamad Chaarawi d926bf1c90 update configure.ac to look under lib64 for DAOS libs
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-11-06 23:10:45 +00:00
Adrian Jackson 339bb200c8 Fixing configure so pmdk backend not built without --with-pmdk being specified 2019-11-05 17:44:26 +00:00
Adrian Jackson c24b2f156a
Merge pull request #2 from hpc/master
Merging in latest master changes
2019-11-05 15:00:47 +00:00
Glenn K. Lockwood bdfc43125a
Merge pull request #198 from daos-stack/daos-devel
fix config error with older autoconf/automake versions.
2019-11-04 15:56:44 -08:00
Mohamad Chaarawi 847b8ad408 fix config error with older autocont/automake versions.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-11-04 22:57:47 +00:00
Afrian Jackson fa25c31994 Adding PMDK functionality 2019-10-31 15:29:09 +00:00
Petros Koutoupis b3ecd756ce Implemented AC_CHECK_HEADERS fix in style of #190 2019-09-30 04:25:59 -07:00
Mohamad Chaarawi e9dd5fe15e Merge branch 'master' into daos-devel
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	configure.ac
	src/Makefile.am
	src/aiori-MPIIO.c
	src/aiori.c
	src/aiori.h
	src/ior.c
	src/mdtest.c
	src/option.c
2019-09-09 19:55:56 +00:00
Petros Koutoupis 36d13d3b97 Enable global default dir layout for subdirs in Lustre 2019-09-02 22:26:12 -07:00
Osamu Tatebe a822363ee0 style fix 2019-06-22 10:48:41 +09:00
Osamu Tatebe 6cae44b512 abstract I/O interface for Gfarm file system 2019-06-21 21:05:41 +09:00
Mohamad Chaarawi 4a788a0c23 re-write the DAOS IOR driver to use the Array API
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2019-03-30 19:11:44 +00:00
Mohamad Chaarawi 2c87b5e0f5 Merge remote-tracking branch 'upstream/master' into daos
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	src/aiori.c
	src/aiori.h
	src/ior.c
	src/mdtest-main.c
	src/mdtest.c
	src/option.c
2019-01-24 00:31:12 +00:00
Andreas Dilger 1f217ae48c lustre: fix checks for lustre_user.h header
Fix the configure check if --with-lustre is specified, but
the linux/lustre/lustre_user.h header is not present.  Only
one of the headers needs to be included if both are found.

In some cases, FASYNC is not defined, but forms part of the
O_LOV_DELAY_CREATE value, add a #define in that case.

Fixes #115.

Fixes: cb88c4c19a831d94b864c49a162e2635730540e5
Signed-off-by: Andreas Dilger <adilger@dilger.ca>
2018-12-14 16:45:43 -07:00
Julian M. Kunkel 5a8071fbd8 GNU99 extracted _XOPEN_SOURCE to config.h. 2018-10-28 18:34:22 +00:00
Glenn K. Lockwood 87c5c9ef04 compile correctly with -std=c99
This patch enables correct compilation on both MacOS and Linux using only
POSIX.1-2008-compliant C with XSI extensions.

Specifically, POSIX.1-2008 is the minimum version because we use strdup(3);
explicit XSI is required to expose putenv from glibc.
2018-10-08 13:47:28 -07:00
Rob Latham 7d2464f733 HDF5: option to enable collective metadata
In HDF5-1.10 the library has an option to read and write the metadata of a file collectively.  One has to opt-in to this optimization with a property list but particularly at large scale it can have tremendous performance implications.

Further information:
- https://support.hdfgroup.org/HDF5/docNewFeatures/CollMDataIO/RFC-CollectiveMetadataReads.pdf
- https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetAllCollMetadataOps
2018-09-20 11:12:08 -05:00
Mohamad Chaarawi 46ff4e26be Merge remote branch 'daos-stack/daos'
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>

Conflicts:
	configure.ac
	src/Makefile.am
	src/aiori.c
	src/aiori.h
	src/ior.c
	src/mdtest.c
	src/parse_options.c
2018-08-29 21:14:19 +00:00
Rob Latham cb88c4c19a including lustre_user now deprecated
put in a configure test (same one used in ROMIO) in case someone's using
old Lustre client headers
2018-08-28 16:27:21 -05:00
Mohamad Chaarawi f4b03efd72 Add the DAOS ior driver.
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-08-27 17:22:38 +00:00
Jean-Yves VET 44ce8e5aaf Add backend for DDN's Infinite Memory Engine (IME)
This patch adds the support of IME Native interface
as a new AIORI backend.
2018-08-07 11:41:07 +01:00
Julian M. Kunkel bd253cf058 Added compilation test for HDF5. 2018-07-14 11:23:43 +01:00
Julian M. Kunkel ac7600d662 Unit-test with automake for library version. Minor fix for library version. 2018-07-14 10:27:31 +01:00
Julian M. Kunkel 19a0102181 Silencing makefile by default. 2018-07-14 06:50:55 +01:00
Julian Kunkel 060c47e80a
Merge branch 'testing' into rados-hacks 2018-07-12 15:53:04 +01:00
Shane Snyder 7981691a63 add a new AIORI backend for RADOS
inform aiori interface about RADOS backend

stubbed out aiori backend for rados

additions to get RADOS backend compiling/linking

first cut at rados create/open patha

make sure to return RADOS oid on open/create

implement rados xfer path for WRITE

refactor + implement getfilesize and close

remember to use read_op interface for stat

implement RADOS delete function

don't error in RADOS_Delete for now

implement RADOS set_version

handle open/create flags appropriately

cleanup RADOS error handling

implement read/readcheck/writecheck for RADOS

rados doesn't support directio

implement unsupported aiori ops for RADOS

implement RADOS access call

define rados types if no rados support
2018-07-12 09:36:49 -05:00
Julian M. Kunkel bcaea2a39f Replaced getopt with an alternative version (on some machines optind was broken). 2018-07-07 08:41:33 +01:00
Julian M. Kunkel 2148f68d36 Allow to use README.md instead of README. 2018-07-06 22:42:46 +01:00
Mohamad Chaarawi 1768eff552 update DFS plugin
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-06-20 21:25:22 +00:00
Mohamad Chaarawi 8d0cddd21e Add DFS ior/mdtest driver
Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@intel.com>
2018-06-13 18:37:37 +00:00
Li Dongyang 6ba22a19e4 Add mmap as a new AIORI backend
It shares the create/open/delete/set_version/get_file_size
functions with POSIX backend.
The mmap backend also supports fsync and fsyncPerWrite options,
and it will use msync() instead and fsync().

Signed-off-by: Li Dongyang <dongyangli@ddn.com>
2018-05-11 12:13:42 +10:00
Ian Kirker 1dce7ffdb6 Hardens test for MPI
Previously, the configure script could fail to find an MPI
implementation but continue regardless. This alters the behaviour so
that the configure script fails if the check for MPI does.
2017-11-22 20:13:53 +00:00
Glenn K. Lockwood b0bffa0d0e fix for ac < 2.67; untested, but doesn't break newer autoconfs 2017-11-22 11:27:18 -08:00