This patch enables correct compilation on both MacOS and Linux using only
POSIX.1-2008-compliant C with XSI extensions.
Specifically, POSIX.1-2008 is the minimum version because we use strdup(3);
explicit XSI is required to expose putenv from glibc.
inform aiori interface about RADOS backend
stubbed out aiori backend for rados
additions to get RADOS backend compiling/linking
first cut at rados create/open patha
make sure to return RADOS oid on open/create
implement rados xfer path for WRITE
refactor + implement getfilesize and close
remember to use read_op interface for stat
implement RADOS delete function
don't error in RADOS_Delete for now
implement RADOS set_version
handle open/create flags appropriately
cleanup RADOS error handling
implement read/readcheck/writecheck for RADOS
rados doesn't support directio
implement unsupported aiori ops for RADOS
implement RADOS access call
define rados types if no rados support
It shares the create/open/delete/set_version/get_file_size
functions with POSIX backend.
The mmap backend also supports fsync and fsyncPerWrite options,
and it will use msync() instead and fsync().
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
Previously, the configure script could fail to find an MPI
implementation but continue regardless. This alters the behaviour so
that the configure script fails if the check for MPI does.
This commit makes changes to the AIORI backends to add support for
abstacting statfs, mkdir, rmdir, stat, and access. These new
abstractions are used by a modified mdtest. Some changes:
- Require C99. Its 2017 and most compilers now support C11. The
benefits of using C99 include subobject naming (for aiori backend
structs), and fixed size integers (uint64_t). There is no reason to
use the non-standard long long type.
- Moved some of the aiori code into aiori.c so it can be used by both
mdtest and ior.
- Code cleanup of mdtest. This is mostly due to the usage of the IOR
backends rather than a mess of #if code.
Signed-off-by: Nathan Hjelm <hjelmn@lanl.gov>
Commit aa604c1d38 introduced gpfs_fcntl
operations, depending if gpfs.h and gpfs_fcntl.h were found. These
routines need the GPFS client library. This configure check will bring
in libgpfs -- you will still need to set LDFLAGS and CPPFLAGS if gpfs is
installed in some non-default locaiton (as it is on Blue Gene).
fixes https://github.com/chaos/ior/issues/15
Testing on our EMC ViPR installation. Therefore, we also have available
some EMC extensions. For example, EMC supports a special "byte-range"
header-option ("Range: bytes=-1-") which allows appending to an object.
This is not needed for N:1 (where every write creates an independent part),
but is vital for N:N (where every write is considered an append, unless
"transfer-size" is the same as "block-size").
We also use a LANL-extended implementation of aws4c 0.5, which provides
some special features, and allows greater efficiency. That is included in
this commit as a tarball. Untar it somewhere else and build it, to produce
a library, which is linked with IOR. (configure with --with-S3).
TBD: EMC also supports a simpler alternative to Multi-Part Upload, which
appears to have several advantages. We'll add that in next, but wanted to
capture this as is, before I break it.
This provides an HDFS back-end, allowing IOR to exercise a Hadoop
Distributed File-System, plus corresponding changes throughout, to
integrate the new module into the build. The commit compiles at LANL, but
hasn't been run yet. We're currently waiting for some configuration on
machines that will eventually provide HDFS. By default, configure ignores
the HDFS module. You have to explicitly add --with-hdfs.
GPFS supports a "gpfs_fcntl" method for hinting various things,
including "i'm about to write this block of data". Let's see if, for
the cost of a few system calls, we can wrangle the GPFS locking system
into allowing concurrent access with less overhead. (new IOR parameter
gpfsHintAccess)
Also, drop all locks on a file immediately after open/creation in the
shared file case, since we know all processes will touch unique regions
of the file. It may or may not be a good idea to release all file locks
after opening. Processes will then have to re-acquire locks already
held. (new IOR parameter gpfsReleaseToken)
Removing AC_FUNC_MALLOC from configure.ac, to allow compilation
on BG/P systems. This check can fail in cross-compilation environments,
which unnecessarily forces autoconf to require an rpl_malloc()
replacement for malloc(). We could implement the conditional addition
of rpl_malloc(), but removing AC_FUNC_MALLOC is a quite work-around.
fixes#4
Glenn K. Lockwood