Commit Graph

17 Commits (2f066624f0b168233faa81800bd744b1c366fb36)

Author SHA1 Message Date
Jeffrey Thornton Inman 2f066624f0 S3 with Multi-Part Upload for N:1 is working.
Testing on our EMC ViPR installation.  Therefore, we also have available
some EMC extensions.  For example, EMC supports a special "byte-range"
header-option ("Range: bytes=-1-") which allows appending to an object.
This is not needed for N:1 (where every write creates an independent part),
but is vital for N:N (where every write is considered an append, unless
"transfer-size" is the same as "block-size").

We also use a LANL-extended implementation of aws4c 0.5, which provides
some special features, and allows greater efficiency.  That is included in
this commit as a tarball.  Untar it somewhere else and build it, to produce
a library, which is linked with IOR.  (configure with --with-S3).

TBD: EMC also supports a simpler alternative to Multi-Part Upload, which
appears to have several advantages.  We'll add that in next, but wanted to
capture this as is, before I break it.
2014-10-27 13:29:44 -06:00
Jeffrey Thornton Inman ca801fe0ec First-cut at PLFS support, integrating Brett K's code.
Builds, but not running yet.  Requires 'configure ... --with-plfs', so it seems okay to leave this in as an optional build, pending further work.
2014-09-18 11:20:37 -06:00
Jeff Inman 28c94f8875 Fixed bug in aiori-HDFS.c where it was calling "free(fd);", which is bad.
Along the way, added a bunch of diagnostic output in the HDFS calls, which
only shows up at verbosity >= 4.  I'll probably remove this stuff before
merging with master.  Also, there's an #ifdef'ed-out sleep() in place,
which I used to attach gdb to a running MPI task.  I'll get rid of that
later, too.

Also, added another hdfs-related parameter to the IOR_param_t structure;
hdfs_user_name gets the value of the USER environment-variable as the
default HDFS user for connections.  Does this cause portability problems?
2014-08-28 15:39:44 -06:00
Jeff Inman 0be8973c0e Initial commit of the new aiori-HDFS module.
This provides an HDFS back-end, allowing IOR to exercise a Hadoop
Distributed File-System, plus corresponding changes throughout, to
integrate the new module into the build.  The commit compiles at LANL, but
hasn't been run yet.  We're currently waiting for some configuration on
machines that will eventually provide HDFS.  By default, configure ignores
the HDFS module.  You have to explicitly add --with-hdfs.
2014-08-13 17:02:43 -06:00
Rob Latham aa604c1d38 Teach IOR about GPFS hints (gpfs_fcntl)
GPFS supports a "gpfs_fcntl" method for hinting various things,
including "i'm about to write this block of data".  Let's see if, for
the cost of a few system calls, we can wrangle the GPFS locking system
into allowing concurrent access with less overhead. (new IOR parameter
gpfsHintAccess)

Also, drop all locks on a file immediately after open/creation in the
shared file case, since we know all processes will touch unique regions
of the file.  It may or may not be a good idea to release all file locks
after opening.  Processes will then have to re-acquire locks already
held.   (new IOR parameter gpfsReleaseToken)
2013-10-03 18:07:49 -07:00
Christopher J. Morrone 5f162061bd Change testNum to refNum 2012-01-13 15:45:02 -08:00
Christopher J. Morrone ef70266a84 Allow long summary to be printed for every test. 2012-01-13 15:43:54 -08:00
Christopher J. Morrone 8e06f07b28 Add header to set code preferences for vi and emacs. 2012-01-08 19:43:41 -08:00
Christopher J. Morrone 416c430811 Cleanup maxTimeDuration code. 2012-01-08 19:43:41 -08:00
Christopher J. Morrone 349f747ea5 Remove undocumented options. 2012-01-08 19:43:41 -08:00
Christopher J. Morrone 4084e14d02 Add memoryPerNode option 2012-01-08 19:36:43 -08:00
Christopher J. Morrone 5ea51638df Add memoryPerTask option
Allows every task to allocate a specified amount of memory as
a rough simulation of a real application's memory usage.

Every page of the allocated memory is touch to defeat lazy
memory allocation.

Original patch by Michael Kluge <michael.kluge@tu-dresden.de>
2012-01-08 10:54:10 -08:00
Christopher J. Morrone 511a0b1435 Refactor the test summary code.
Only print total summary after all tests run.

Put calculated results from each iteration of a test in a separate
IOR_results_t structure.  Clean up the allocation and freeing code
for these caluclated bits, which allowing us to hang onto the results
until the end of all tests.  That in turn allows us to perform one
big summary at the end of all of the tests.
2011-12-13 01:08:58 -08:00
Christopher J. Morrone 672d4065de Refactor the SummarizeResults code. 2011-12-11 00:33:58 -08:00
Christopher J. Morrone a60ad0b088 Impose proper file scoping and header usage.
Clean up the header files to only contain those things that
need to be shared between .c files.

Functions that are not shared are now declared static to
make their file scope explicit.  Functions that ARE shared
are declared in appropriate headers.

I am not going to claim that I caugh everything, but at
least it is a good start.
2011-11-11 17:30:20 -08:00
Christopher J. Morrone e7fea4f102 Fix prototype locations 2011-11-11 15:18:17 -08:00
Christopher J. Morrone 08d5629302 Rename IOR.[ch] to ior.[ch]. I don't like CAPS. 2011-10-27 18:46:29 -07:00