Added aws-0.5.1.tgz with LANL-extensions to open-source tool-kit, for S3-related interactions.
We'll pass our changes to the maintainers of aws4c (current released rev is 0.5). Thus, this library should eventually be available by other means. But, until then, I'm adding it to IOR, so the S3 module will have support. S3 also needs libxml2, but this is publicly available.master
parent
4368cc2dc4
commit
a50da47323
|
@ -0,0 +1,90 @@
|
|||
-- fix indentation in all source files
|
||||
|
||||
Do not use some fixed-width spacing. Use tabs. Someday, editors will
|
||||
distinguish between "indentation" (which should be done with tabs), and
|
||||
"alignment" which should be done with spaces. That way, you choose your
|
||||
own indentation, and everything just works.
|
||||
|
||||
|
||||
-- change "HDFS" everywhere to "HDFs" (or something)
|
||||
|
||||
for easier distinction from "HDF5"
|
||||
|
||||
|
||||
-- maybe parse_options.c should just parse options, not create tests.
|
||||
|
||||
The latter brings an ugly mix of calls across files. Fixing involves
|
||||
having parse routines just set flags in params, then, in ior.c, someone
|
||||
checks flags and calls test-creation stuff. Should also move
|
||||
test-creation to ior.c [Results creation, too.]
|
||||
|
||||
|
||||
-- collect various special-purpose parameter-fixing into one place.
|
||||
|
||||
see e.g. CheckRunSettings(). Should each aiori have a "fix the
|
||||
parameters" generic function, as well? This could be called at a known
|
||||
time, allowing incompatibilities to be handled in a regular way.
|
||||
|
||||
|
||||
-- does MPIIO_GetFileSize() have to be in aiori.h?
|
||||
|
||||
|
||||
-- change integers to 0x'..', in aiori.h
|
||||
|
||||
|
||||
-- add hdfs-related command-line options
|
||||
|
||||
(e.g. HDFS user-name, replica-count, etc.)
|
||||
see ior.h
|
||||
see init_IOR_Param_t(), in ior.c
|
||||
|
||||
|
||||
-- is the getenv() I added to ior.c a problem for Win support?
|
||||
|
||||
|
||||
-- see "TBD" notes in src/Makefile.am
|
||||
|
||||
(1) for HDFS:
|
||||
|
||||
# TBD: figure out how to find this from the corresponding bin/ dir in $PATH
|
||||
# or pick an environment var to use (and set it in modulefiles)
|
||||
# or provide a config-flag, to set a variable we use here
|
||||
|
||||
|
||||
(2) for PLFS:
|
||||
|
||||
# TBD: modulefiles don't add PLFS_VER to env, but you can find it embedded in
|
||||
# $PLFS_HOME. configure.ac should extract this for us. Then we should
|
||||
# add -D... to ior_CPPFLAGS, here. Then aiori-PLFS.c can conditionalize
|
||||
# compilation of calls that vary according to PLFS version.
|
||||
|
||||
|
||||
|
||||
|
||||
|
||||
---------------------------------------------------------------------------
|
||||
S3
|
||||
---------------------------------------------------------------------------
|
||||
|
||||
|
||||
-- RemoveFile (ior.c) calls POSIX "access" to see whether file exists
|
||||
|
||||
This is called between iterations, when iterations > 1. Should use
|
||||
backend->GetFileSize(), or something, and catch errors. Or the aiori
|
||||
struct could be extended to include a lower-level "stat" that could
|
||||
return 0 for non-existent files.
|
||||
|
||||
|
||||
-- add 'init' and 'exit' functions to aiori struct
|
||||
|
||||
This would allow HDFS/S3/etc to do their one-time initializations and
|
||||
cleanups. Call from top-level in ior.c
|
||||
|
||||
Something also needed in AioriBind().
|
||||
|
||||
|
||||
-- ior should assert the IOR_TRUNC flag in params, when opening?
|
||||
|
||||
The CREAT flag does not reset a file, if it already exists, when opening
|
||||
for write. [But ... wait. If you're opening for WRITE, you should be
|
||||
resetting it anyhow. D'oh.]
|
Binary file not shown.
BIN
lanl_aws4c.tgz
BIN
lanl_aws4c.tgz
Binary file not shown.
Loading…
Reference in New Issue