c6580dc93b
Merge pull request #99 from glennklockwood/issue87
...
fixes #87
2018-10-06 16:54:23 +01:00
a0ef65e61f
stop the segfault when an invalid api is specified in configfile
2018-10-05 23:29:21 -07:00
d8d4742523
fixes #87
...
Leading whitespace is stripped from each line of the ior input file. This
allows indented comments to be treated as comments. However it does NOT allow
one to specify excessive whitespace inside of the `ior start` and `ior stop`
magic phrases.
Also added a test to catch regressions in this functionality.
2018-10-05 22:50:34 -07:00
3aa3ff10f5
fixes #96
...
Call MPI_Abort() if MPI is initialized; otherwise just exit()
2018-10-05 22:20:18 -07:00
acee84bedc
Merge pull request #88 from roblatham00/hdf5-coll-md
...
HDF5: option to enable collective metadata
2018-10-02 16:13:05 +01:00
60820da874
Merge pull request #91 from glennklockwood/makedist-fix
...
fix various issues related to release packages
2018-10-01 21:30:58 -07:00
1f1e56c363
Usleep feature added according to Nathan's suggestions.
2018-09-30 10:01:21 +01:00
0a103147ab
Merge pull request #92 from DDNStorage/backend-legacy
...
Support old legacy name in backends
2018-09-24 11:38:13 +01:00
f472162784
Support old legacy name in backends
...
Context: Some backends may have used different names in
the past (like IME backend use to be IM). Legacy scripts
may break.
This patch adds a legacy name option in the aiori structure.
Both name and legacy name work to select the interface.
But the following warning is printed if the legacy name is used:
ior WARNING: [legacy name] backend is deprecated use [name] instead.
2018-09-24 11:51:00 +02:00
e448ff266d
fix various issues related to release packages
...
- travis now tests the packaged source to detect missing source/headers
- basic tests are less sensitive to the directory from where they are run
- fixed some missing files from the `make dist` manifest
- updated the format of NEWS to work with `make dist`
2018-09-21 18:35:48 -04:00
7a7655e959
Fix erroneous write bandwidth with stonewalling
...
Context: write and read results from the same iteration
use the same length value in Bytes. When stonewalling is
used the size variates depending on the performance of
the access. This leads to wrong max bandwidths reported
for writes as shown in the following example:
write 10052 ...
read 9910 ...
write 10022 ...
read 9880 ...
write 10052 ...
read 9894 ...
Max Write: 9371.43 MiB/sec (9826.66 MB/sec)
Max Read: 9910.48 MiB/sec (10391.89 MB/sec)
This patch makes IOR separate variables used for read
and write tests.
2018-09-21 11:43:33 +02:00
8c727fa99c
Simplify and factorize a few functions
...
This patch factorizes some duplicated code. It also
simplifies the way times are stored to prepare next
patch.
2018-09-21 11:38:44 +02:00
f03e7c6778
Make write verification work even without read test
...
Context: IOR outputs errors when the '-w -W' flags are used
without '-r'. Write a file using with check write option
should be possible even without setting read.
This patch fixes a condition which was introduced for
HDFS to remove RDWR flag in some particular cases.
Write check was set with the write only flag but it
requieres the read flag.
2018-09-20 18:18:48 +02:00
7d2464f733
HDF5: option to enable collective metadata
...
In HDF5-1.10 the library has an option to read and write the metadata of a file collectively. One has to opt-in to this optimization with a property list but particularly at large scale it can have tremendous performance implications.
Further information:
- https://support.hdfgroup.org/HDF5/docNewFeatures/CollMDataIO/RFC-CollectiveMetadataReads.pdf
- https://support.hdfgroup.org/HDF5/doc/RM/RM_H5P.html#Property-SetAllCollMetadataOps
2018-09-20 11:12:08 -05:00
e5c8a11769
Merge pull request #83 from DDNStorage/initbackends
...
Initialize only the required backends
2018-09-19 10:27:11 +01:00
2bcaea564e
Initialize only the required backends
...
Context: IOR initializes all available backends. If one
backend fails to initialize IOR cannot be used.
This patch makes IOR initialize only the backends
which will be used. The initialization is done after
that the parameters are checked so that the help message
can still be dispayed is something goes wrong.
2018-09-18 19:24:29 +02:00
44e8f03917
Fix bug in computing max write and read bandwidths
...
This patch fixes a bug when computing the maximum
bandwidths in short summary output.
2018-09-18 19:23:36 +02:00
fd34b08e05
Added patch fix O_DIRECT again.
2018-09-18 16:48:59 +01:00
151a9daa52
Check provided IO API is valid
...
Context: IOR gets a segfault if an unsupported API string
is provided.
This patch checks that the I/O backend is supported, otherwise
IOR stops with an error.
2018-09-10 19:16:28 +02:00
1a4fd7d095
Fix -i option #81
2018-09-05 09:14:22 +01:00
2d42c25f69
Fix issue #79 for the last run in a configuration file.
2018-08-30 15:10:38 +01:00
ba2f0330b7
Merge pull request #80 from roblatham00/more-fixes
...
We are getting there ^-^
2018-08-29 21:09:31 +01:00
c1f3f485ec
Bugfix #79
2018-08-29 21:07:34 +01:00
cb88c4c19a
including lustre_user now deprecated
...
put in a configure test (same one used in ROMIO) in case someone's using
old Lustre client headers
2018-08-28 16:27:21 -05:00
4a32a2cbb3
fix "number of arguments doesn’t match prototype" error
2018-08-28 16:24:34 -05:00
6b782226c7
fix pnetcdf build issues
...
It does not look like anyone has built the pnetcdf driver in a long time
2018-08-28 14:29:36 -05:00
982f266a73
clean up hints in HDF5 error cases
2018-08-28 13:30:51 -05:00
4dcf36c1a4
Help text now outputs *compiled* APIs.
2018-08-15 17:07:17 +01:00
fefefa002d
make sure to allow fsyncs in RADOS
2018-08-14 17:08:04 -05:00
dfc86b60b8
add RADOS backend options for pool, config file
2018-08-14 16:56:19 -05:00
0ca5f26e84
cleanup RADOS func names and docs
2018-08-14 16:18:35 -05:00
dc7187794c
Merge pull request #70 from DDNStorage/fix-ime-backend
...
Fix compilation of some aiori backends and add new function calls in IME backend
2018-08-10 18:00:37 +01:00
13b885ce21
Merge pull request #71 from DDNStorage/fixwarnings
...
Fix warnings reported at compilation time
2018-08-09 09:46:24 -07:00
490a912e6e
Add support for new function calls in the IME backend
...
This patch adds initialize, finalize and stat calls to
the IME backend. statfs, mkdir and rmdir are currently
not supported. This patch also fixes the IME_GetVersion
call.
2018-08-09 13:08:43 +02:00
9596b4bbe0
Fix compilation of aiori backends requiring external libraries
...
This patchs makes a few adjustements in the automake file
in order to properly add include and library directories.
Those are required to be able to compile some aiori backends.
2018-08-09 13:08:36 +02:00
df34f024ba
Fix warnings reported at compilation time
...
This patch fixes most of the warnings caused
by unused variables and assignments from incompatible
type.
2018-08-09 13:04:52 +02:00
54fa3fa8e9
Fixed wrong output.
2018-08-07 23:28:19 +01:00
44ce8e5aaf
Add backend for DDN's Infinite Memory Engine (IME)
...
This patch adds the support of IME Native interface
as a new AIORI backend.
2018-08-07 11:41:07 +01:00
e708681f19
Option: check for missing arg.
2018-08-06 18:38:34 +01:00
0870ad78b3
Support shared large runs with limited number of files per directory by creating one directory for a bunch of files.
...
Therefore, it uses the -I argument (number of files per directory) and divides it by -n (items).
This will be a number of sub directories that are created on the top-level.
2018-07-26 10:38:06 +01:00
580fb8cdf7
Bugfix offsetof; thanks George for reporting.
2018-07-15 20:23:10 +01:00
f55761d5d2
Refactored results structure to AoS; allowed to keep results per repeat.
2018-07-15 19:38:17 +01:00
c7a598a435
New option parser: Fixed overflow for blocksize. Added warning.
2018-07-15 18:37:15 +01:00
bb6d4fb2a2
Update create for stonewall.
2018-07-15 10:02:45 +01:00
d4ecc76ee8
Stonewall bugfix for delete/stat.
2018-07-14 16:46:24 +01:00
c0657866fd
Minor fixes for stonewalling options.
2018-07-14 12:22:36 +01:00
5d4ac1a0a0
Bugfix parser: wrong datatype
2018-07-14 11:32:39 +01:00
04ba846da2
Fix for SetHints (after changing way strings are managed).
2018-07-14 11:11:47 +01:00
b613bf9396
AIORI update to initialization and fix HDF5 with changed API.
2018-07-14 10:47:25 +01:00
ac7600d662
Unit-test with automake for library version. Minor fix for library version.
2018-07-14 10:27:31 +01:00
5ef95675bf
AIORI initialize() and finalize() functions.
2018-07-14 09:29:08 +01:00
a0b8659cc8
IOR appears to work with the new option parser.
2018-07-14 09:09:48 +01:00
5e98fa0408
Dummy: option for xfer delay.
2018-07-14 08:50:27 +01:00
54e47cf729
IOR: use new option parser.
...
TODO: Parse "-O" options
2018-07-14 08:41:35 +01:00
0835191372
Option parser replaced.
...
Benefits: shows currently set options when used with -h
Supports options for child modules, example for aiori-dummy works, RADOS untested.
2018-07-12 18:09:13 +01:00
205b20f0f4
Bugfix merge
2018-07-12 16:13:40 +01:00
060c47e80a
Merge branch 'testing' into rados-hacks
2018-07-12 15:53:04 +01:00
7981691a63
add a new AIORI backend for RADOS
...
inform aiori interface about RADOS backend
stubbed out aiori backend for rados
additions to get RADOS backend compiling/linking
first cut at rados create/open patha
make sure to return RADOS oid on open/create
implement rados xfer path for WRITE
refactor + implement getfilesize and close
remember to use read_op interface for stat
implement RADOS delete function
don't error in RADOS_Delete for now
implement RADOS set_version
handle open/create flags appropriately
cleanup RADOS error handling
implement read/readcheck/writecheck for RADOS
rados doesn't support directio
implement unsupported aiori ops for RADOS
implement RADOS access call
define rados types if no rados support
2018-07-12 09:36:49 -05:00
831a5ef823
JSON: Slight output adaption.
2018-07-09 16:25:35 +01:00
0f17d4860f
Pretty printing for normal output.
2018-07-08 16:56:04 +01:00
2bc30410cc
JSON output works.
2018-07-08 16:47:23 +01:00
20ebeb71b8
Further adaptions to generic output, support of JSON output (partially complete).
2018-07-08 14:59:54 +01:00
8525f9734c
Extraction of output completed.
2018-07-08 13:50:20 +01:00
02883d4a63
Moved most output to ior-output.c
2018-07-08 13:38:05 +01:00
0f7a1f14b9
Support IO redirect from main()
2018-07-08 13:12:18 +01:00
0052bff3e9
use backend->access in IOR source
...
Signed-off-by: Julian M. Kunkel <juliankunkel@googlemail.com>
2018-07-08 12:13:38 +01:00
8069e018e3
make ncmpi/hdf5 use mpiio access/delete
...
Signed-off-by: Julian M. Kunkel <juliankunkel@googlemail.com>
2018-07-08 12:13:38 +01:00
1a3c92718b
Fix merge
2018-07-08 12:13:08 +01:00
7d3c7e901e
expose generic aiori_ calls for access, etc.
...
Signed-off-by: Julian M. Kunkel <juliankunkel@googlemail.com>
2018-07-08 12:09:44 +01:00
9e97a16fe4
fix output to be allinge with column headers
...
Signed-off-by: Julian M. Kunkel <juliankunkel@googlemail.com>
2018-07-08 11:54:49 +01:00
d938f0efe1
MPIIO support fsync.
2018-07-08 00:16:30 +01:00
fd4c306b98
MDTest stonewalling with status file.
...
Appears to work; stonewall timer is honored only in the creation phase.
This implies a limit to the objects that then are used for other phases.
2018-07-07 23:56:39 +01:00
58f13ba352
Bugfix: extraction to count ranks per host used local communicator.
2018-07-07 22:54:40 +01:00
31e3cc2b60
Bugfix extraction further.
2018-07-07 22:46:13 +01:00
b27b50184a
Bugfix Refactoring MaxLen
2018-07-07 22:42:05 +01:00
ed5787811c
Extracted max pathlen variable.
2018-07-07 22:39:14 +01:00
48c17b4d81
Extracted (nearly identical) function DelaySeconds.
2018-07-07 22:26:57 +01:00
6f8d2e8845
Some simplification / unification between IOR and MDTest.
...
MDTest needs more refactoring to be really maintainable.
2018-07-07 22:19:42 +01:00
7bc868d5cf
Dummy AIORI supporting MDtest ops.
2018-07-07 20:01:03 +01:00
b4f356a7b2
Bugfix: StoneWallingStatusFile.
2018-07-07 14:14:55 +01:00
812b798f05
IOR allows to store actual performaned stonewalling count into a status file.
2018-07-07 14:01:11 +01:00
378789737f
Dummy AIORI driver
2018-07-07 13:25:17 +01:00
15c383b0d1
Extended description for #44
2018-07-07 12:20:29 +01:00
e10bebe653
MDTest supports only POSIX at the moment. Added command line check.
2018-07-07 12:14:34 +01:00
e31b2dcec1
Added missing file.
2018-07-07 11:57:04 +01:00
21b8cca8cf
IOR library version.
2018-07-07 11:42:21 +01:00
a6bfa0f94b
MDTest library option.
...
Moved core functionality / variables used by both implementations to utilities
2018-07-07 10:42:48 +01:00
bcaea2a39f
Replaced getopt with an alternative version (on some machines optind was broken).
2018-07-07 08:41:33 +01:00
156a514c8b
open file in RDWR when using write + checkWrite
2018-05-16 16:52:20 -07:00
43cc47b7f0
use MPI_COMM_SELF when opening file to get file size in MPIIO
2018-05-16 12:31:23 -07:00
6ba22a19e4
Add mmap as a new AIORI backend
...
It shares the create/open/delete/set_version/get_file_size
functions with POSIX backend.
The mmap backend also supports fsync and fsyncPerWrite options,
and it will use msync() instead and fsync().
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
2018-05-11 12:13:42 +10:00
92dfb67e5c
allow fsync in POSIX, MPIIO, and HDFS
2018-04-27 12:44:20 -05:00
76f3d341b9
implement MPIIO sync
2018-04-27 11:33:52 -05:00
9b75f071f4
use backend->access in IOR source
2018-04-25 17:35:10 -05:00
7233bee953
make ncmpi/hdf5 use mpiio access/delete
2018-04-25 17:34:25 -05:00
60243e0277
expose MPIIO access and delete calls
2018-04-25 17:32:16 -05:00
d7fc07163e
expose generic aiori_ calls for access, etc.
2018-04-25 17:21:48 -05:00
7a371cfeda
Merge pull request #46 from lidongyang/iops
...
Add OPs result columns back
2018-04-23 13:12:43 -07:00
a80cc44fb1
fixes #47
2018-04-23 13:10:04 -07:00
0d2c9e5451
Add OPs result columns back
...
Signed-off-by: Li Dongyang <dongyangli@ddn.com>
2018-04-18 14:24:42 +10:00
Julian Kunkel