Commit Graph

364 Commits (7b8fb3cf0ad524be6c08e2e2b4d978e228a9f5b8)

Author SHA1 Message Date
Gyu-Ho Lee fb81fb44fa etcd-tester: check expired lease with -1 TTL
Following the change at 2ca1823a96

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-03-03 11:41:53 -08:00
Manjunath A Kumatagi 0914b8b707 test: Fix gosimple errors
Getting gosimple suggestion while running test script, so this PR is for fixing gosimple S1019 check.
raft/node_test.go:456:40: should use make([]raftpb.Entry, 1) instead (S1019)
raft/node_test.go:457:49: should use make([]raftpb.Entry, 1) instead (S1019)
raft/node_test.go:458:43: should use make([]raftpb.Message, 1) instead (S1019)

Refer https://github.com/dominikh/go-tools/blob/master/cmd/gosimple/README.md#checks for more information.
2017-02-09 08:01:28 -05:00
Anthony Romano 13420b33a0 benchmark: update for new stm interface 2017-01-20 16:22:43 -08:00
Gyu-Ho Lee 6f9a20803c
etcd-tester: use 10K for '--snapshot-count'
Since we want to send snapshot more often in failure injected cluster
2017-01-14 17:29:35 -08:00
Anthony Romano 6e730af65a benchmark: option to rate limit range benchmark 2017-01-11 14:36:46 -08:00
Anthony Romano fd72ecfe92 Merge pull request #7087 from sinsharat/make_etcd-runner_command_compliant
etcd-runner: make command compliant
2017-01-04 16:33:19 -08:00
sharat d0f301adb7 etcd-runner:add flags in watcher for hardcoded values 2017-01-04 15:17:53 +05:30
Anthony Romano b8444d4d35 benchmark: enable grpc error logging on stderr
Lets you see connection errors (e.g., if tls is misconfigured)
2017-01-04 00:26:43 -08:00
sharat 5cb6dd268b etcd-runner: make command compliant 2017-01-03 14:43:58 +05:30
Xiang Li 3519a9784e Merge pull request #7039 from mitake/benchmark-dialtimeout
benchmark: a new option for configuring dial timeout
2016-12-28 13:12:11 -08:00
Hitoshi Mitake a662ddefbb benchmark: a new option for configuring dial timeout
Current benchmark doesn't have an option for configuring dial timeout
of gRPC. This commit adds --dial-timeout for the purpose. It is useful
for stopping long sticking benchmarks.
2016-12-28 14:07:43 +09:00
Gyu-Ho Lee db8b15bf8f etcd-tester: remove unused err var from maxRev 2016-12-27 12:16:43 -08:00
Gyu-Ho Lee 591bb5e7f6
etcd-tester: fix typo, add endpoint in logs 2016-12-22 12:51:27 -08:00
Gyu-Ho Lee 301abddc72 etcd-tester: cancel lease stream; fix OOM panic
It was never closing lease keep-alive streams, leaking memory.
Fix OOM panics in etcd-tester (after 1K rounds).
2016-12-13 09:56:30 -08:00
Gyu-Ho Lee 7d16e7d27e
etcd-tester: add 'enable-pprof' option 2016-12-13 05:03:27 -08:00
Gyu-Ho Lee b8e09bf849 tools: simplify boolean comparison, remove unused 2016-12-12 10:07:14 -08:00
Xiang Li e62e9ce193 benchmark: add rate limit 2016-12-05 09:54:30 -08:00
fanmin shi b9e9c9483b Merge pull request #6885 from fanminshi/refractor_lease_checker
etcd-tester: refactor lease checker
2016-12-01 10:11:15 -08:00
fanmin shi 394ab43587 etcd-tester: refactor lease checker
Move few checking logic from lease stresser to lease checker and change connection logic for lease stresser and checker
2016-11-30 17:29:58 -08:00
fanmin shi ec5f9bce63 Merge pull request #6886 from fanminshi/fix_dial_grpc
functional-tester: add withBlock() to grpc dial
2016-11-21 11:33:31 -08:00
fanmin shi 2ccba33dd1 functional-tester: add withBlock() to grpc dial
grpc dail withTimeout() only works if withBlock() option is present.
2016-11-21 11:15:12 -08:00
fanmin shi 2ac1c4c9ed etcd-tester:limit max retry backoff delay
grpc uses expoential retry if a connection is lost. grpc will sleep base on exponential delay.
if delay is too large, it slows down tester.
2016-11-21 10:58:55 -08:00
Xiang Li b1178469be etcd-tester: do not resolve localhost 2016-11-19 18:38:26 -08:00
sharat e5b6324771 benchmark: added watch-latency 2016-11-12 01:08:35 +05:30
Xiang Li 0a8e28524b Merge pull request #6779 from xiang90/watch_clean
etcd-runner: clean up watcher runner
2016-11-10 09:59:08 -08:00
Xiang Li 523a859ad9 etcd-runner: clean up watcher runner 2016-11-10 08:56:19 -08:00
Gyu-Ho Lee 55d25f6f4d tools: remove 'etcd-top'
Travis CI breaks because of cgo dependencies on 'etcd-top'.
This can leave outside of project.
2016-11-09 15:59:47 -08:00
fanmin shi 0f5d9f00ad Merge pull request #6808 from fanminshi/functional-tester-compaction-deadline-fix
etcd-tester: increase compaction timeout limit
2016-11-08 17:18:40 -08:00
fanmin shi 107d7b663c etcd-tester: changed compaction timeout calculation
functional tester sometime experiences timeout during compaction phase. I changed the timeout calculation base on number of entries created and deleted.

FIX #6805
2016-11-08 17:00:04 -08:00
Anthony Romano 2af31f99c3 etcd-top: make build require -tags pcap
Fixes travis.
2016-11-07 15:54:40 -08:00
Gyu-Ho Lee f82aac2fc6 Merge pull request #6797 from fanminshi/lease_checker_println_fix
etcd-tester: fix lease checker logging format.
2016-11-03 10:17:54 -07:00
fanmin shi 6968028020 etcd-tester: fix lease checker logging format.
lease checker used a wrong print format for a variable. this change fixes it.
2016-11-03 10:11:00 -07:00
fanmin shi 649fe7f2af etcd-tester: add retry logic on retriving lease info
getting lease and keys info through raw rpcs rarely experience error such as EOF. This is considered as a failure and causes tester to clean up.
however, they are just transient problem with temporary connection issue which should not be considered as a testing failure. so we add retry logic in case of transient failure.

FIX #6754
2016-11-03 10:05:06 -07:00
fanmin shi 8a03c95dd4 etcd-tester: refactor checking short lived lease logic
move the logic of waiting lease expired from stresser to checker
2016-11-01 14:06:22 -07:00
fanmin shi c781f30ed5 functional-tester: remove stablilization limit
This change removes the waiting needed to ensure the cluster to be stable.

FIX #6760
2016-11-01 10:01:59 -07:00
Anthony Romano 4969a0e9e7 Merge pull request #6758 from heyitsanthony/move-checker
etcd-tester: refactor stresser / checker management
2016-10-31 13:59:54 -07:00
Anthony Romano 308f2a1695 etcd-tester: refactor stresser/checker organization
The checkers and stressers should be composable without special cases; this
patch tries to address that while refactoring out some old cruft.

Namely,
* Single stresser/checker for a tester; built from composition
* Composite stresser via comma-separated list of stressers
* Split stressers into separate files
* Removed v2 only flags and special cases
* Rate limiter shared among key stresser and leases stresser
* Composite checker is now concurrent
* Stresser can return a Checker to check its invariants
* Each lease checker only operates on a single lease stresser
2016-10-31 13:59:04 -07:00
Xiang Li 9f0ee53e86 etcd-runner: move string generation to pkg/stringutil 2016-10-31 12:20:02 -07:00
Gyu-Ho Lee 5bd00ab1f6 *: fix minor typos 2016-10-31 09:47:15 -07:00
Xiang Li 7e06a95942 Merge pull request #6759 from xiang90/tester
etcd-runner: refactor code structure and flag cleanup
2016-10-31 09:07:18 -07:00
Xiang Li e5c3978725 etcd-runner: refactor code structure and flag cleanup 2016-10-30 18:45:16 -07:00
Anthony Romano 86c4a74139 etcd-tester: move stresser and checker to tester
These really belong in tester code; the stressers and
checkers are higher order operations that are orchestrated
by the tester. They're not really cluster primitives.
2016-10-29 10:57:17 -07:00
fanmin shi 34c906be55 functional-tester: fix log statement
simple fix for wrongly printed statement.
2016-10-28 14:27:09 -07:00
fanmin shi 7f5a7d1da5 functional-tester: always clean up if tester encouters an error
The current tester doesn't not clean up if any of the failure injection/recovery fails. if tester fails to recover a dead node, tester hangs in the next round because the tester will keep waiting until cluster becomes healthy which is impossible since a node is down. To fix this issue, we will always clean up if any error happens during each round so that cluster will be healthy for next round.

FIX #6743
2016-10-27 15:07:58 -07:00
fanmin shi 8d9e2623e1 functional-tester: add short lived leases checking
lease stresser now generates short lived leases that will expire before invariant checking.
this addition verifies that the expired leases are indeed being deleted on the sever side.
2016-10-26 14:46:57 -07:00
fanmin shi 1dc60bb97e functional-tester: increase lease TTL
increasing lease TTL ensure that lease doesn't expire during hashes stabilization period.
I observed that it can take a long time for etcd cluster to become stable.
2016-10-26 10:32:52 -07:00
Xiang Li 12e4dfa9c4 Merge pull request #6715 from fanminshi/lease_hash_fix
Lease hash fix
2016-10-25 16:28:11 -07:00
Xiang Li b398233f4f Merge pull request #6730 from gyuho/round-prefix
etcd-runner: fix typo in round prefix
2016-10-25 16:24:27 -07:00
Gyu-Ho Lee 12488d4a70 etcd-runner: fix typo in round prefix 2016-10-25 15:59:44 -07:00
fanmin shi bb97adda0d functional-tester: add retries to hash checking
allows hashes to converge through retrying.
2016-10-25 14:27:36 -07:00
sharat 6e7e346c93 etcd-runner: Added client connection timeout flag 2016-10-26 01:58:22 +05:30
Gyu-Ho Lee 924ece6ae7 etcd-dump-db: initial commit 2016-10-25 10:18:51 -07:00
sharat 69ae49a1dd etcd-runner: remove unused code and change name for randClient 2016-10-25 19:16:27 +05:30
Xiang Li 92c987f75d Merge pull request #6695 from sinsharat/watch_runner_respect_rounds
etcd-runner: watcher runner respect rounds
2016-10-21 20:28:29 -07:00
sharat 90146d863c etcd-runner: watcher runner respect rounds 2016-10-22 05:00:10 +05:30
fanmin shi d582fdcc1b functional-tester: add rate limiter to lease stresser
too many leases created can cause compaction to timeout. adding a rate limiter limits number of leases and attched keys.
2016-10-21 12:34:49 -07:00
fanmin shi 161eb2c457 Merge pull request #6696 from fanminshi/lease_expire_fix
functional-tester: modify lease renew logic
2016-10-20 16:25:04 -07:00
fanmin shi a25d4ac821 functional-tester: modify lease renew logic
only renew a lease if the lease is present.
2016-10-20 15:27:46 -07:00
fanmin shi 94ea82c00d functional-tester: add logic to ensure etcd node is alive after fault recovery returns
failure recovery needs to wait etcd node to become alive before returning

FIX #6654
2016-10-20 10:31:08 -07:00
sharat 50523e22d8 etcd-runner: make run watcher fail safe 2016-10-20 00:23:35 +05:30
fanmin shi ab2b58a80f functional-tester: add lease stresser
Add lease stresser to test lease code under stress and etcd failures

resolve #6380
2016-10-18 14:20:26 -07:00
fanmin shi ed75d93625 Merge pull request #6666 from fanminshi/function-tester-refractor
functional-tester: move checker logic to cluster
2016-10-18 11:44:37 -07:00
fanmin shi 7d86d1050e functional-tester: move checker logic to cluster
I move the checker logic from tester to cluster so that stressers and checkers can be initialized at the same time.
this is useful because some checker depends on stressers.
2016-10-18 11:17:40 -07:00
Xiang Li 24c284160b Merge pull request #6635 from sinsharat/etcd_runner_add_watcher_runner
etcd-runner:added watch runner
2016-10-17 11:02:06 -07:00
sharat 8297322176 etcd-runner:added watch runner 2016-10-17 23:04:33 +05:30
Hitoshi Mitake b2b03d9926 functional-tester: a new option -failure-wrapper for enabling/disabling external fault injector
This commit adds a new option -failure-wrapper to etcd-tester. The
option receives a path of script that is used for enabling/disabling
external fault injectors. The script is called with an option "enable"
when it needs to be enabled (when failure.Inject() is called) and
called with "disabled" in an opposite case (when failure.Recover() is
called).
2016-10-14 11:31:28 +09:00
Anthony Romano c1d115b322 benchmark: submit keepalive requests concurrently with report.Run()
Otherwise report won't consume the results and the benchmark hangs.
2016-10-07 15:57:38 -07:00
Anthony Romano e7d8292cd1 benchmark: add --precise flag
Usually benchmark writes with %4.4f; this adds optional %g formatting.
2016-10-06 16:18:47 -07:00
Anthony Romano 3d28faa3eb pkg/report, tools/benchmark: refactor report out of tools/benchmark
Only tracks time series when requested. Can configure output precision.
2016-10-06 16:18:47 -07:00
Hitoshi Mitake 7d48855630 functional-tester: decouple failures from tester
This commit adds a new option --failures to etcd-tester. The option
receives a comma-delimited argument like this:
"default,failpoints". The given arguments are interpreted as names of
failures and they are injected to an etcd cluster. Available failures
are default (default scenario in etcd-tester) and failpoints. If no
args are passed to the option (--failures=""), no failures are
injected during testing.
2016-09-28 11:30:53 +09:00
Hitoshi Mitake 0e493c11c2 functional-tester: decouple stresser from tester
This commit decouples stresser from the tester of
functional-tester. For doing it, this commit adds a new option
--stresser to etcd-tester. The option accepts two types of stresser:
"default" and "nop". If the option is "default", etcd-tester stresses
its etcd cluster with the existing stresser. If the option is "nop",
etcd-tester does nothing for stressing.

Partially fixes https://github.com/coreos/etcd/issues/6446
2016-09-24 01:04:57 +09:00
Gyu-Ho Lee 4ec0fce109 Merge pull request #6493 from gyuho/tester-build
functional-tester: build from repo root, vendor
2016-09-21 16:57:34 -07:00
Gyu-Ho Lee b2c4992a82 functional-tester: use different ports in Procfile 2016-09-21 02:39:45 -07:00
Gyu-Ho Lee e8adc24c32 functional-tester: build from repo root, vendor 2016-09-21 02:06:13 -07:00
Anthony Romano d4eff5381c etcd-tester: split out consistency checking code from tester 2016-09-20 10:26:58 -07:00
fanmin shi 8a63071463 etcd-tester: fix peer-port parsing bug with localhost url
The following format "http://localhost:1234" causes existing port parser to fail. Add new logic to parse the host name first then extract port.

Fixes #6409
2016-09-12 13:29:52 -07:00
fanmin shi a571655983 functional-tester: correct goreman command in readme
update readme file to have the correct goreman command to start the functional tester locally.
2016-09-09 14:56:23 -07:00
Anthony Romano aa6b1e6a10 functional-tester: add Procfile 2016-09-08 16:35:55 -07:00
Anthony Romano 55ba3d95fb etcd-tester: support per-agent client/peer/failpoint ports 2016-09-08 16:15:18 -07:00
Anthony Romano 2e25a772a5 etcd-agent: support rootless operation and configurable gofail ports 2016-09-08 16:12:00 -07:00
Gyu-Ho Lee 0bf2ef3c1b etcd-tester: return error for mismatch rev/hash 2016-08-27 22:14:42 -07:00
Xiang Li 017aac88a8 etcd-tester: do not restart stresser on error 2016-08-27 20:47:45 -07:00
Gyu-Ho Lee 0c37df7265 etcd-tester: fix compact rev counting 2016-08-22 22:58:44 -07:00
Anthony Romano 47b6449934 functional-tester: put large keys
For testing writes that must span multiple pages.
2016-08-17 09:51:44 -07:00
Xiang Li feaff17259 session: remove session manager and add ttl 2016-08-15 14:12:25 -07:00
Gyu-Ho Lee c6cf015e26 etcd-tester: fix tester for 5-node cluster
1. fix failure case counting
2. match ErrClientConnClosing in stresser
3. longer timeout for set-health-key
4. fixed range for range/delete stresser
5. remove Limit in RangeRequest
2016-08-07 21:15:01 -07:00
Anthony Romano 978ae9de29 functional-tester: share limiter among stresser
Otherwise, adding more members stresses the cluster with more ops.
2016-08-07 19:15:00 -07:00
Xiang Li 7678b84f2c Merge pull request #6123 from xiang90/fix_limiter
tools/functional-tester: fix limiter
2016-08-07 16:20:17 -07:00
Xiang Li f6a1585902 functional-tester: reduce rate to 3000 2016-08-07 14:34:01 -07:00
Xiang Li 107a07563f tools/functional-tester: fix limiter 2016-08-07 14:28:16 -07:00
Xiang Li 69204397ee tools/functional-tester: better logging 2016-08-07 14:21:44 -07:00
Gyu-Ho Lee 854a215329 etcd-tester: delete/range with limit, clean up 2016-08-05 11:21:36 -07:00
Gyu-Ho Lee 755eee0d30 etcd-tester: remove unnecessary arg from stresser 2016-08-01 15:35:31 -07:00
Gyu-Ho Lee 969bcd282b etcd-tester: stress with range, delete 2016-07-18 15:17:08 -07:00
Anthony Romano 0419d3ecf7 etcd-tester: add FailFast(false) to grpc calls 2016-07-14 19:16:41 -07:00
Gyu-Ho Lee ceb9fe4822 etcd-tester: stop stress before compact, fix races
fix race condition between stresser cancel, start
2016-07-14 12:16:42 -07:00
Gyu-Ho Lee 8e59fb749c etcd-tester: increase default qps, fix cleanup 2016-07-14 11:20:16 -07:00
Xiang Li 7114a27345 Merge pull request #5922 from xiang90/l_b
tools/benchmark: add benchmark for lease keepalive
2016-07-12 10:48:04 -07:00
Xiang Li 8273e1c07e tools/benchmark: add benchmark for lease keepalive 2016-07-12 10:40:56 -07:00
Xiang Li 4bc29e2b9c Merge pull request #5902 from mitake/bench-auth
tools: add --user for auth in benchmarks
2016-07-11 18:38:07 -07:00