Gyu-Ho Lee
870302afa6
mvcc/backend: enable 'NoFreelistSync' by default (linux)
...
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-07-05 16:10:04 -07:00
Gyu-Ho Lee
318e9c766f
*: replace 'boltdb' import paths with 'coreos/bbolt'
...
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-07-05 14:32:13 -07:00
Anthony Romano
522e75cb4f
mvcc: use GaugeFunc metric to load db size when requested
...
Relying on mvcc to set the db size metric can cause it to
miss size changes when a txn commits after the last write
completes before a quiescent period. Instead, load the
db size on demand.
Fixes #8146
2017-06-21 23:58:37 -07:00
Anthony Romano
51a568aa81
mvcc: restore into tree index with one key index
...
Clobbering the mvcc kvindex with new keyIndexes for each restore
chunk would cause index corruption by dropping historical information.
2017-06-19 12:04:01 -07:00
Anthony Romano
02164874d9
mvcc: test restore and deletes with small chunk sizes
2017-06-19 12:04:01 -07:00
Anthony Romano
7f149d8fb6
mvcc: set db size metric on restore
...
Fixes #8080
2017-06-16 11:27:34 -07:00
Anthony Romano
da48f1feaf
mvcc: create TxnWrites from TxnRead with NewReadOnlyTxnWrite
...
Already used internally by mvcc, but needed by etcdserver txns.
2017-06-09 09:20:38 -07:00
Anthony Romano
300feea177
Merge pull request #8052 from heyitsanthony/watch-victim-test
...
mvcc: test watch victim/delay path
2017-06-08 11:10:33 -07:00
Anthony Romano
83b2ea2f60
mvcc: test watch victim/delay path
...
Current tests don't normally trigger the watch victim path because the
constants are too large; set the constants to small values and hammer
the store to cause watch delivery delays.
2017-06-07 17:02:00 -07:00
Anthony Romano
0352ce79b8
mvcc: count range/put/del operations for txns
...
Txns were previously only bumping the txn counter; now bumps all operation
counters.
2017-06-07 16:53:50 -07:00
Anthony Romano
fd71da47d1
mvcc: remove unused store.Equals function
2017-06-07 09:25:42 -07:00
Anthony Romano
ef63abdf7f
mvcc: don't use pointer for storeTxnRead in storeTxnWrite
...
Saves an allocation when creating a storeTxnWrite.
2017-06-06 09:51:57 -07:00
Anthony Romano
6846e49edf
Merge pull request #7859 from heyitsanthony/cache-consistent-get
...
mvcc: cache consistent index
2017-05-26 10:52:53 -07:00
Anthony Romano
ac4855e911
mvcc: benchmark ConsistentIndex
2017-05-26 09:49:40 -07:00
Anthony Romano
73dee0bec4
mvcc: cache consistentIndex
...
Called on every entry apply and boltdb requests aren't free.
2017-05-26 09:49:40 -07:00
Anthony Romano
0506f49f9e
backend: don't hold boltdb read txn lock on cursor scanning
...
Large fetches hold the lock when they do not need to do so.
2017-05-26 09:28:08 -07:00
Anthony Romano
343a018361
Merge pull request #7900 from heyitsanthony/chunk-restore
...
mvcc: chunk reads for restoring
2017-05-26 09:21:59 -07:00
Anthony Romano
8516d8ccc5
backend: force initial mmap size to 0 for windows
...
boltdb on windows allocates a file with the full mmap size even if the
db is empty. Force the initial mmap size to 0 so there's no huge initial
db file on windows.
Fixes #7910
2017-05-12 14:34:07 -07:00
fanmin shi
8468b38631
backend: dynamically set snapshotWarningTimeout based on db size
2017-05-11 15:25:35 -07:00
Anthony Romano
1aca63e9e0
mvcc: time restore in restore benchmark
...
This never worked.
2017-05-09 20:14:58 -07:00
Anthony Romano
163fd2d76b
mvcc: chunk reads for restoring
...
Loading all keys at once would cause etcd to use twice as much
memory than it would need to serve the keys, causing RSS to spike on
boot. Instead, load the keys into the mvcc by chunk. Uses pipelining
for some concurrency.
Fixes #7822
2017-05-09 20:14:58 -07:00
fanmin shi
230106dd3c
backend: add prometheus metric for large snapshot duration.
...
FIXES #7878
2017-05-05 17:27:33 -07:00
fanmin shi
f7f30f2361
backend: print snapshotting duration warning every 30s
...
FIXES #7870
2017-05-04 16:41:03 -07:00
Anthony Romano
14d6ed9e5f
*: clear redundant return statement warnings (S1027)
2017-04-21 14:01:00 -07:00
Gyu-Ho Lee
5000d29b4a
mvcc: remove stopc select case in Hash
...
Revert change in 33acbb694b
.
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-04-17 14:19:48 -07:00
Gyu-Ho Lee
8ffd58fb3b
mvcc/backend: remove t.tx.DB()==nil checks with GracefulStop
...
Revert https://github.com/coreos/etcd/pull/6662 .
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-04-17 14:17:00 -07:00
Gyu-Ho Lee
cd470f9ccd
Revert "mvcc: test inflight Hash to trigger Size on nil db"
...
This reverts commit 994e8e4f40
.
Since now etcdserver gracefully shuts down the gRPC server
2017-04-17 14:15:43 -07:00
Anthony Romano
78a5eb79b5
*: add swagger and grpc-gateway assets for v3lock and v3election
2017-04-10 15:21:07 -07:00
Anthony Romano
f67bdc2eed
*: support checking that an interval tree's keys cover an entire interval
2017-04-03 15:38:07 -07:00
Gyu-Ho Lee
161c7f6bdf
Merge pull request #7579 from gyuho/fix-defrage
...
*: fix panic during defrag operation
2017-03-23 10:08:33 -07:00
Anthony Romano
7ef75e373a
Merge pull request #7525 from heyitsanthony/big-backend
...
etcdserver, backend: configure mmap size based on quota
2017-03-23 10:06:00 -07:00
Gyu-Ho Lee
26abd25cd3
mvcc/backend: hold 'readTx.Lock' until completing bolt.Tx reset
...
Fix https://github.com/coreos/etcd/issues/7526 .
When resetting `bolt.Tx` in `defrag` and `batchTxBuffered.commit`
operation, we do not hold `readTx` lock, so the inflight range
requests can trigger panic in `mvcc.Range` paths. This fixes by
moving mutexes out and hold it while resetting the `readTx`.
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-03-23 09:47:43 -07:00
Xiang Li
7698a2a546
Merge pull request #7553 from xiang90/fix_defrag
...
backend: add FillPercent option
2017-03-21 11:16:17 -07:00
Xiang
95870a21eb
backend: add FillPercent option
2017-03-21 08:06:03 -07:00
Anthony Romano
8a3fee15a3
etcdserver, backend: only warn if exceeding max quota
2017-03-17 15:38:57 -07:00
Anthony Romano
5e4b008106
*: base initial mmap size on quota size
2017-03-17 15:38:49 -07:00
Anthony Romano
2f1542c06d
*: use filepath.Join for files
2017-03-16 07:46:06 -07:00
Anthony Romano
33acbb694b
mvcc: txns and r/w views
...
Clean-up of the mvcc interfaces to use txn interfaces instead of an id.
Adds support for concurrent read-only mvcc transactions.
Fixes #7083
2017-03-08 20:52:59 -08:00
Anthony Romano
8d438c2939
backend: readtx
...
ReadTxs are designed for read-only accesses to the backend using a
read-only boltDB transaction. Since BatchTx's are long-running
transactions, all writes to BatchTx will writeback to ReadTx, overlaying
the base read-only transaction.
2017-03-08 20:52:59 -08:00
Gyu-Ho Lee
3d75395875
*: remove never-unused vars, minor lint fix
...
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-03-06 14:59:12 -08:00
Manjunath A Kumatagi
0914b8b707
test: Fix gosimple errors
...
Getting gosimple suggestion while running test script, so this PR is for fixing gosimple S1019 check.
raft/node_test.go:456:40: should use make([]raftpb.Entry, 1) instead (S1019)
raft/node_test.go:457:49: should use make([]raftpb.Entry, 1) instead (S1019)
raft/node_test.go:458:43: should use make([]raftpb.Message, 1) instead (S1019)
Refer https://github.com/dominikh/go-tools/blob/master/cmd/gosimple/README.md#checks for more information.
2017-02-09 08:01:28 -05:00
sharat
43078d3ced
mvcc: remove unused restore method
2016-11-18 23:04:39 +05:30
sharat
aa2b5aec1b
mvcc : Added benchmark for store.resotre
2016-11-17 04:01:15 +05:30
sharat
f014cca644
mvcc: TestStoreRestore fix
2016-11-16 16:58:42 +05:30
sharat
95fb41a923
mvcc: store.restore taking too long triggering snapshot cycle fix
2016-11-16 16:31:20 +05:30
Gyu-Ho Lee
b8b72f80f9
*: revendor, update proto files
2016-11-10 12:02:00 -08:00
Gyu-Ho Lee
425acb28c4
mvcc: return -1 for wrong watcher range key >= end
...
Fix https://github.com/coreos/etcd/issues/6819 .
2016-11-08 17:02:28 -08:00
Gyu-Ho Lee
4e1d3f0f52
mvcc: expose 'backend.IgnoreKey'
2016-10-25 10:07:08 -07:00
Gyu-Ho Lee
994e8e4f40
mvcc: test inflight Hash to trigger Size on nil db
2016-10-21 11:02:09 -07:00
Gyu-Ho Lee
7d30326968
backend: skip *bolt.DB.Size call when nil
...
Fix https://github.com/coreos/etcdlabs/issues/30 .
2016-10-21 11:01:23 -07:00
Gyu-Ho Lee
46716fe9fb
mvcc: fix gofmt issues from Go tip
2016-10-20 16:32:47 -07:00
Xiang Li
93225ebafc
mvcc: fix rev inconsistency
...
Try:
./etcdctl put foo bar
./etcdctl del foo
./etcdctl compact 3
restart etcd
./etcdctl get foo
mvcc: required revision has been compacted
The error is unexpected when range over the head revision.
Internally, we incorrectly set current revision smaller than the
compacted revision when we remove all keys around compacted revision.
This commit fixes the issue by recovering the current revision at least
to compacted revision.
2016-10-12 10:42:57 -07:00
Nikita Vetoshkin
064e02f4b3
mvcc: Optimize updating key by storing lease in lessor
2016-10-12 09:37:09 +05:00
Nikita Vetoshkin
9970ded79f
mvcc: add BenchmarkWatchableStoreTxnPut benchmark
2016-10-06 22:44:25 +05:00
Xiang Li
fa1e28102e
Merge pull request #5316 from ajityagaty/too_many_allocs
...
mvcc: Reduce number of allocs in PUT when watchableStore has no watchers.
2016-10-06 09:47:59 -07:00
Gyu-Ho Lee
9b56e51ca7
*: regenerate proto + gofmt change
2016-10-03 15:34:34 -07:00
Xiang Li
962433c17f
*: set repo correctly for logging
2016-10-03 17:03:22 +08:00
ychen11
69f5b4ba79
Documentation:made watch request doc more clear
2016-09-23 23:13:55 +08:00
Xiang Li
1437388f77
mvcc: force commit and hash should be atomic for getting hash
2016-08-27 19:22:22 -07:00
Xiang Li
e1789aa531
mvcc: only write txn should update index
2016-08-22 22:05:51 -07:00
Xiang Li
de864d3b58
mvcc: fix count
2016-08-10 10:54:25 -07:00
Xiang Li
bd62b0a646
mvcc: attach keys to leases after recover all state
...
The previous logic is wrong. When we have hisotry like Put(foo, bar, lease1),
and Put(foo, bar, lease2), we will end up with attaching foo to two leases 1 and
2. Similar things can happen for deattach by clearing the lease of a key.
Now we try to fix this by starting to attach leases at the end of the recovery.
We use a map to keep the last lease attachment state.
2016-08-04 11:17:58 -07:00
Gyu-Ho Lee
982e18d80b
*: regenerate proto with latest grpc-gateway
2016-07-27 13:21:03 -07:00
Xiang Li
fffa484a9f
*: regenerate proto for adding deleterange
2016-07-23 16:17:44 -07:00
Gyu-Ho Lee
50be793f09
*: regenerate proto
2016-07-18 09:33:32 -07:00
Anthony Romano
ba2725c2d0
build, backend: add backend commit failpoints
2016-07-14 12:26:35 -07:00
Xiang Li
c853704ac9
*: support get-old-kv in watch
2016-07-05 16:17:09 -07:00
Xiang Li
bc6d7659af
Merge pull request #5795 from xiang90/filter
...
*: support watch with filters
2016-06-28 14:07:12 -07:00
Xiang Li
dced92f8bd
*: support watch with filters
...
Now user can filter events with types. The API is also extensible.
It might make sense for the proxy to filter out events based on
more expensive/customized filter.
2016-06-28 13:46:57 -07:00
Xiang Li
38035c8c13
Merge pull request #5794 from xiang90/fix_c
...
mvcc: do not hash consistent index
2016-06-28 12:25:32 -07:00
Xiang Li
ef9754910e
mvcc: do not hash consistent index
2016-06-28 09:36:26 -07:00
Xiang Li
9614dc6e71
etcdserver: check index of the kv when restarting
2016-06-27 10:27:27 -07:00
Nikita Vetoshkin
dbc7c2cf4e
backend: reuse timer in run().
...
Benchmarks:
```
import (
"testing"
"time"
)
func BenchmarkTimeAfter(b *testing.B) {
b.ReportAllocs()
for n := 0; n < b.N; n++ {
select {
case <- time.After(1 * time.Millisecond):
}
}
}
func BenchmarkTimerReset(b *testing.B) {
b.ReportAllocs()
t := time.NewTimer(1 * time.Millisecond)
for n := 0; n < b.N; n++ {
select {
case <- t.C:
}
t.Reset(1 * time.Millisecond)
}
}
```
Running reveals that each loop results in 3 allocs:
```
BenchmarkTimeAfter-4 2000 1112134 ns/op 192 B/op 3 allocs/op
BenchmarkTimerReset-4 2000 1109774 ns/op 0 B/op 0 allocs/op
```
2016-06-23 18:49:41 +05:00
Xiang Li
def21f11a9
*: support count in range query
2016-06-21 16:20:55 -07:00
Xiang Li
6f28b43806
*: fix pending events metrics
2016-06-19 23:00:39 -07:00
Xiang Li
5a7b7f7595
main: add grpc-gateway support
...
Now etcd can serve HTTP json request at /v3alpha/
2016-06-14 17:09:06 -07:00
Gyu-Ho Lee
6bbd8b7efb
mvcc: add keyIndex benchmark test
...
Useful later when trying to optimize our restore operations.
2016-06-09 14:13:18 -07:00
Gyu-Ho Lee
1610391449
*: following changes for proto update
2016-06-07 13:33:03 -07:00
Anthony Romano
f57b4eb46d
mvcc: don't cancel watcher if stream is already closed
...
Close() already cancels all the watchers but doesn't bother to clear out
the bookkeeping maps so Cancel() may try to cancel twice.
Fixes #5533
2016-06-03 11:12:46 -07:00
Anthony Romano
9c767cbf98
Merge pull request #5464 from heyitsanthony/fix-victim-watchers
...
mvcc: tighten up watcher cancelation and revision handling
2016-05-30 20:09:39 -06:00
Gyu-Ho Lee
3266c809e4
mvcc: delete EXPIRE event type
...
Addressing https://github.com/coreos/etcd/pull/5484#discussion_r65005236 .
etcd v3 doesn't expire keys. It's either PUT of DELETE.
2016-05-29 14:54:38 -07:00
Anthony Romano
cfb3f96c2b
mvcc: tighten up watcher cancelation and revision handling
...
Makes w.cur into w.minrev, the minimum revision for the next update, and
retries cancelation if the watcher isn't found (because it's being processed
by moveVictims).
Fixes : #5459
2016-05-27 17:19:32 -07:00
Gyu-Ho Lee
bf8cf39daf
mvcc: use capnslog
2016-05-20 22:31:22 -07:00
Anthony Romano
394ce5f3b8
mvcc: move blocked unsynced watchers to victim list
2016-05-20 15:56:02 -07:00
Anthony Romano
5984e46364
mvcc: move blocked sync watcher work to victim list
...
Instead of holding the store lock while doing a lot of work like when syncung
unsynced watchers, the work from a blocked synced notify can be reused and
dispatched without holding the store lock for long.
2016-05-20 15:56:02 -07:00
Anthony Romano
0b34b236d6
mvcc: benchmark for synced watchers
2016-05-19 23:31:27 -07:00
Gyu-Ho Lee
aa85cf037f
mvcc: remove defer in watchable store
2016-05-19 13:51:51 -07:00
Xiang Li
6f2e7875aa
etcdctl: add migrate command
...
Migrate command accepts a datadir and an optional user-provided
transformer function that transform v2 keys to v2 keys.
Migrate command then builds a v3 backend state based on the existing
v2 keys and the output of the transformer function.
2016-05-19 12:17:15 -07:00
Gyu-Ho Lee
77775e8e92
mvcc: preallocate bytes buffer for saveIndex
2016-05-18 10:01:57 -07:00
Nick Owens
e4e4c9dc2c
mvcc: set bolt options to nil for non-linux systems
2016-05-17 12:46:44 -07:00
Gyu-Ho Lee
9d9f02c1ee
mvcc: update LICENSE header
2016-05-12 20:50:33 -07:00
Anthony Romano
080272be17
mvcc: limit total watchers synced per sync
...
Fixes #4567
2016-05-11 11:16:43 -07:00
Xiang Li
0fb7cb8b00
*: add disk operation metrics for monitoring
2016-05-11 09:36:45 -07:00
Ajit Yagaty
34b0736f2c
mvcc: Reduce number of allocs when watchableStore if no watchers.
...
When there are no watchers the number of allocations made while handling
a PUT operation can be reduced by exiting early.
2016-05-11 00:51:00 -07:00
Gyu-Ho Lee
a288188001
*: typo, remove string type assertions
2016-05-03 10:59:57 -07:00
Gyu-Ho Lee
015acabdbb
*: rerun genproto -g
2016-05-02 23:02:31 -07:00
Gyu-Ho Lee
7a6d9ea01a
mvcc: Hash to return Revision
2016-05-02 15:04:24 -07:00
Xiang Li
ad327e01d0
mvcc: add benchmark for watch put and improve it
2016-04-29 19:58:37 -07:00
Xiang Li
3ddcc21179
mvcc: fix watch deleteRange
2016-04-29 09:40:28 -07:00
Xiang Li
67645095e9
*: add debugging metrics
2016-04-26 09:52:56 -07:00
Anthony Romano
b7ac758969
*: rename storage package to mvcc
2016-04-25 15:25:51 -07:00