027c073d55
storage: refine Watch comment in WatchableKV
...
Explain explicitly how these arguments are used.
2015-10-21 14:47:52 -07:00
2673e657e6
storage: fix WatchableKV interface
...
We delete endRev from the watch functionality, so the interface needs
to be fixed.
2015-10-21 11:50:25 -07:00
0f7374ce89
storage: KV field -> store field in watchableStore
...
We need to access the underlying store to use its RangeEvents function.
It is not good to use unnecessary type conversion.
The underlying store is also needed for further store upon
watchableStore.
2015-10-20 19:23:20 -07:00
5efdd7bc6d
storage/backend: avoid creating new bolt.tx during a batchTx
2015-10-17 21:31:34 -07:00
6556bf1643
storage: remove the endRev of watcher
2015-10-17 15:59:49 -07:00
21179d929f
Merge pull request #3616 from yichengq/storage-txn
...
storage: hold batchTx lock during KV txn
2015-10-05 17:12:52 -07:00
0aa2f1192a
storage: add metrics for db total size
2015-10-05 16:56:30 -07:00
ba949ae2be
Merge pull request #3640 from xiang90/watch_metrics
...
storage: add metrics for watchers
2015-10-05 11:46:50 -07:00
09157d4f1a
storage: add metrics for watchers
2015-10-05 11:32:56 -07:00
c97dda766e
storage: hold batchTx lock during KV txn
...
One txn is treated as atomic, and might contain multiple Put/Delete/Range
operations. For now, between these operations, we might call forecCommit
to sync the change to disk, or backend may commit it in background.
Thus the snapshot state might contains an unfinished multiple objects
transaction, which is dangerous if database is restored from the snapshot.
This PR makes KV txn hold batchTx lock during the process and avoids
commit to happen.
2015-10-03 16:01:05 -07:00
36f4303fc3
storage/etcdserver: update KV.Snapshot function
...
When using Snapshot function, it is expected:
1. know the size of snapshot before writing data
2. split snapshot-ready phase and write-data phase. so we could cut
snapshot first and write data later.
Update its interface to fit the requirement of etcdserver.
2015-10-03 10:15:23 -07:00
385e17583f
storage: fix hash by iterating kv
2015-09-23 11:28:33 -07:00
4113509828
storage/watchable_store: defer to Unlock s.mu
...
New PR from https://github.com/coreos/etcd/pull/3575 .
This add `defer` to `s.mu`. Current code does not `Unlock`
in the correct scope, I think.
(Sorry, I accidentally deleted my fork so the changes
might not sound continuous from my previous pull requests.)
2015-09-22 23:25:07 -07:00
d72914c36f
storage: clarify comment for store.RangeEvents and fix related bugs
...
Change to the function:
1. specify the meaning of startRev and endRev parameters
2. specify the meaning of returned nextRev
Moreover, it adds unit tests for the function.
2015-09-19 23:17:03 -07:00
5709b66dfb
storage: add unit test for index.RangeEvents
2015-09-19 23:08:24 -07:00
87b5143b15
storage: fix missing continue in keyIndex.since
...
It should continue to skip following operations.
The test from rev14 to rev0 fails if it doesn't call continue and append
all revisions of the same main rev to the list.
2015-09-19 23:01:18 -07:00
158d6e0e03
storage: fix calculating generation in keyIndex.since
...
It should skip last empty generation when the key is just tombstoned.
The rev15 and rev16 in the test fails if it doesn't skip last empty generation
and find previous generations.
2015-09-19 22:58:45 -07:00
be80d11948
storage: enhance test for keyIndex.Get and keyIndex.Compact
...
It covers the case that one key is set multiple times in one main
revision now.
2015-09-17 18:26:17 -07:00
f7efbe8b14
storage/backend: extend wait timeout for commit to finish
...
It needs to take more time on travis. Fix:
```
--- FAIL: TestBackendBatchIntervalCommit (0.01s)
backend_test.go:113: bucket test does not exit
```
2015-09-16 14:14:51 -07:00
7848ac3979
*: add missing license headers
2015-09-15 14:09:01 -07:00
6d1f0ce89f
storage: add rev into kv interface
2015-09-15 12:11:00 -07:00
c082488e23
Merge pull request #3507 from yichengq/watch
...
storage: support basic watch
2015-09-15 00:04:36 -07:00
ec43e0a4c3
storage: introduce WatchableKV and watch feature
...
WatchableKV is an interface upon KV, and supports watch feature.
2015-09-14 23:53:03 -07:00
451cce4a90
Merge pull request #3516 from xiang90/hash_improved
...
storage: support hash state
2015-09-13 21:46:12 -07:00
714b5e0b08
storage: support hash state
2015-09-13 21:34:58 -07:00
b2f4a5f587
*: fix spelling issues (codespell).
...
Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>
2015-09-11 10:22:29 +10:00
3f18ded10a
*: v3api index->revision
2015-09-04 10:41:20 -07:00
215f27c2f5
storage: add mock tests for store struct
2015-09-04 08:53:49 -07:00
5ae2eb4731
storage: avoid one extra round of wait
...
It could exit early if it knows that there is no more keys.
2015-09-03 19:12:27 -07:00
9175df7c71
storage: correct revision for range when deleteRange
...
to make it logically reasonable.
2015-09-03 19:12:27 -07:00
797a4796d9
storage: remove check for DELETE type KeyValue
...
kvindex always returns kvs that exist at given revision, so there is no
need to check for whether the KeyValue range from backend is DELETE type.
2015-09-03 19:12:27 -07:00
00e31f13a6
storage: remove unnecessary rev parameter
2015-09-03 19:12:27 -07:00
ef7cf058a2
*: update gogoproto
2015-09-03 15:32:25 -07:00
45390b9fb8
*: regenerate proto to use local import path
...
Using Go-style import paths in protos is not idiomatic. Normally, this
detail would be internal to etcd, but the path from which gogoproto
is imported affects downstream consumers (e.g. cockroachdb).
In cockroach, we want to avoid including `$GOPATH/src` in our protoc
include path for various reasons. This patch puts etcd on the same
convention, which allows this for cockroach.
More information: https://github.com/cockroachdb/cockroach/pull/2339#discussion_r38663417
This commit also regenerates all the protos, which seem to have
drifted a tiny bit.
2015-09-03 13:38:28 -04:00
1eaf169057
Merge pull request #3395 from yichengq/backend-test
...
storage/backend: add unit tests for backend and batchTx
2015-09-03 07:23:38 -07:00
44fd734038
storage/backend: add unit tests for backend and batchTx
2015-09-02 16:57:13 -07:00
a21166c3aa
storage: extend timeout to wait for put complete
...
travis is sometimes slow, and it could fail to complete the put in 10ms.
2015-09-01 09:03:03 -07:00
4b9b0cbcc1
storage: add newBackend and newBatchTx
...
This is for ease of testing.
2015-08-31 13:25:10 -07:00
b9632e0f8d
storage: register txnCounter
2015-08-28 15:17:16 -07:00
dd443be41b
storage: report total number of keys
2015-08-28 15:16:53 -07:00
054fab84ee
storage/backend: remove startc var
...
This makes start logic cleaner.
2015-08-28 13:52:31 -07:00
b5838edb93
storage: add initial metrics for kv
2015-08-28 13:41:42 -07:00
f04884f74d
storage/backend: fix off-by-one error for pending var
...
Or it may commit until batchLimit + 1.
2015-08-27 22:51:32 -07:00
7ed929fb3d
storage/backend: fix limit doesn't effect in range
2015-08-27 22:51:32 -07:00
19a28c8efd
storage: Fixed backend test
...
./backend_test.go:23: multiple-value batchTx.UnsafeRange() in single-value context
2015-08-27 15:20:29 -04:00
0d38c13990
storage: use temp path to handle test file
2015-08-26 15:01:41 -07:00
2d01eb4e11
storage: add tests for kvstore_compaction
2015-08-26 15:01:13 -07:00
0813139140
storage: add more tests for index
2015-08-26 12:53:30 -07:00
3723f01b48
storage: add more unit tests for keyIndex
2015-08-26 12:53:30 -07:00
ad8a291dc1
storage: return error when tombstone on new generation
...
It is not allowed to put tombstone on an empty generation.
2015-08-26 12:53:30 -07:00
ffa87f9678
storage: fix the comment in generation.walk
2015-08-26 12:53:30 -07:00
e8f40b0412
storage/backend: add commitAndStop
...
After the upgrade of boltdb, db.Close waits for all txn to finish.
CommitAndStop commits the current txn and stop creating new ones.
2015-08-25 10:57:25 -07:00
e3ef1d363a
Merge pull request #3366 from xiang90/v3_proto
...
update v3 proto and doc
2015-08-24 11:22:29 -07:00
3a60d490d1
storagepb: fix comment location
2015-08-24 10:42:16 -07:00
9c0c314425
storage: add functional tests for the package
...
It adds and reorganize tests to construct functional tests.
2015-08-23 20:59:06 -07:00
9960651c3f
storage: let range work in the process of txn
...
range should work in the process of txn to help check the status during the
txn.
2015-08-23 20:59:06 -07:00
6d97dcaf3f
storage: ensure that desired compaction is persisted
...
It needs to persist the desired compaction, so it won't forget the compaction
if it crashes later.
2015-08-23 20:59:06 -07:00
353f10ca2b
storage: reject to compact on future rev
...
Compaction on future rev is unreasonable.
2015-08-23 20:59:06 -07:00
47b243be5d
storage: let TxnDeleteRange return rev if no error
...
If it doesn't return error, it should return valid rev.
2015-08-23 20:59:06 -07:00
62f7481b19
storage: keyIndex.get returns err when key is tombstoned
...
Before this commit, it will return wrong create index, mod index.
It lets findGeneration return error when rev is at the gap of two
generations. This leads to the change of compact() code.
2015-08-23 20:59:02 -07:00
3b2fa9f1de
storage: fix TestKeyIndexCompact
...
It fails to pass before.
2015-08-23 17:22:49 -07:00
c09b667d57
*: fix go vet reported issues
2015-08-22 12:19:02 -05:00
6b23a8131f
*: test gofmt with -s and fix reported issues
2015-08-21 18:52:16 -07:00
acd7a92f03
storage: reversion -> revision
2015-08-20 08:39:07 -07:00
53a77fa519
*: tnx -> txn
2015-07-24 23:21:09 +08:00
883bb47dcf
Merge pull request #3074 from xiang90/storage_restore
...
storage: correctly restore create and ver
2015-06-30 09:20:19 -07:00
f8b947a00b
storage: fix small issues
2015-06-29 22:02:21 -07:00
581ef05bab
*: resolve proto warnings
2015-06-29 18:39:46 -07:00
13f44e4b79
*: update generated proto code
2015-06-29 16:45:25 -07:00
433f2ee1bc
storage: correctly restore create and ver
...
Add a restore func to correctly restore create reversion and
version of keys for the index.
2015-06-29 13:44:43 -07:00
ccca2b04da
storage: save version
2015-06-29 13:15:09 -07:00
c069119abe
Merge pull request #3067 from xiang90/storage_created_mod
...
storage: save created index and modified index
2015-06-27 23:11:05 -07:00
4581064060
storage: save created index and modified index
2015-06-26 12:10:26 -07:00
5787fabe5f
Merge pull request #3008 from yichengq/storage-index-test
...
storage: add range and tombstone test for index
2015-06-18 19:29:31 -07:00
b20598eea0
storage: add range and tombstone test for index
2015-06-18 18:05:37 -07:00
9f2e4c8a57
storage: remove unnecessary ForceCommit in kvstore.Close
...
s.b.Close will commit pending ops, so there is no need to FroceCommit
it in kvstore.Close()
2015-06-18 13:36:23 -07:00
789e2f3426
Merge pull request #3003 from yichengq/storage-kvstore-test
...
storage: add restore test and fix some bug
2015-06-18 12:19:05 -07:00
7cba42fb73
storage: wait for compact goroutine to exit before close backend
...
If backend is closed, the operations on backend in compact
goroutine will panic. So this PR waits for compact goroutine to exit
before close backend.
This fixes the TestWorkflow failure too.
2015-06-18 12:18:39 -07:00
148394f66f
storage: fix schedule compaction bug in recover process
...
It uses wrong schedule compaction reversion before.
2015-06-18 11:11:37 -07:00
26a09d8479
storage: enhance TestRestore and kill TODO
2015-06-18 10:37:12 -07:00
0ef53ee500
Merge pull request #2999 from yichengq/storage-rev-test
...
storage: add reversion test
2015-06-18 07:39:18 -07:00
74fbf9d6a7
storage: add reversion test
2015-06-17 18:06:42 -07:00
80a59f00b7
storage: fix limit mismatch in Range func
2015-06-17 17:43:08 -07:00
93f477944b
storage: return ErrFutureRev if rev is a future one
2015-06-17 17:42:43 -07:00
94924d04db
storage: add TestRangeBadRev
2015-06-17 16:22:28 -07:00
9ad5e1e64f
storage: kill TODO in TestRange
2015-06-17 15:58:28 -07:00
500894dfe5
storage: add TestWorkflow
2015-06-17 14:38:21 -07:00
7b1a93e1ef
storage: put storage info keys into information bucket
...
They used to be in key bucket, and make recover failed because they
cannot be parsed as normal key.
2015-06-17 14:37:29 -07:00
ba9a46aa02
storage: initial snapshot and restore
...
Snapshot takes an io.Writer and writes the entire backend data to
the given writer. Snapshot writes a consistent view and does not
block other storage operations.
Restore restores the in-memory states (index and book keeping) of
the storage from the backend data.
2015-06-10 11:32:10 -07:00
f47ed4a364
storage: initial compact
2015-06-05 09:22:44 -07:00
d417b36e5d
storage: refactor key_index
2015-05-31 15:24:04 -07:00
6c207b9277
stroage: kill todo
2015-05-27 14:46:59 -07:00
69d02410cf
stroage: adopt KV interface
2015-05-27 14:24:23 -07:00
cbb8b9bb08
stroage: add tnx id
2015-05-27 10:35:51 -07:00
9c1aec6877
storage: add rangeKeys func
2015-05-27 09:58:21 -07:00
fb12a4e412
storage: fix a deadlock in batch tx
2015-05-27 09:31:11 -07:00
93ecf36855
storage: support tnx
2015-05-27 09:31:11 -07:00
9db360387d
storage: support Range
2015-05-27 09:31:11 -07:00
7bb388ed52
storage: initial kv api
2015-05-27 09:31:11 -07:00
49da7b6556
storage: add boltdb as dependency
2015-05-27 09:24:49 -07:00
e332e86b5d
storage: address barak's comments
2015-05-20 17:47:35 -07:00
845cb61213
storage: add kv and event proto
2015-05-18 14:35:10 -07:00
9575cc4258
storage: add delete example
2015-05-15 19:33:59 -07:00
4b0d9f69c7
storage: add a simple backend and kv example
2015-05-14 20:43:32 -07:00
9d831e3075
*: godep btree
2015-05-14 17:59:55 -07:00
660fd5e3e1
storage: add comment around compact
2015-05-14 17:55:54 -07:00
ee47973199
storage: initial index
2015-05-14 17:53:41 -07:00
32d44aa3b2
storage: initial key index
2015-05-14 17:35:12 -07:00
Yicheng Qin