185d2bced4
wal: use leveled logger
2015-06-01 13:38:50 -07:00
34ac145b38
*: use namespace and subsystem in metrics
...
Fix #2841 .
From Prometheus developer:
```
the recommended way for etcd as an open source project and under
consideration of its size would be etcd_<subsystem>_<name>.
```
We made the naming change accordingly.
2015-05-26 14:39:04 -07:00
d1d7feacc9
wal: change io.EOF returned by readFull to io.ErrUnexpectedEOF
...
Decoder should return error for any broken block including the
one that only contains the length field. We should change io.EOF
to io.ErrUnexpectedEOF before return the error.
2015-04-23 09:53:36 -07:00
aed18395c9
wal: report throughput in wal bench
2015-04-12 21:35:08 -07:00
89242d4659
wal: better log msg
2015-04-09 09:54:20 -07:00
53792ccbdc
wal: never leave a corrupted wal file
...
If the process dies during wal.cut(), it might leave a corrupted wal
file. This commit solves the problem by creating a temp wal file first,
then atomically rename it to a wal file when we are sure it is vaild.
2015-04-08 15:57:20 -07:00
44de670de7
wal: allow at most one WAL function called at one time
...
SaveSnap and Save are called in separate goroutines now. Allow at most
one WAL function being called at one time to protect internal fields and
guarantee execution order.
Or one possible bug is that the new cut file is started with snapshot
entry instead of crc entry.
2015-04-08 00:34:30 -07:00
8bcaa2bfdf
wal: reduce allocation when encoding int64
2015-03-30 20:41:31 -07:00
c32cca3a4f
wal: reduce allocation when encoding entries
2015-03-30 19:20:46 -07:00
3e9a033cd2
wal: repair decoder needs to update its crc
2015-03-30 13:45:23 -07:00
684ebd95ae
wal: backup broken wal before repairing
2015-03-29 15:42:59 -07:00
8b4eed29e5
wal: fix the unexpectedEOF error in the last wal.
...
It is safe to repair the unexpectedEOF error in the last wal. raft
will not send out message before the entry successfully comitted
into wal. Thus we can safely truncate the last entry in the wal
to repair.
2015-03-28 21:08:14 -07:00
05e240b892
*: update protobuf
2015-03-25 10:14:35 -07:00
3dd6e0b88f
wal: fix missing import
2015-03-24 22:53:15 -07:00
6e6669d696
wal: releastTo should work with large release index
2015-03-24 22:34:26 -07:00
5e0077cc0c
etcdserver: print out extra files in data dir instead of erroring
2015-03-24 18:56:22 -07:00
b66eb3d81c
wal: fix ReleaseLockTo
...
ReleaseLockTo should not release the lock on the WAL
segment that is right before the given index. When
restarting etcd, etcd needs to read from the WAL segment
that has a smaller index than the snapshot index.
The correct behavior is that ReleaseLockTo releases
the locks w is holding so that w only holds one lock
that has an index smaller than the given index.
2015-03-09 19:52:54 -07:00
ab72c3ec88
wal: do not race reader and writer
2015-03-05 20:19:17 -08:00
86429264fb
wal: support auto-cut in wal
...
WAL should control the cut logic itself. We want to do falloc to
per allocate the space for a segmented wal file at the beginning
and cut it when it size reaches the limit.
2015-02-28 11:18:59 -08:00
84485643fe
*: expose wal metrics at /metrics
2015-02-28 11:06:11 -08:00
fb1a28c65d
*: vendor prometheus
2015-02-28 11:06:11 -08:00
9e63b1fb63
wal: record metrics
2015-02-28 10:12:35 -08:00
3fd9136740
migrate/starter: fix v2 data dir checking
2015-02-24 11:47:56 -08:00
92dca0af0f
*: remove shadowing of variables from etcd and add travis test
...
We've been bitten by this enough times that I wrote a tool so that
it never happens again.
2015-02-17 16:31:42 -05:00
fade9b6065
etcdserver: Refactor 2.0.1 directory rename into a proper migration
...
fix all instances
fix detection test
2015-02-12 11:53:19 -05:00
e966e565c4
etcdctl/backup_command: handle datadir with missed snapshot mark
...
This helps to recover from the data dir created in v2.0.0-rc1.
2015-01-29 13:32:59 -08:00
f1ed69e883
*: switch to line comments for copyright
...
Build tags are not compatible with block comments.
Also adds copyright header to a few places it was missing.
2015-01-26 09:53:30 -08:00
05e591f805
wal: remove unused encoder.buffered func
2015-01-09 14:59:46 -08:00
9bdc343b7c
wal: add ReleaseLockTo test
2015-01-09 14:59:41 -08:00
270e67db84
wal: not export unnecessary public functions
2015-01-09 14:55:10 -08:00
50c179ec1c
wal: add DetectVersion test
2015-01-09 14:55:05 -08:00
f08d1090d0
wal: refine parseWalName function
...
According to http://godoc.org/fmt#Scan , if scan number is less than the
number of arguments, err will report why. So we don't need to handle
this error case.
2015-01-08 14:56:21 -08:00
9532810f76
wal: remove unused max function
2015-01-08 14:49:14 -08:00
6460e49a33
wal: save empty snapshot when create
...
So caller can open at empty snapshot to read all entries.
2015-01-06 19:48:21 -08:00
78bb207bac
wal: update doc about snapshot
2015-01-06 19:33:57 -08:00
84f62f21ee
wal: record and check snapshot
2015-01-06 16:27:40 -08:00
1ec98cb795
pkg/fileutil: sort filenames during ReadDir
2014-12-18 16:36:11 -08:00
502396edd5
wal: fix wal doc
2014-12-14 19:36:37 -08:00
53bf7e4b5e
wal: rename openAtIndex -> open; OpenAtIndexUntilUsing -> openNotInUse
2014-12-14 19:33:06 -08:00
f538cba272
*: do not backup files still in use
2014-12-14 19:27:22 -08:00
ea94d19147
*: lock the in using files; do not purge locked the wal files
2014-12-14 19:27:22 -08:00
cf7690cb51
detect more cases of empty directories and actual errors
2014-12-11 13:37:32 -05:00
421fe128c3
Return Unknown instead of NotExist
...
Unless the data dir truly does not exist.
2014-12-11 13:09:50 -05:00
b9bf957c6d
wal: sync after writing data to disk in Cut function
2014-12-04 22:56:34 -08:00
af0f34c595
wal: save latest state into new WAL
...
So we could always read out state when open at valid index.
2014-12-04 12:19:21 -08:00
aa61009560
wal: not return ErrIndexNotFound in ReadAll
...
This IndexNotFound case is reasonable now because we don't write dummy
entries into wals any more.
2014-12-02 00:28:54 -08:00
d3db010190
*: support purging old wal/snap files
2014-12-01 11:50:17 -08:00
5139257b8d
Merge pull request #1743 from barakmich/auto_upgrade
...
etcdserver: autodetect v0.4 WALs and upgrade them to v0.5 automatically
2014-11-20 17:20:34 -05:00
59a0c64e9f
fix import loop, add set to types, and fix comments
2014-11-20 15:38:08 -05:00
78ea3335bf
etcdserver: autodetect v0.4 WALs and upgrade them to v0.5 automatically
2014-11-20 15:38:08 -05:00
d0dd205b0e
wal: add a bench for write entry
2014-11-20 12:07:17 -08:00
5a964f49a5
wal: propagate errors
2014-11-08 17:16:23 -08:00
791b2fd503
*: handle panic and fatal more consistently
...
1. etcd fatals if there is critical error in the system and operator should
do something for it
2. etcd panics if there happens something unexpected, and it should be
reported to us to debug.
2014-11-05 13:53:24 -08:00
075ab6415f
Merge pull request #1587 from xiangli-cmu/fix_wal
...
wal: sync before returning from create
2014-11-03 15:58:47 -08:00
ff1f5a9d57
wal: sync before returning from create
2014-11-03 14:28:59 -08:00
5da481213e
Merge pull request #1478 from unihorn/190
...
etcdserver: panic on storage error
2014-11-03 11:07:55 -08:00
433b4138c5
etcdserver: panic on storage error
...
It is a critical error to etcd, and etcd is not able to recover it now.
2014-11-03 10:46:04 -08:00
513c72ec8b
wal: update the docs to show the optional metadata field
2014-10-31 11:32:17 -07:00
aa50af1c69
*: clean log.Print
...
1. only log things by default that the operator of etcd may need to react to
2. put package name at the head of log lines
2014-10-30 18:15:53 -07:00
7a4d42166b
*: add license header to all source files
2014-10-17 15:41:22 -07:00
f98fbbfc14
*: proto refactoring
2014-10-14 21:07:23 +08:00
4183b69e12
*: move from third_party to Godep
2014-10-14 00:37:52 -07:00
447caf1afc
etcdserver/wal: record info at the head of WAL file
2014-10-10 11:57:09 -07:00
8bbbaa88b2
*: raft related int64 -> uint64
2014-10-09 14:29:21 +08:00
1d5d2e3726
*: Id -> ID for protobuf types
...
We use ID instead of Id in this project based on golang conventions.
2014-09-26 11:49:30 -07:00
2f6086de22
pkg: move /crc to /pkg/crc
2014-09-25 10:50:33 -07:00
c8c55aa378
scripts: consolidate and standardize protobuf generation
2014-09-24 13:45:00 -07:00
cf4af47f7e
wal: remove logging messages
...
These are noisy and unhelpful by default. We can re-add if at some point
we reintroduce debugging/multiple log levels.
2014-09-22 19:08:47 -07:00
df70f653a4
Merge pull request #1099 from unihorn/128
...
wal: OpenFromIndex fails if it cannot find previous index
2014-09-17 16:00:51 -07:00
29f6d8a9e6
wal: add ErrFileNotFound and ErrIndexNotFound
2014-09-17 15:58:06 -07:00
7160b5ae26
wal: OpenFromIndex fails if it cannot find previous index
...
Example:
We save entry 1, 2, 3 to WAL.
If we try to open 100, it should fail.
2014-09-17 15:07:07 -07:00
ab61a8aa9a
*: init for on disk snap support
2014-09-17 13:56:12 -07:00
de21c39ca5
raft: isStateEqual -> isHardStateEqual, IsEmptyState -> IsEmptyHardState
2014-09-16 13:55:00 -07:00
023dc7cba2
etcdserver: add SYNC request
2014-09-16 13:42:03 -07:00
69f2d5c590
wal: refactor
2014-09-15 10:35:36 -07:00
77fbd2610c
wal: add comment
2014-09-15 10:17:21 -07:00
1d09c25f5f
wal: add walName function; cleanup test
2014-09-15 10:17:21 -07:00
cec1956b8f
wal: better comment
2014-09-15 10:14:18 -07:00
8e0ee1cc5e
wal: remove wrong comment for cut
2014-09-15 10:14:18 -07:00
1164c4b83d
wal: recover writting seq
2014-09-15 10:14:18 -07:00
21860bc017
wal: cut(i uint64) -> cut
2014-09-15 10:14:18 -07:00
140fd6d6c4
raft: restart using last written entry also
2014-09-15 09:56:33 -07:00
a9af70c52b
raft: write entry 0 into log
2014-09-15 09:55:52 -07:00
2030ca202f
wal: change wal filename format
...
Make raftIndex section to be expected raftIndex of next entry.
It makes filename more intuitive and straight-forward.
The commit also adds comments for filename format.
2014-09-12 11:24:28 -07:00
884c702512
wal: wal.OpenFromIndex -> wal.OpenAtIndex
...
The first entry read from the wal should be the index provided. This
name makes that more clear.
2014-09-11 12:52:06 -07:00
413b6a59ff
wal: add more explanation to wal filename format
2014-09-11 12:51:50 -07:00
04abd5603f
wal: add a doc file
2014-09-10 20:40:20 -07:00
c251304068
*: s/Id/ID/
...
golang convention dictates that the individual characters in an
abbreviation should all have the same case. Use ID instead of Id.
The protobuf generator still generates code that does not meet
this convention, but that's a fight for another day.
2014-09-10 16:09:08 -07:00
24fd126822
*: stop using 0xBEEF
...
Using 0xBEEF is annoying in examples because it makes it makes it look
like the user can use ascii or something. In the Procfile use
0x0,0x1,0x2,etc and use 0xBAD0 in test.
2014-09-09 16:40:51 -07:00
6f06923e96
wal: test save empty state
2014-09-09 10:41:15 -07:00
90c0db3d42
wal: do not save empty state
2014-09-09 10:15:29 -07:00
9997c9488a
*: fix a few small issues identified by go vet
2014-09-08 23:52:36 -07:00
54734b0903
main/wal: add a const for 0700 magic number
2014-09-08 15:45:58 -07:00
b094410066
wal: change entries->ents for consistency
2014-09-08 15:40:12 -07:00
adff0f3813
wal: named return values for ReadAll.
2014-09-08 15:36:25 -07:00
0461c517e4
wal: clarify TODO
2014-09-08 13:57:35 -07:00
1a6e908971
*: add wal
2014-09-05 09:52:40 -07:00
b98cf17209
wal: move pb files to walpb
2014-09-03 16:46:42 -07:00
3ee83bc194
Merge pull request #82 from etcd-team/b-fiximports
...
coerce gogoproto files to import from third_party
2014-09-03 16:20:58 -07:00
10b73418cf
wal: coerce import gogoproto from third_party
2014-09-03 16:00:42 -07:00
46a7a61b7d
wal: fix broken tests
2014-09-03 15:56:28 -07:00
bdb954b2f5
wal: fix wal
2014-09-03 15:10:15 -07:00
85103adfe0
wal: move record method to record.go
2014-09-03 09:20:11 -07:00
442cae6844
wal: support multiple files
2014-09-03 09:20:10 -07:00
b5b2031d5b
wal: prune repeative logging
2014-09-03 09:20:04 -07:00
659eb5fd2a
wal: make record a protobuf type
2014-09-03 09:20:03 -07:00
8e06333d45
raft: make State a protobuf type
2014-09-03 09:20:03 -07:00
d6c3ebb1a0
raft: make Info a protobuf type
2014-09-03 09:20:03 -07:00
2d870fa65b
wal: add first level logging
2014-09-03 09:20:01 -07:00
969b529b08
wal: fix O_RDONLY attr when opening old file
2014-09-03 09:20:01 -07:00
38ec659cd6
raft: make Entry a protobuf type
2014-09-03 09:20:01 -07:00
43c9ca895b
wal: fix append entry
2014-09-03 09:20:01 -07:00
215820dd40
server: move TestMultiNodeKillOne to etcd/
2014-09-03 09:20:00 -07:00
28634fce47
wal: Flush -> Sync
2014-09-03 09:20:00 -07:00
d70df4a15d
wal: reorgnize wal file
2014-09-03 09:19:59 -07:00
a25cd45876
wal: add CoreOS copyright
2014-09-03 09:19:59 -07:00
9f5c7b310c
wal: add newWal helper func
2014-09-03 09:19:59 -07:00
9c6f2ed5bb
wal: use buffer
2014-09-03 09:19:59 -07:00
5baefcce26
wal: cleanup
2014-09-03 09:19:59 -07:00
363e952551
wal: add Read
2014-09-03 09:19:59 -07:00
3c7935a21d
wal: check written data in tests
2014-09-03 09:19:59 -07:00
d1cdc02afc
wal: add tests for write
2014-09-03 09:19:59 -07:00
7b180d585e
wal: make new work
2014-09-03 09:19:58 -07:00
e72e75876c
wal: init
2014-09-03 09:19:58 -07:00
Xiang Li