Xiang Li
39a4b6a5e5
pkg/fileutil: support perallocate
2015-08-06 10:10:58 -07:00
Hitoshi Mitake
ba76e27875
wal: log errors in wal.Close()
...
This patch adds error logging in wal.Close() if unlocking and
destroying fail. Though it is hard to handling the errors, logging
would be helpful for trouble shooting.
2015-08-05 15:03:45 +09:00
Xiang Li
f59da0e453
*:fix point-in-time backup
...
Backup process should be able to read all WALs until io.EOF to
generate a point-in-time backup.
Our WAL file is append-only. And the backup process will lock all
files before start reading, which can prevent the gc routine from
removing any files in the middle.
2015-06-15 11:12:28 -07:00
Xiang Li
711451ce2d
*: rename logger to plog
2015-06-02 14:58:24 -07:00
Xiang Li
185d2bced4
wal: use leveled logger
2015-06-01 13:38:50 -07:00
Xiang Li
89242d4659
wal: better log msg
2015-04-09 09:54:20 -07:00
Xiang Li
53792ccbdc
wal: never leave a corrupted wal file
...
If the process dies during wal.cut(), it might leave a corrupted wal
file. This commit solves the problem by creating a temp wal file first,
then atomically rename it to a wal file when we are sure it is vaild.
2015-04-08 15:57:20 -07:00
Yicheng Qin
44de670de7
wal: allow at most one WAL function called at one time
...
SaveSnap and Save are called in separate goroutines now. Allow at most
one WAL function being called at one time to protect internal fields and
guarantee execution order.
Or one possible bug is that the new cut file is started with snapshot
entry instead of crc entry.
2015-04-08 00:34:30 -07:00
Xiang Li
c32cca3a4f
wal: reduce allocation when encoding entries
2015-03-30 19:20:46 -07:00
Xiang Li
8b4eed29e5
wal: fix the unexpectedEOF error in the last wal.
...
It is safe to repair the unexpectedEOF error in the last wal. raft
will not send out message before the entry successfully comitted
into wal. Thus we can safely truncate the last entry in the wal
to repair.
2015-03-28 21:08:14 -07:00
Xiang Li
6e6669d696
wal: releastTo should work with large release index
2015-03-24 22:34:26 -07:00
Xiang Li
b66eb3d81c
wal: fix ReleaseLockTo
...
ReleaseLockTo should not release the lock on the WAL
segment that is right before the given index. When
restarting etcd, etcd needs to read from the WAL segment
that has a smaller index than the snapshot index.
The correct behavior is that ReleaseLockTo releases
the locks w is holding so that w only holds one lock
that has an index smaller than the given index.
2015-03-09 19:52:54 -07:00
Xiang Li
86429264fb
wal: support auto-cut in wal
...
WAL should control the cut logic itself. We want to do falloc to
per allocate the space for a segmented wal file at the beginning
and cut it when it size reaches the limit.
2015-02-28 11:18:59 -08:00
Xiang Li
84485643fe
*: expose wal metrics at /metrics
2015-02-28 11:06:11 -08:00
Xiang Li
fb1a28c65d
*: vendor prometheus
2015-02-28 11:06:11 -08:00
Xiang Li
9e63b1fb63
wal: record metrics
2015-02-28 10:12:35 -08:00
Yicheng Qin
e966e565c4
etcdctl/backup_command: handle datadir with missed snapshot mark
...
This helps to recover from the data dir created in v2.0.0-rc1.
2015-01-29 13:32:59 -08:00
Jonathan Boulle
f1ed69e883
*: switch to line comments for copyright
...
Build tags are not compatible with block comments.
Also adds copyright header to a few places it was missing.
2015-01-26 09:53:30 -08:00
Yicheng Qin
270e67db84
wal: not export unnecessary public functions
2015-01-09 14:55:10 -08:00
Yicheng Qin
6460e49a33
wal: save empty snapshot when create
...
So caller can open at empty snapshot to read all entries.
2015-01-06 19:48:21 -08:00
Yicheng Qin
84f62f21ee
wal: record and check snapshot
2015-01-06 16:27:40 -08:00
Jonathan Boulle
1ec98cb795
pkg/fileutil: sort filenames during ReadDir
2014-12-18 16:36:11 -08:00
Xiang Li
53bf7e4b5e
wal: rename openAtIndex -> open; OpenAtIndexUntilUsing -> openNotInUse
2014-12-14 19:33:06 -08:00
Xiang Li
f538cba272
*: do not backup files still in use
2014-12-14 19:27:22 -08:00
Xiang Li
ea94d19147
*: lock the in using files; do not purge locked the wal files
2014-12-14 19:27:22 -08:00
Yicheng Qin
b9bf957c6d
wal: sync after writing data to disk in Cut function
2014-12-04 22:56:34 -08:00
Yicheng Qin
af0f34c595
wal: save latest state into new WAL
...
So we could always read out state when open at valid index.
2014-12-04 12:19:21 -08:00
Yicheng Qin
aa61009560
wal: not return ErrIndexNotFound in ReadAll
...
This IndexNotFound case is reasonable now because we don't write dummy
entries into wals any more.
2014-12-02 00:28:54 -08:00
Xiang Li
d3db010190
*: support purging old wal/snap files
2014-12-01 11:50:17 -08:00
Jonathan Boulle
5a964f49a5
wal: propagate errors
2014-11-08 17:16:23 -08:00
Xiang Li
075ab6415f
Merge pull request #1587 from xiangli-cmu/fix_wal
...
wal: sync before returning from create
2014-11-03 15:58:47 -08:00
Xiang Li
ff1f5a9d57
wal: sync before returning from create
2014-11-03 14:28:59 -08:00
Yicheng Qin
433b4138c5
etcdserver: panic on storage error
...
It is a critical error to etcd, and etcd is not able to recover it now.
2014-11-03 10:46:04 -08:00
Jonathan Boulle
7a4d42166b
*: add license header to all source files
2014-10-17 15:41:22 -07:00
Xiang Li
f98fbbfc14
*: proto refactoring
2014-10-14 21:07:23 +08:00
Yicheng Qin
447caf1afc
etcdserver/wal: record info at the head of WAL file
2014-10-10 11:57:09 -07:00
Xiang Li
8bbbaa88b2
*: raft related int64 -> uint64
2014-10-09 14:29:21 +08:00
Yicheng Qin
1d5d2e3726
*: Id -> ID for protobuf types
...
We use ID instead of Id in this project based on golang conventions.
2014-09-26 11:49:30 -07:00
Jonathan Boulle
cf4af47f7e
wal: remove logging messages
...
These are noisy and unhelpful by default. We can re-add if at some point
we reintroduce debugging/multiple log levels.
2014-09-22 19:08:47 -07:00
Yicheng Qin
df70f653a4
Merge pull request #1099 from unihorn/128
...
wal: OpenFromIndex fails if it cannot find previous index
2014-09-17 16:00:51 -07:00
Yicheng Qin
29f6d8a9e6
wal: add ErrFileNotFound and ErrIndexNotFound
2014-09-17 15:58:06 -07:00
Yicheng Qin
7160b5ae26
wal: OpenFromIndex fails if it cannot find previous index
...
Example:
We save entry 1, 2, 3 to WAL.
If we try to open 100, it should fail.
2014-09-17 15:07:07 -07:00
Xiang Li
ab61a8aa9a
*: init for on disk snap support
2014-09-17 13:56:12 -07:00
Yicheng Qin
de21c39ca5
raft: isStateEqual -> isHardStateEqual, IsEmptyState -> IsEmptyHardState
2014-09-16 13:55:00 -07:00
Yicheng Qin
023dc7cba2
etcdserver: add SYNC request
2014-09-16 13:42:03 -07:00
Xiang Li
69f2d5c590
wal: refactor
2014-09-15 10:35:36 -07:00
Xiang Li
77fbd2610c
wal: add comment
2014-09-15 10:17:21 -07:00
Xiang Li
1d09c25f5f
wal: add walName function; cleanup test
2014-09-15 10:17:21 -07:00
Xiang Li
cec1956b8f
wal: better comment
2014-09-15 10:14:18 -07:00
Xiang Li
8e0ee1cc5e
wal: remove wrong comment for cut
2014-09-15 10:14:18 -07:00
Xiang Li
1164c4b83d
wal: recover writting seq
2014-09-15 10:14:18 -07:00
Xiang Li
21860bc017
wal: cut(i uint64) -> cut
2014-09-15 10:14:18 -07:00
Yicheng Qin
140fd6d6c4
raft: restart using last written entry also
2014-09-15 09:56:33 -07:00
Yicheng Qin
a9af70c52b
raft: write entry 0 into log
2014-09-15 09:55:52 -07:00
Yicheng Qin
2030ca202f
wal: change wal filename format
...
Make raftIndex section to be expected raftIndex of next entry.
It makes filename more intuitive and straight-forward.
The commit also adds comments for filename format.
2014-09-12 11:24:28 -07:00
Brandon Philips
884c702512
wal: wal.OpenFromIndex -> wal.OpenAtIndex
...
The first entry read from the wal should be the index provided. This
name makes that more clear.
2014-09-11 12:52:06 -07:00
Brian Waldon
c251304068
*: s/Id/ID/
...
golang convention dictates that the individual characters in an
abbreviation should all have the same case. Use ID instead of Id.
The protobuf generator still generates code that does not meet
this convention, but that's a fight for another day.
2014-09-10 16:09:08 -07:00
Xiang Li
90c0db3d42
wal: do not save empty state
2014-09-09 10:15:29 -07:00
Xiang Li
54734b0903
main/wal: add a const for 0700 magic number
2014-09-08 15:45:58 -07:00
Xiang Li
b094410066
wal: change entries->ents for consistency
2014-09-08 15:40:12 -07:00
Xiang Li
adff0f3813
wal: named return values for ReadAll.
2014-09-08 15:36:25 -07:00
Xiang Li
0461c517e4
wal: clarify TODO
2014-09-08 13:57:35 -07:00
Xiang Li
1a6e908971
*: add wal
2014-09-05 09:52:40 -07:00
Xiang Li
b98cf17209
wal: move pb files to walpb
2014-09-03 16:46:42 -07:00
Xiang Li
bdb954b2f5
wal: fix wal
2014-09-03 15:10:15 -07:00
Xiang Li
85103adfe0
wal: move record method to record.go
2014-09-03 09:20:11 -07:00
Yicheng Qin
442cae6844
wal: support multiple files
2014-09-03 09:20:10 -07:00
Yicheng Qin
b5b2031d5b
wal: prune repeative logging
2014-09-03 09:20:04 -07:00
Xiang Li
659eb5fd2a
wal: make record a protobuf type
2014-09-03 09:20:03 -07:00
Xiang Li
8e06333d45
raft: make State a protobuf type
2014-09-03 09:20:03 -07:00
Xiang Li
d6c3ebb1a0
raft: make Info a protobuf type
2014-09-03 09:20:03 -07:00
Yicheng Qin
2d870fa65b
wal: add first level logging
2014-09-03 09:20:01 -07:00
Yicheng Qin
969b529b08
wal: fix O_RDONLY attr when opening old file
2014-09-03 09:20:01 -07:00
Xiang Li
38ec659cd6
raft: make Entry a protobuf type
2014-09-03 09:20:01 -07:00
Xiang Li
43c9ca895b
wal: fix append entry
2014-09-03 09:20:01 -07:00
Yicheng Qin
28634fce47
wal: Flush -> Sync
2014-09-03 09:20:00 -07:00
Xiang Li
d70df4a15d
wal: reorgnize wal file
2014-09-03 09:19:59 -07:00
Xiang Li
a25cd45876
wal: add CoreOS copyright
2014-09-03 09:19:59 -07:00
Xiang Li
9f5c7b310c
wal: add newWal helper func
2014-09-03 09:19:59 -07:00
Xiang Li
9c6f2ed5bb
wal: use buffer
2014-09-03 09:19:59 -07:00
Xiang Li
5baefcce26
wal: cleanup
2014-09-03 09:19:59 -07:00
Yicheng Qin
363e952551
wal: add Read
2014-09-03 09:19:59 -07:00
Xiang Li
d1cdc02afc
wal: add tests for write
2014-09-03 09:19:59 -07:00
Xiang Li
7b180d585e
wal: make new work
2014-09-03 09:19:58 -07:00
Xiang Li
e72e75876c
wal: init
2014-09-03 09:19:58 -07:00