66cb045d47
etcdserver, et al: add --unsafe-no-fsync flag
...
This makes it possible to run an etcd node for testing and development
without placing lots of load on the file system.
Fixes #11930 .
Signed-off-by: David Crawshaw <crawshaw@tailscale.com>
2020-05-26 09:21:49 +10:00
1836cbb15e
wal: fix crc mismatch crash bug
2020-05-20 13:21:07 +08:00
b68eea236e
etcdserver/*, wal/*:Add comments, clean up error messages and tests
2020-05-14 13:34:55 -07:00
bd16071846
etcdserver/*, wal/*: find valid snapshots by cross checking snap files and wal snap entries
2020-05-14 09:22:45 -07:00
50517039ae
etcdserver/*, wal/*: add Sync method
2020-05-13 08:47:47 -07:00
3d2b565f98
etcdserver/*: changes to snapshots and wal logic to fix #10219
2020-05-13 08:46:15 -07:00
9162cd613d
etcdserver/*, wal/*: changes to snapshots and wal logic to fix #10219
2020-05-13 08:42:03 -07:00
fb77f9b1d5
Merge pull request #11814 from tedyu/dir-closer
...
wal: close the directory when Fsync() encounters error
2020-04-28 23:27:08 -07:00
3b79ed8506
Merge pull request #11821 from tedyu/wal-close-all
...
closeAll should attempt to close all ReadCloser's
2020-04-28 20:54:13 -07:00
62755f484e
wal: closeAll should attempt to close all ReadCloser's
...
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
2020-04-28 20:33:40 -07:00
96cce208c2
go.mod: use go.etcd.io/etcd/v3 versioning
...
This change makes the etcd package compatible with the existing Go
ecosystem for module versioning.
Used this tool to update package imports:
https://github.com/KSubedi/gomove
2020-04-28 00:57:35 +00:00
99eb5cdfdb
wal: close the directory when Fsync() encounters error
...
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
2020-04-25 11:08:39 -07:00
9d0ecc3ba3
wal: Use known dir-path in log message
...
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
2020-04-25 09:14:30 -07:00
0d5c692eb6
wal: fix panic when decoder not set
...
Handle the related panic and clarify doc.
2020-04-23 15:53:05 -04:00
c716d764d0
wal: check out of range slice in "ReadAll", "decoder"
...
wal: add slice bound checks in decoder
CHANGELOG-3.5: add wal slice bound check
CHANGELOG-3.5: add "decodeRecord"
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2020-04-22 19:07:40 -07:00
af261f1a1b
wal: add "etcd_wal_writes_bytes_total"
...
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2020-03-31 20:31:32 -07:00
2fee0b153e
wal: remove capnslog
2020-02-06 06:11:59 -08:00
adbae79565
wal : wal.Verify defer close the opened WAL files
...
wal.Verify() : The opened WAL files are not closed in error cases. Fixed by adding a
defer.
2019-09-04 23:38:14 +01:00
34bd797e67
*: revert module import paths
...
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2019-05-28 15:39:35 -07:00
a0c889d14b
wal: add a test for wal cleanup, improve comments
...
To add test coverage of wal cleanup.
2019-05-10 22:36:26 +01:00
f7f7e9c762
wal: Improve cleanup for robustness and debuggability
...
Rename wal with '.suffix.<timestamp>' instead of delete it and call cleanup when perr in a 'defer'ed statement.
2019-05-07 21:38:40 +01:00
51035bfd84
wal: cleanup wal directory if creation fails
...
delete <data-dir>/member/wal if any operation after the rename in
wal.Create fails to avoid reading an inconsistent WAL on restart.
Fixes #10688
2019-05-04 01:58:57 +01:00
9150bf52d6
go modules: Fix module path version to include version number
2019-04-26 15:29:50 -07:00
914e5edb00
wal: include logger in WAL returned by openAtIndex
...
Signed-off-by: Shreyas Rao <shreyas.sriganesh.rao@sap.com>
2019-04-02 13:09:10 +05:30
3d6862fe0d
wal: add Verify function to perform corruption check on wal contents
...
Signed-off-by: Shreyas Rao <shreyas.sriganesh.rao@sap.com>
2019-03-12 22:25:25 +05:30
038fd844ac
wal: update Go import paths to "go.etcd.io/etcd"
...
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2018-08-28 17:47:55 -07:00
b0b966c43c
wal: document, clean up fsync histogram
...
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-23 14:03:28 -07:00
e15ce28168
wal: add missing logs, improve pipeline test coverage
...
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-17 11:01:31 -07:00
f3d9a85697
wal: add warnings on fsync, flock fail paths
...
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-03 14:01:06 -07:00
fdbedacc83
wal: support structured logger
...
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-16 17:36:00 -07:00
7c50c06fb8
wal: tiny refactor
...
a. add comment of reopening file in cut function.
b. add const frameSizeBytes in decoder.
c. return directly if locked files empty in ReleaseLockTo function.
2017-09-07 02:50:37 +08:00
fe1ddab714
wal: fall back to closing wal if locked dir rename fails
...
Detecting windows at compile time isn't enough since etcd might be
on linux but the fs is backed by windows.
Fixes : #8178
Fixes : #6984
2017-07-20 13:30:41 -07:00
aca2abd8fe
*: use 'io.Seek*' for go1.7+
...
For https://github.com/coreos/etcd/issues/6174 .
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-03-20 15:15:24 -07:00
39c733ebe7
wal: use path/filepath instead of path
...
Use the path/filepath package instead of the path package. The
path package assumes slash-separated paths, which doesn't work
on Windows. But path/filepath manipulates filename paths in a way
that's compatible across OSes.
2017-03-15 17:30:23 -07:00
ab03a42f06
raft: add Ready.MustSync
...
Add Ready.MustSync which indicates that the hard state and raft log
entries in a Ready message must be synchronously written to persistent
storage.
2017-02-13 15:13:21 -05:00
c26ebe3262
Merge pull request #6453 from vimalk78/wal-optimize-marshal-outside-lock
...
wal/wal.go: optimized WAL.SaveSnapshot to do Marshal outside the mutex lock
2016-10-03 11:50:11 -07:00
f5588526cc
wal: set PageWriter offset in file encoder
2016-09-28 11:03:24 -07:00
f4ec303d1b
wal/wal.go: modified WAL.SaveSnapshot to do the Marshal before aquiring the mutex
2016-09-28 10:35:19 +05:30
064411b51c
wal/wal.go : improved coverage by testing WAL.Save which causes a WAL.cut to happen
2016-09-21 16:50:55 +05:30
ccb46d2024
wal: simplify dir.Close call
2016-09-09 09:23:55 +09:00
bd7107bd4b
wal: fsync directory after wal file rename
...
Fixes #6368
2016-09-08 00:09:16 -07:00
af4f82228c
wal: hold file lock while renaming WAL directory on non-Windows
...
Windows requires this lock to be released before the directory is
renamed. But on unix-like operating systems, releasing the lock and
trying to reacquire it immediately can be flaky if a process is forked
around the same time. The file descriptors are marked as close-on-exec
by the Go runtime, but there is a window between the fork and exec where
another process will be holding the lock.
2016-08-26 09:27:51 -07:00
f1ead43482
wal: zero out wal tail past its first zero record
...
Whenever the WAL is opened for writes, it should write zeroes to its tail
starting from the first zero record. Otherwise, if there are entries past
the first zero record due to a torn write, any new writes that overlap the
old entries will lead to a garbage record on the tail and cause a CRC
mismatch.
2016-08-25 14:24:46 -07:00
2b996b6038
wal: Export SegmentSizeBytes as a variable
...
In test situations, it's useful to create smaller than usual WAL files
to test rotation and to avoid the overhead of preallocation on old-style
filesystems that don't handle it efficiently. This commit changes
segmentSizeBytes to an exported variable so that tests can override it
from an init() function.
2016-08-09 15:38:30 -07:00
5991209c2d
wal: release wal locks before renaming directory on init
...
Fixes #5852
2016-07-02 12:14:37 -07:00
6cfc03a5f9
wal: use CreateDirAll
2016-06-22 15:57:55 -07:00
b4aa4607cb
wal: use bytes.Equal, other minor updates
...
- Replace reflect.Equal with bytes.Equal where possible
- Remove some TODOs
- Some minor simplifications
2016-06-13 01:33:53 -07:00
3243795522
wal: simplify boolean return
2016-06-11 10:36:52 -07:00
4570eddc2c
wal: PrivateFileMode/DirMode as in pkg/fileutil
...
To make it consistent with pkg/fileutil
2016-06-10 15:20:57 -07:00
39eaa37dcf
wal: warn if sync exceeds a second
2016-06-08 11:03:18 -07:00
David Crawshaw