Commit Graph

80 Commits (8907b146d449a5edd680d6388d7af409be5cd2c9)

Author SHA1 Message Date
Piotr Tabor 00de56a4a4 etcdserver, wal: Fix tests that were performing unintended casting of int to String.
Fixes following problems during "./etcd# go test ./..."

> go.etcd.io/etcd/v3/etcdserver/api/v2store_test
etcdserver/api/v2store/store_test.go:847:24: conversion from int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)

> go.etcd.io/etcd/v3/wal
wal/wal_test.go:242:68: conversion from int to string yields a string of one rune, not a string of digits (did you mean fmt.Sprint(x)?)
2020-08-04 20:28:41 +02:00
tangcong 02b9ad3939 wal: add TestValidSnapshotEntriesAfterPurgeWal testcase 2020-05-21 00:51:53 +08:00
Joe Betz b68eea236e etcdserver/*, wal/*:Add comments, clean up error messages and tests 2020-05-14 13:34:55 -07:00
Joe Betz bd16071846
etcdserver/*, wal/*: find valid snapshots by cross checking snap files and wal snap entries 2020-05-14 09:22:45 -07:00
Brandon Philips 96cce208c2 go.mod: use go.etcd.io/etcd/v3 versioning
This change makes the etcd package compatible with the existing Go
ecosystem for module versioning.

Used this tool to update package imports:
  https://github.com/KSubedi/gomove
2020-04-28 00:57:35 +00:00
Sahdev P. Zala 0d5c692eb6 wal: fix panic when decoder not set
Handle the related panic and clarify doc.
2020-04-23 15:53:05 -04:00
Gyuho Lee c716d764d0 wal: check out of range slice in "ReadAll", "decoder"
wal: add slice bound checks in decoder

CHANGELOG-3.5: add wal slice bound check
CHANGELOG-3.5: add "decodeRecord"

Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2020-04-22 19:07:40 -07:00
Gyuho Lee 34bd797e67 *: revert module import paths
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2019-05-28 15:39:35 -07:00
Joshua Coutinho a0c889d14b wal: add a test for wal cleanup, improve comments
To add test coverage of wal cleanup.
2019-05-10 22:36:26 +01:00
shivaramr 9150bf52d6 go modules: Fix module path version to include version number 2019-04-26 15:29:50 -07:00
Shreyas Rao bb3eb8fea9 wal: Add test for Verify
Signed-off-by: Shreyas Rao <shreyas.sriganesh.rao@sap.com>
2019-03-12 22:25:25 +05:30
Gyuho Lee 038fd844ac wal: update Go import paths to "go.etcd.io/etcd"
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2018-08-28 17:47:55 -07:00
Gyuho Lee 6fabe6d5ab wal: clean up code format
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-07-21 15:49:58 -07:00
Gyuho Lee e15ce28168 wal: add missing logs, improve pipeline test coverage
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-05-17 11:01:31 -07:00
Gyuho Lee fdbedacc83 wal: support structured logger
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-16 17:36:00 -07:00
lorneli 7c50c06fb8 wal: tiny refactor
a. add comment of reopening file in cut function.
b. add const frameSizeBytes in decoder.
c. return directly if locked files empty in ReleaseLockTo function.
2017-09-07 02:50:37 +08:00
Gyu-Ho Lee aca2abd8fe *: use 'io.Seek*' for go1.7+
For https://github.com/coreos/etcd/issues/6174.

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-03-20 15:15:24 -07:00
Tess Rinearson 39c733ebe7 wal: use path/filepath instead of path
Use the path/filepath package instead of the path package. The
path package assumes slash-separated paths, which doesn't work
on Windows. But path/filepath manipulates filename paths in a way
that's compatible across OSes.
2017-03-15 17:30:23 -07:00
Gyu-Ho Lee 3d75395875 *: remove never-unused vars, minor lint fix
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-03-06 14:59:12 -08:00
Gyu-Ho Lee f5588526cc wal: set PageWriter offset in file encoder 2016-09-28 11:03:24 -07:00
Vimal Kumar 064411b51c wal/wal.go : improved coverage by testing WAL.Save which causes a WAL.cut to happen 2016-09-21 16:50:55 +05:30
Aaron Lehmann af4f82228c wal: hold file lock while renaming WAL directory on non-Windows
Windows requires this lock to be released before the directory is
renamed. But on unix-like operating systems, releasing the lock and
trying to reacquire it immediately can be flaky if a process is forked
around the same time. The file descriptors are marked as close-on-exec
by the Go runtime, but there is a window between the fork and exec where
another process will be holding the lock.
2016-08-26 09:27:51 -07:00
Anthony Romano 0d8d9a374c wal: test for truncation on torn writes 2016-08-25 14:24:46 -07:00
Gyu-Ho Lee b4aa4607cb wal: use bytes.Equal, other minor updates
- Replace reflect.Equal with bytes.Equal where possible
- Remove some TODOs
- Some minor simplifications
2016-06-13 01:33:53 -07:00
Gyu-Ho Lee 4570eddc2c wal: PrivateFileMode/DirMode as in pkg/fileutil
To make it consistent with pkg/fileutil
2016-06-10 15:20:57 -07:00
Gyu-Ho Lee 4a5befc2de wal: update LICENSE header 2016-05-12 20:50:04 -07:00
Anthony Romano 17391336af wal: atomically initialize wal directory
Fixes #5270
2016-05-11 16:50:17 -07:00
Gyu-Ho Lee 641a1a66e1 *: fix govet -shadow in go tip 2016-04-15 07:39:52 -07:00
Anthony Romano bfe3a3d08e wal: fix tail corruption
On ReadAll, WAL seeks to the end of the last record in the tail. If the tail did not
end with preallocated space, the decoder would report 0 as the last offset and begin
writing at offset 0 of the tail.

Fixes #4903
2016-04-01 15:05:52 -07:00
Anthony Romano 24b806d2ee wal: preallocate WAL files with initial size equal to segment size
Avoids having to update file size metadata during fdatasync on common path.

Fixes #4755
2016-03-21 11:56:53 -07:00
Anthony Romano 7397e14c0a fileutil, wal: refactor file locking
File lock interface was more verbose than it needed to be while
simultaneously making it difficult to support systems (e.g., Windows)
that only permit locked writes on a single fd holding the lock.
2016-03-16 15:02:15 -07:00
Gyu-Ho Lee a4de207d53 wal: fix shadowed variables
Fixes for https://github.com/coreos/etcd/issues/3954.
2015-12-12 09:38:26 -08:00
Dmitry Smirnov b2f4a5f587 *: fix spelling issues (codespell).
Signed-off-by: Dmitry Smirnov <onlyjob@member.fsf.org>
2015-09-11 10:22:29 +10:00
Xiang Li f59da0e453 *:fix point-in-time backup
Backup process should be able to read all WALs until io.EOF to
generate a point-in-time backup.

Our WAL file is append-only. And the backup process will lock all
files before start reading, which can prevent the gc routine from
removing any files in the middle.
2015-06-15 11:12:28 -07:00
Yicheng Qin 3dd6e0b88f wal: fix missing import 2015-03-24 22:53:15 -07:00
Xiang Li 6e6669d696 wal: releastTo should work with large release index 2015-03-24 22:34:26 -07:00
Xiang Li b66eb3d81c wal: fix ReleaseLockTo
ReleaseLockTo should not release the lock on the WAL
segment that is right before the given index. When
restarting etcd, etcd needs to read from the WAL segment
that has a smaller index than the snapshot index.

The correct behavior is that ReleaseLockTo releases
the locks w is holding so that w only holds one lock
that has an index smaller than the given index.
2015-03-09 19:52:54 -07:00
Xiang Li 86429264fb wal: support auto-cut in wal
WAL should control the cut logic itself. We want to do falloc to
per allocate the space for a segmented wal file at the beginning
and cut it when it size reaches the limit.
2015-02-28 11:18:59 -08:00
Barak Michener 92dca0af0f *: remove shadowing of variables from etcd and add travis test
We've been bitten by this enough times that I wrote a tool so that
it never happens again.
2015-02-17 16:31:42 -05:00
Jonathan Boulle f1ed69e883 *: switch to line comments for copyright
Build tags are not compatible with block comments.
Also adds copyright header to a few places it was missing.
2015-01-26 09:53:30 -08:00
Yicheng Qin 9bdc343b7c wal: add ReleaseLockTo test 2015-01-09 14:59:41 -08:00
Yicheng Qin 270e67db84 wal: not export unnecessary public functions 2015-01-09 14:55:10 -08:00
Yicheng Qin 6460e49a33 wal: save empty snapshot when create
So caller can open at empty snapshot to read all entries.
2015-01-06 19:48:21 -08:00
Yicheng Qin 84f62f21ee wal: record and check snapshot 2015-01-06 16:27:40 -08:00
Xiang Li 53bf7e4b5e wal: rename openAtIndex -> open; OpenAtIndexUntilUsing -> openNotInUse 2014-12-14 19:33:06 -08:00
Xiang Li ea94d19147 *: lock the in using files; do not purge locked the wal files 2014-12-14 19:27:22 -08:00
Yicheng Qin b9bf957c6d wal: sync after writing data to disk in Cut function 2014-12-04 22:56:34 -08:00
Yicheng Qin af0f34c595 wal: save latest state into new WAL
So we could always read out state when open at valid index.
2014-12-04 12:19:21 -08:00
Yicheng Qin aa61009560 wal: not return ErrIndexNotFound in ReadAll
This IndexNotFound case is reasonable now because we don't write dummy
entries into wals any more.
2014-12-02 00:28:54 -08:00
Xiang Li ff1f5a9d57 wal: sync before returning from create 2014-11-03 14:28:59 -08:00