Upgrading from v1.0.1.
Upgrading related dependencies
------------------------------
The following dependencies also had to be upgraded:
- go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.26.1
From v0.25.0. This gets rid of a transitive dependency on go.opentelemetry.io/otel@v1.0.1.
- google.golang.org/genproto@v0.0.0-20211118181313-81c1377c94b1
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
Motivation is as follows:
- etcdctl we only depend on clientv3 APIs, no dependencies of bolt, backend, mvcc, file-layout
- etcdctl can be officially supported across wide range of versions, while etcdutl is pretty specific to file format at particular version.
it's step towards desired modules layout, documented in: https://etcd.io/docs/next/dev-internal/modules/
clientv3 logs (especially tests) were poluted with unattributed to testing.T log lines:
```
{"level":"warn","ts":"2021-04-29T12:42:11.055+0200","logger":"etcd-client","caller":"v3/retry_interceptor.go:64","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000fafc0/#initially=[unix://localhost:m10]","attempt":0,"error":"rpc error: code = ResourceExhausted desc = etcdserver: mvcc: database space exceeded"}
```
The reasons were 2 fold:
- Interceptors were copying logger before "WithLogger" could modify it.
- We were not propagating the loggers in a few testing contexts.
Prior to this CL, `ETCDCTL_API=2 etcdctl backup --with-v3` was readacting WAL log
(by removal of some entries), but was NOT updating consistent_index in the backend.
Also the WAL editing logic was buggy, as it didn't took in consideration the fact
that when TERM changes, there can be entries with duplicated indexes in
the log. So its NOT sufficient to subtract number of removed entries to
get accurate log indexes.
The PR replaces removing and shifting of WAL entries with replacing them with an no-op entries.
Thanks to this consistent-index references are staying up to date.
The PR also:
- updates 'verification' logic to check whether consistent_index does not lag befor last snapshot
- env-gated execution of verification framework in `etcdctl backup`.
Tested with:
```
(./build.sh && cd tests && EXPECT_DEBUG=TRUE 'env' 'go' 'test' '-timeout=300m' 'go.etcd.io/etcd/tests/v3/e2e' -run=TestCtlV2Backup --count=1000 2>&1 | tee TestCtlV2BackupV3.log)
```
correct 'backend' (bbolt) context in aspect of membership.
Prior to this change the 'restored' backend used to still contain:
- old memberid (mvcc deletion used, why the membership is in bolt
bucket, but not mvcc part):
```
mvs := mvcc.NewStore(s.lg, be, lessor, ci, mvcc.StoreConfig{CompactionBatchLimit: math.MaxInt32})
defer mvs.Close()
txn := mvs.Write(traceutil.TODO())
btx := be.BatchTx()
del := func(k, v []byte) error {
txn.DeleteRange(k, nil)
return nil
}
// delete stored members from old cluster since using new members
btx.UnsafeForEach([]byte("members"), del)
```
- didn't get new members added.
ClusterVersionSet, ClusterMemberAttrSet, DowngradeInfoSet functions are
writing both to V2store and backend. Prior this CL there were
in a branch not executed if shouldApplyV3 was false,
e.g. during restore when Backend is up-to-date (has high
consistency-index) while v2store requires replay from WAL log.
The most serious consequence of this bug was that v2store after restore
could have different index (revision) than the same exact store before restore,
so potentially different content between replicas.
Also this change is supressing double-applying of Membership
(ClusterConfig) changes on Backend (store v3) - that lackilly are not
part of MVCC/KeyValue store, so they didn't caused Revisions to be
bumped.
Inspired by jingyih@ comment:
https://github.com/etcd-io/etcd/pull/12820#issuecomment-815299406
Motivation:
- ServerConfig is part of 'embed' public API, while etcdserver is more 'internal'
- EtcdServer is already too big and config is pretty wide-spread leaf
if we were to split etcdserver (e.g. into pre & post-apply part).
The flag protects etcd memory from being swapped out to disk.
This can happen in memory constrained systems where mmaped bbolt
area is natural condidate for swapping out.
This flag should provide better tail latency on the cost of higher RSS
ram usage. If the experiment is successful, the logic should get moved
into bbolt layer, where we can protect specific bbolt instances
(e.g. avoid protecting both during defragmentation).
Seems old versions of https://github.com/certifi/gocertifi where
categorized as "Do What The F*ck You Want To Public License".
Update to newer version that is explicit `Mozilla Public License` 2.0 (MPL 2.0).
dgrijalva/jwt-go has been abandoned and contains several serious
security issues. Most projects are now switching to the form3tech fork.
See https://snyk.io/vuln/SNYK-GOLANG-GITHUBCOMDGRIJALVAJWTGO-596515 for
info on the issues.
Signed-off-by: Dan Lorenc <dlorenc@google.com>
instead of v3.0.0-000101010000000-00000000000,
that might be misleading as we don't develop etcd v3.0.0 any longer.
This version is a virtual version and is not supposed to be tagged
within the repository. We should tag real versions like: 3.5.0-alpha.0.
Please notice that go.etcd.io/etcd/client/v2 will be versioned as `v2.305.0-pre`.
The reason is that client v2 must have v2 version. I propose a
convention to envode the major version as 100x in minor version to make
the association to the underlying repository clear, staying within v2
version family.
The change was generated using:
```
DRY_RUN=false TARGET_VERSION="v3.5.0-pre" ./scripts/release_mod.sh update_versions
```
We make v2 client code a module go.etcd.io/etcd/client/v2.
Pretty mechanical change that can be summarized as:
mkdir client/v2
cd client/v2 && git mod init go.etcd.io/etcd/client/v2
git mv client/*.go client/v2/
find -name '*.go' | xargs sed -i --follow-symlinks 's|/v3/client["]|/client/v2\"|g'
+ fixing changelog, bom, go.mod etc.
"snapshot" Restore/Status code was the only remaining dependency of client on 'server'
code. The code is solelly used by etcdctl. Long-term the snapshot code
should be migrated to 'etcdadm' style of tool such that we can
distinguish tool solelly depending on networking API vs. tools that
operation on etcd files directly.
We left snapshot.Save() code in clientv3.snapshot package, such that
clients can benefits from automated download&safe to file snapshot
functionality over the wire.
This CL fixes:
COVERDIR=./coverage PASSES="build_cov" && go test --tags cov -v ./tests/e2e
and is part of the effort to make:
COVERDIR=coverage PASSES="build_cov cov" ./test
fully pass.
The args passed to ./bin/etcd_test and ./bin/etcdctl_test binaries were
mismatched. The protocol of passing the arguments using
environment variables has been replaces with proper passing of flags.
How the measurement of coverage by e2e tests works:
1. COVERDIR=./coverage PASSES="build_cov" are generating
./bin/etcd_test and ./bin/etcdctl_test binaries.
2. These binaries are tests (as coverage can be computed only for
tests) [see ./main_test.go ./etcdctl/main_test.go], but this tests are
running the main logic of the server and uppon termination (or SIGTERM
signal) are writting proper .coverprofile files in the $COVERDIR folder.
The binaries used to take arguments using env variables, but its not
needed any longer. The binaries can consume any command line arguments
that either test (so --test.fooo) or the original binary can consume.
3. The tests/e2e (when compiled with the --tags cov) are starting the
_test binaries instead of the original binaries, such that the coverage
is being collected.
This change makes the etcd package compatible with the existing Go
ecosystem for module versioning.
Used this tool to update package imports:
https://github.com/KSubedi/gomove
This changes have started at etcdctl under auth.go, and make changes to stub out everything down into the internal raft. Made changes to the .proto files and regenerated them so that the local version would build successfully.
This commit adds a feature for creating a user without password. The
purpose of the feature is reducing attack surface by configuring bad
passwords (CN based auth will be allowed for the user).
The feature can be used with `--no-password` of `etcdctl user add`
command.
Fix https://github.com/coreos/etcd/issues/9590
Add documentation to clarify that when writing TXN commands, multi-line values should be written using "\n" and not a literal newline (as in other commands).
Fixes#10169
Don't panic if command is given in interactive mode, give a nice error
message instead.
Before:
$ ./bin/etcdctl watch -i
<hit return>
panic: runtime error: index out of range
goroutine 1 [running]:
etcdctl/ctlv3/command.watchInteractiveFunc(...)
etcd/etcdctl/ctlv3/command/watch_command.go:104 ...
After:
$ ./bin/etcdctl watch -i
<hit return>
Invalid command: (watch and progress supported)
foo
Invalid command foo (only support watch)
Maintain existing error message for not-enough-args
Add "too many args" if too many args
Add more helpful error message if v2 syntax was used
New output:
```
sauer@host:~/dev/etcd$ ./bin/etcdctl --endpoints http://localhost:5001 member add
Error: member name not provided.
sauer@host:~/dev/etcd$ ./bin/etcdctl --endpoints http://localhost:5001 member add node2 node2
Error: too many arguments
sauer@host:~/dev/etcd$ ./bin/etcdctl --endpoints http://localhost:5001 member add node2 http://localhost:6002
Error: too many arguments, did you mean "--peer-urls http://localhost:6002"
sauer@host:~/dev/etcd$ ./bin/etcdctl --endpoints http://localhost:5001 member add http://localhost:6002 node2
Error: too many arguments, did you mean "--peer-urls http://localhost:6002"
```
Piotr Tabor