To fix a panic that happens when trying to get ids of etcd members in
force new cluster mode, the issue happen if the cluster previously had
etcd learner nodes added to the cluster
Fixes#12285
Currently, the grpc gateway is enabled by default when run the etcd server
using command line configuration. However, when provide config through
config file, the grpc gateway is disabled by default. We should either use
the same approach or at least document this.
related #https://github.com/etcd-io/etcd/issues/12093
grpc proxy opens additional 2 watching channels. The metric is shared
between etcd-server & grpc_proxy, so all assertions on number of open
watch channels need to take in consideration the additional "2"
channels.
Prior to the fix:
```
go test -mod=mod -timeout 30m -tags cluster_proxy "./integration/..."
```
Was failing with:
```
integration/cluster_proxy.go:58:37: not enough arguments in call to grpcproxy.NewWatchProxy
have (*clientv3.Client)
want (*zap.Logger, *clientv3.Client)
FAIL go.etcd.io/etcd/v3/integration [build failed]
```
This CL fixes:
COVERDIR=./coverage PASSES="build_cov" && go test --tags cov -v ./tests/e2e
and is part of the effort to make:
COVERDIR=coverage PASSES="build_cov cov" ./test
fully pass.
The args passed to ./bin/etcd_test and ./bin/etcdctl_test binaries were
mismatched. The protocol of passing the arguments using
environment variables has been replaces with proper passing of flags.
How the measurement of coverage by e2e tests works:
1. COVERDIR=./coverage PASSES="build_cov" are generating
./bin/etcd_test and ./bin/etcdctl_test binaries.
2. These binaries are tests (as coverage can be computed only for
tests) [see ./main_test.go ./etcdctl/main_test.go], but this tests are
running the main logic of the server and uppon termination (or SIGTERM
signal) are writting proper .coverprofile files in the $COVERDIR folder.
The binaries used to take arguments using env variables, but its not
needed any longer. The binaries can consume any command line arguments
that either test (so --test.fooo) or the original binary can consume.
3. The tests/e2e (when compiled with the --tags cov) are starting the
_test binaries instead of the original binaries, such that the coverage
is being collected.
Change tests/e2e to use proper (client-nocn.crt) certificate when
running in tags="cluster_proxy" mode.
Thanks to this (and previous in this PR) changes, the following test run
finally succeeds:
./build && go test --tags "cluster_proxy" -v ./tests/e2e/...
We have following communication schema:
client --- 1 ---> grpc-proxy --- 2 --- > etcd-server
There are 2 sets of flags/certs in grpc proxy [ https://github.com/etcd-io/etcd/blob/master/etcdmain/grpc_proxy.go#L140 ]:
A. (cert-file, key-file, trusted-ca-file, auto-tls) this are controlling [1] so client to proxy connection and in particular they are describing proxy public identity.
B. (cert,key, cacert ) - these are controlling [2] so what's the identity that proxy uses to make connections to the etcd-server.
If 2 (B.) contains certificate with CN and etcd-server is running with --client-cert-auth=true, the CN can be used as identity of 'client' from service perspective. This is permission escalation, that we should forbid.
If 1 (A.) contains certificate with CN - it should be considered perfectly valid. The server can (should) have full identity.
So only --cert flag (and not --cert-file flag) should be validated for empty CN.
Executed:
(cd ./integration/fixtures && ./gencerts.sh)
This in particular cereated a new client-nocn.crt (and key) that can be
used for testing grpc-proxy -> etcd-server connections.
integration/fixtures/gencerts.sh:
- refactored common logic to a helper function
- added definition for client-nocn certificate
(used for grpc-proxy -> etcd-server) communication.
This is to aid with debugging the effectiveness of systems that
manually take care of cluster compaction, and have greater visibity
into recent compactions.
It can be handy to alert on the exactly how long it was since a
compaction (and also to put on dashboards) had happened.
---
Tested using a test cluster, the final result looks like this:
```
root@etcd-1:~# ETCDCTL_API=3 /tmp/test-etcd/etcdctl --endpoints=192.168.232.10:2379 compact 1012
compacted revision 1012
root@etcd-1:~# curl -s 192.168.232.10:2379/metrics | grep last
# HELP etcd_debugging_mvcc_db_compaction_last The unix time since the last db compaction. Resets to 0 on start.
# TYPE etcd_debugging_mvcc_db_compaction_last gauge
etcd_debugging_mvcc_db_compaction_last 1.595873939e+09
```
From etcd-dev discussion:
https://groups.google.com/u/2/g/etcd-dev/c/oMGSBqs_7sc
I have been working on this system called Asset Transparency[1] which
helps users verify they have received the correct contents from a URL.
If you are familiar with the "download a file, download a SHA256SUM
file, run `sha256sum -c`, etc" process? This tool helps to automate
that for users into something like this[2]:
$ tl get https://github.com/etcd-io/etcd/releases/download/v3.4.12/etcd-v3.4.12-darwin-amd64.zip
And a best practice for this Asset Transparency system is that URLs
are registered with the log as soon as possible. Why? Well, the sooner
a URL is entered the longer it can protect people consuming a URL from
unexpected content modification from say a GitHub credential
compromise.
To that end I have written a GitHub Action[3] that will automatically
do that on every release. It is easy to activate and should be hands
free after installation. So, before I enable it I want to see if there
are any concerns from maintainers. The only change to our repo will be
a new file in .github/workflows.
[1] https://www.transparencylog.com
[2] https://github.com/transparencylog/tl
[3] https://github.com/transparencylog/publish-releases-asset-transparency-action
Similar counts are exposed via Prometheus.
This adds the one that are perceived by etcd server.
e.g.
os_fd_limit 120000
os_fd_used 14
process_cpu_seconds_total 0.31
process_max_fds 120000
process_open_fds 17
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
Piotr Tabor