Commit Graph

16429 Commits (5c726cfe255a2ea50c2a1c8e5d0f893e60eb451a)

Author SHA1 Message Date
Vitaliy Filippov 5c726cfe25 etcdserver: Fix 64 KB websocket notification message limit
This fixes etcd being unable to send any message longer than 64 KB as
a notification over the websocket. This was because the older version
of grpc-websocket-proxy was used and WithMaxRespBodyBufferSize option
wasn't set.
2020-10-19 15:25:52 +03:00
Piotr Tabor b7f0f52a16
*: Refresh of dependencies (#12399)
This PR focuses on dependencies that have new stable versions.
2020-10-15 15:32:00 -07:00
Gyuho Lee e42127658c
Merge pull request #12391 from ptabor/20201013-module-client
Modularization: Move /client/ -> /client/v2 and make it a module
2020-10-15 10:26:44 -07:00
Piotr Tabor 45e4306508 client: Move client to client/v2 as a module.
We make v2 client code a module go.etcd.io/etcd/client/v2.

Pretty mechanical change that can be summarized as:

  mkdir client/v2
  cd client/v2 && git mod init go.etcd.io/etcd/client/v2

  git mv client/*.go client/v2/
  find -name '*.go' | xargs sed -i --follow-symlinks 's|/v3/client["]|/client/v2\"|g'

  + fixing changelog, bom, go.mod etc.
2020-10-15 14:39:43 +02:00
Gyuho Lee d0d927f0db
Merge pull request #12397 from ptabor/20201015-update-protos2
tools: Update tools & go.mod for protos (gogo/protobuf 1.3.1, grpc 1.29.1, grpc-gateway 1.14.6)
2020-10-14 10:31:00 -07:00
Piotr Tabor 371ddf0b69 tests: Update diagnostic update in tests after change of proto version. 2020-10-14 18:46:38 +02:00
Piotr Tabor cc7b4fab9f *: Execution of ./scripts/genproto.sh after versions change 2020-10-14 18:46:38 +02:00
Piotr Tabor da5ca2c0d5 tools: Update of go.mod for protos (gogo/protobuf 1.3.1, grpc 1.29.1, grpc-gateway 1.14.6)
This brings consistency between proto-generation code and actual versions of libraries being used in runtime:

github.com/gogo/protobuf                        v1.2.1,v1.0.0 -> v1.3.1
github.com/golang/protobuf                             v1.3.2 -> v1.3.5
github.com/grpc-ecosystem/grpc-gateway  v1.9.5,v1.4.1,v1.15.2 -> v1.14.6
google.golang.org/grpc                                v1.26.0 -> v1.29.1

Moved as far as possible, without bumping on grpc 1.30.0 "naming" decomissioning.
Please also notice that gogo/protobuf is likely to reach EOL: https://github.com/gogo/protobuf/issues/691
2020-10-14 18:46:38 +02:00
Jingyi Hu b08a46d37d
Merge pull request #12396 from ptabor/20201014-deps
test: Adding dependencies consistency check
2020-10-14 20:44:29 +08:00
Piotr Tabor f7f17dd6a1 test: Adding dependencies consistency check
Examplar output for artificially inconsistent dependencies:

```
% PASSES="dep" ./test
% 'env' 'GO111MODULE=off' 'go' 'get' 'github.com/myitcv/gobin'
% 'env' 'GO111MODULE=off' 'go' 'get' 'github.com/myitcv/gobin'
% 'gofail' 'disable' 'etcdserver/' 'mvcc/backend/'
Running with -cpu=4 --race

'dep' started at Wed 14 Oct 2020 09:56:34 AM CEST
% (cd api && 'go' 'list' '-m')
% (cd api && 'go' 'list' '-f' '{{if not .Indirect}}{{if .Version}}{{.Path}},{{.Version}},go.etcd.io/etcd/api/v3{{end}}{{end}}' '-m' 'all')
% (cd pkg && 'go' 'list' '-m')
% (cd pkg && 'go' 'list' '-f' '{{if not .Indirect}}{{if .Version}}{{.Path}},{{.Version}},go.etcd.io/etcd/pkg/v3{{end}}{{end}}' '-m' 'all')
% 'go' 'list' '-m'
% 'go' 'list' '-f' '{{if not .Indirect}}{{if .Version}}{{.Path}},{{.Version}},go.etcd.io/etcd/v3{{end}}{{end}}' '-m' 'all'
% (cd tests && 'go' 'list' '-m')
% (cd tests && 'go' 'list' '-f' '{{if not .Indirect}}{{if .Version}}{{.Path}},{{.Version}},go.etcd.io/etcd/tests/v3{{end}}{{end}}' '-m' 'all')
FAIL: inconsistent versions for depencency: github.com/coreos/go-semver
github.com/coreos/go-semver:
  - v0.2.0 from: go.etcd.io/etcd/api/v3
  - v0.3.0 from: go.etcd.io/etcd/v3
FAIL: inconsistent dependencies
FAIL: 'dep' failed at Wed 14 Oct 2020 09:56:35 AM CEST
```
2020-10-14 09:57:27 +02:00
Joe Betz afc5a717c9
Merge pull request #12390 from ptabor/20201013-scripts-fixes
tools: Migrate remaining tools to gobin
2020-10-13 13:58:51 -04:00
Piotr Tabor bc9e433ca2 tools: Migrate remaining tools to gobin
Replace ./scripts/install_tool.sh with `gobin`, such that we have
consistent handling for all tools needed for build and consistent
versioning within ./tools/mod/go.mod.

Side changes:
  - Expose /scripts/fix.sh that fixes formatting and bom across modules
  - Expose *.sh variants of scripts like build and ./test (first step
towards replacement).
  - Make stderr output of commands explicit and make commands use
different color than callouts.
2020-10-13 19:33:01 +02:00
Joe Betz 061d8e8ead
Merge pull request #12354 from ptabor/202009027-module-pkgs
Modularization: Make pkg a module (and distribute not pkg code)
2020-10-13 12:56:21 -04:00
vivian 667ed96957
wal: remove redundant war tmp dirs if create wal failed (#12388)
If we create 000000000-0000000.war failed in war tmp dir(XXX.tmp) and we should
remove the tmp dirs. If create war successfully in tmp dir and it will
be renamed to common dir and remove tmp dir(XXX.tmp) will also be ok.

Co-authored-by: yangweiwei <yangweiwei@cmss.chinamobile.com>
2020-10-13 08:11:52 -07:00
Piotr Tabor f408939aa1 bom: Update bill-of-materials generation logic (and refresh bom). 2020-10-13 07:50:08 +02:00
Piotr Tabor 04f63b6571 go.mod,tests: Update go.mod for tests and . to reflect new dependencies. 2020-10-13 00:09:27 +02:00
Piotr Tabor de55bb6331 pkg: Rename imports after making 'pkg' a module
find -name '*.go' | xargs sed --follow-symlinks -i 's|go.etcd.io/etcd/v3/pkg/|go.etcd.io/etcd/pkg/v3/|g'
go fmt ./...
2020-10-13 00:09:27 +02:00
Piotr Tabor bfa91d48af pkg: Estabilish pkg as a module 2020-10-12 23:58:09 +02:00
Piotr Tabor 1983c89e27 pkg/logutil, etcdserver: Move zap-raft logger to etcdserver.
This package depends on raft and is solelly used by etcdserver/raft.go.
So it does not fullfills conditions of:
```
pkg/ is a collection of utility packages used by etcd without being specific to etcd itself. A package belongs here
only if it could possibly be moved out into its own repository in the future.
```
2020-10-12 23:58:09 +02:00
Piotr Tabor 5ba46f625f pkg/mock: Update imports to point on new files locations.
find ./ -name '*.go' | xargs sed -i 's|go.etcd.io/etcd/v3/pkg/mock/mockserver|go.etcd.io/etcd/v3/client/mock/mockserver|g'
find ./ -name '*.go' | xargs sed -i 's|go.etcd.io/etcd/v3/pkg/mock|go.etcd.io/etcd/v3/server/mock|g'
go fmt ./...
2020-10-12 23:58:09 +02:00
Piotr Tabor 28036db6f0 Mechanical: Move mock packages out of ./pkg
Mechanical execution of:

```
git mv ./pkg/mock/mockserver ./client/pkg/mock
git mv ./pkg/mock/{mockstorage,mockstore,mockwait} ./server/pkg/mock
```

The packages depend on etcd API / protos - so they are NOT etcd-dependencies.
In such situation thay should be placed in 'pkg' folder.
2020-10-12 23:58:09 +02:00
Gyuho Lee 0b95e8cef1
Merge pull request #12374 from ptabor/20201009-new-genproto
script/genproto.sh: Refactor to be explicit about versions.
2020-10-12 14:25:43 -07:00
Gyuho Lee fa83da446b
Merge pull request #12369 from ptabor/20201007-snapshot-out
Modularization: clientv3/snapshot -> etcdctl/snapshot
2020-10-12 14:24:45 -07:00
Sahdev Zala 891c69a5cf
Merge pull request #12386 from 2hamed/patch-1
README: Fixed the link to `functional` test dir
2020-10-12 14:55:48 -04:00
Hamed Momeni 386bf316a4 README: Fixed the link to test dir 2020-10-11 20:40:34 +02:00
Piotr Tabor 30811a06aa etcdctl, clientv3/snapshot: snapshot (Restore,Status) code out of client
"snapshot" Restore/Status code was the only remaining dependency of client on 'server'
code. The code is solelly used by etcdctl. Long-term the snapshot code
should be migrated to 'etcdadm' style of tool such that we can
distinguish tool solelly depending on networking API vs. tools that
operation on etcd files directly.

We left snapshot.Save() code in clientv3.snapshot package, such that
clients can benefits from automated download&safe to file snapshot
functionality over the wire.
2020-10-10 14:14:57 +02:00
Piotr Tabor 00e49d0c10
build: Fix updating 'go.etcd.io/etcd/api/v3/version.GitSHA' + test (#12382)
During move of code to 'api' (0aab02e7b5),
I overlooked that the ./build script is setting the version on the 'symbol'.

I added a code to ./build script that checks whether the symbol's are in sync.
2020-10-09 17:32:15 -07:00
Sam Batschelet e1bf097928
Merge pull request #12266 from chaitanyaenr/etcd_fsync_alert
Add alerts for etcd fsync duration
2020-10-09 14:42:12 -04:00
Joe Betz bc0d8dad1e
Merge pull request #12380 from ptabor/20201009-move-single-integration
[important] tests/integration: Move misplaced integration test integration/v3_kv_test.go
2020-10-09 13:14:15 -04:00
Piotr Tabor 485030052a tests/integration: Move misplaces integration tests.
There was undetected 'conflict' between
  11ba1a6109
and 2c66612e0e.

Moving the file to proper location.
2020-10-09 09:39:55 +02:00
Josh Bleecher Snyder 24724af7f1
etcdserver: prevent crash using expvars when server is not running (#12376)
The raft.status expvar is added at init time.
This change ensures that evaluating that expvar variable
doesn't panic during evaluation, even when there is
no server running.
2020-10-08 17:37:25 -07:00
CFC4N 11ba1a6109
namespace: check IsWithFromKey if keyLen equal 0. (#12307)
* namespace: check IsWithFromKey if keyLen equal 0.

Rename function isWithFromKey/isWithPrefix to IsOptsWithFromKey/IsOptsWithPrefix.

fixes: #12282

* integration: add test while WithFromKey/WithPrefix called in opts.
2020-10-08 17:34:09 -07:00
Piotr Tabor dfdda47bd8 script/genproto.sh: Refactor to be explicit about versions.
Refactoring script/genproto.sh around state-of-the-art techniques of
managing tooling in go:
  - https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module
  - uses https://github.com/myitcv/gobin instead of customly created gopath.proto dir
  - caches tools between executions
  - guaratees hermetics runs (it was not guaranteed for protoc_grpc_gateway that used latest)

The change is no-op for the generated code.

The commit reveals a few 'worring things':
  1  We depend on : github.com/grpc-ecosystem/grpc-gateway/@v/v1.4.1/protoc-gen-grpc-gateway
  2. And also     : github.com/grpc-ecosystem/grpc-gateway/@v/v1.15.0/protoc-gen-swagger/protoc-gen-swagger
  3. And on extremely old: github.com/gogo/protobuf@v1.0.0 protoc-gen-gofast that is out of sync with the library linked to binaries: github.com/gogo/protobuf@v1.2.1
2020-10-08 19:52:27 +02:00
Jingyi Hu 2c66612e0e
Merge pull request #12339 from ptabor/202009027-module-tests
Modularization: Establish 'testing' module (integration, functional, e2e)
2020-10-08 09:25:57 -07:00
Piotr Tabor f67956cb7a clientv3: Expose clientv3/examples close to the code.
Many of the tests had missing '// Output:' comment, so were not
runnable. They required fining.
2020-10-08 14:27:32 +02:00
Piotr Tabor dd45d04b2d clientv3/concurrency: Expose examples close to the source-code. 2020-10-07 15:46:56 +02:00
Piotr Tabor c5ccebf792 tests/integration: Simplify the testMain for examples.
We introduce a LazyCluster abstraction (instead of copy-pasted logic)
that makes clusters to be created only if there are runnable tests
in need for the infrastructure.
2020-10-07 15:42:35 +02:00
Piotr Tabor 73b92fe688 tests: Make examples (for not client) to be both: documentation and integration-runnable
This CL tries to connect 2 objectives:
  - Examples should be close (the same package) to the original code,
    such that they can participate in documentation.
  - Examples should be runnable - such that they are not getting out of
    sync with underlying API/implementation.

In case of etcd-client, the examples are assuming running 'integration'
style, i.e. thay do connect to fully functional etcd-server.
That would lead to a cyclic dependencies between modules:
  - server depends on client (as client need to be lightweight)
  - client (for test purposes) depend on server.
Go modules does not allow to distingush testing dependency from
prod-code dependency.

Thus to meet the objective:
  - The examples are getting executed within testing/integration packages against real etcd
  - The examples are symlinked to 'unit' tests, such that they included in documentation.
  - Long-term the unit examples should get rewritten to use 'mocks' instead of real integration tests.
2020-10-07 15:38:40 +02:00
Piotr Tabor 313087f8c5 ./test,./scripts: Update go.sum and bill-of-materials generation logic (and refresh bom). 2020-10-07 15:38:39 +02:00
Piotr Tabor 9866c7e8ff .words: Expanded and resorted .words dictionary file.
Should fail following goword complains:
```
clientv3/config.go.48: // ("--max-request-bytes" flag to etcd or "embed.Config.MaxRequestBytes"). (spell: MaxRequestBytes -> ?)
clientv3/config.go.55: // ("--max-request-bytes" flag to etcd or "embed.Config.MaxRequestBytes"). (spell: MaxRequestBytes -> ?)
clientv3/leasing/doc.go.15: // Package leasing serves linearizable reads from a local cache by acquiring (spell: linearizable -> infeasible?)
clientv3/op.go.413: // it's linearizable. Serializable requests are better for lower latency (spell: linearizable -> infeasible?)
clientv3/retry.go.49: // an obvious server-side error (e.g. rpctypes.ErrRequestTooLarge). (spell: ErrRequestTooLarge -> Erectile?)
```
2020-10-07 15:38:39 +02:00
Piotr Tabor 7886ec8574 test: Modules: PASSES="integration" ./test understand new file locations 2020-10-07 15:38:39 +02:00
Piotr Tabor be4e8b7013 integration/fixtures: Move the 'tests/integration/fixtures' directory up and update references.
I moved the files up as they are shared between e2e & integrational tests.
2020-10-07 15:38:28 +02:00
Piotr Tabor bb9703820c tests/integration: Move of naming/snapshot clientv3 tests (and yaml reference)
path change in clientv3/yaml/config_test

git mv clientv3/naming/grpc_test.go         tests/integration/clientv3/grpc_test.go
git mv clientv3/snapshot/*_test.go          tests/integration/snapshot
git mv clientv3/{main_test.go,example_*.go} tests/integration/clientv3/examples/
2020-10-07 15:37:24 +02:00
Piotr Tabor ffd4d5173c tests/mod.go: Update of go.mod after move of integration tests. 2020-10-07 15:37:24 +02:00
Piotr Tabor b27b17b44c tests/functional: Mechanical rename of imports etcd/v3/integration/ -> etcd/tests/v3/integration/... 2020-10-07 15:37:21 +02:00
Piotr Tabor 3153038ffb tests/integration: Moving integration tests to tests/integration directory
git mv integration ./tests

git mv client/integration/* ./tests/integration/client
git mv clientv3/integration/* ./tests/integration/clientv3
git mv client/example_keys_test.go client/main_test.go tests/integration/client/examples
git mv clientv3/concurrency/*_test.go tests/integration/clientv3/concurrency

git mv etcdserver/api/v2store/store_v2v3_test.go etcdserver/api/v2v3/*_test.go tests/integration/v2store
git mv tests/integration/v2store/store_v2v3_test.go tests/integration/v2store/store_v2v3.go
git mv tests/integration/v2store/store_test.go tests/integration/v2store/store_v2v3_test.go
git mv etcdserver/api/v2store/store_test.go tests/integration/v2store
git mv etcdserver/api/v2store/store_v2_test.go tests/integration/v2store

git mv proxy/grpcproxy/*_test.go tests/integration/proxy/grpcproxy

git mv ./clientv3/snapshot/testdata ./clientv3/snapshot/*_test.go ./tests/integration/snapshot
2020-10-07 15:36:21 +02:00
Piotr Tabor b382429d01 tests: Move functional tests to 'tests' module.
Tested with:

PASSES="fmt functional unit" ./test
make build-docker-functional
2020-10-07 15:09:26 +02:00
Piotr Tabor 8907b146d4 tests/functional: Mechanical rename of imports etcd/v3/functional/ -> etcd/tests/v3/functional/...
% find ./ -name "*.go" | xargs sed -i "s|go.etcd.io/etcd/v3/functional/agent|go.etcd.io/etcd/tests/v3/functional/agent|g"
% find ./ -name "*.go" | xargs sed -i "s|go.etcd.io/etcd/v3/functional/runner|go.etcd.io/etcd/tests/v3/functional/runner|g"
% find ./ -name "*.go" | xargs sed -i "s|go.etcd.io/etcd/v3/functional/tester|go.etcd.io/etcd/tests/v3/functional/tester|g"
% find ./ -name "*.go" | xargs sed -i "s|go.etcd.io/etcd/v3/functional/rpcpb|go.etcd.io/etcd/tests/v3/functional/rpcpb|g"
2020-10-07 15:09:12 +02:00
Piotr Tabor 69254d8cf8 functional,tests: git mv functional tests/
Modularization: Mechanical move of "functional" directory into tests module using:

% git mv functional tests/
2020-10-07 15:03:51 +02:00
Piotr Tabor 4fb48b7c14 tests: Estabilishing module for tests
For now it contains only e2e tests, but integration & functional will follow.
2020-10-07 15:03:51 +02:00