Commit Graph

600 Commits (a6ab774458411a6c0ea08f5df97e4dcc9a836345)

Author SHA1 Message Date
Marek Siarkowicz 63346bfead server: Use default logging configuration instead of zap production one
This fixes problem where logs json changes format of timestamp.
2022-04-01 10:23:42 +02:00
Chris Ayoub 125f3c3f9a clientv3: filter learners members during autosync
This change is to ensure that all members returned during the client's
AutoSync are started and are not learners, which are not valid
etcd members to make requests to.
2022-03-29 13:38:21 -04:00
Marek Siarkowicz 0d55a1ca2a
Merge pull request #13821 from ahrtr/configspec_config
Move the newClientCfg into clientv3 package so as to be reused by both etcdctl and v3discovery
2022-03-24 10:12:55 +01:00
ahrtr 49e9a14580 migrate unit test to cover the logic of converting ConfigSpec to Config 2022-03-24 07:24:22 +08:00
ahrtr f978da4f4f move the newClientCfg into clientv3 package so as to be reused by both etcdctl and v3discovery 2022-03-24 06:18:25 +08:00
eval-exec 8d01ac2816 remove endpoints length check in TestValidateSecureEndpoints() 2022-03-23 23:37:31 +08:00
EXEC ad78a74c42 Update client/pkg/transport/tls_test.go
Co-authored-by: Marek Siarkowicz <marek.siarkowicz@protonmail.com>
2022-03-23 23:37:31 +08:00
eval-exec 88e1d6b126 using subtests for TestValidateSecureEndpoints() 2022-03-23 23:37:31 +08:00
EXEC 4786a72cfc Update client/pkg/transport/tls_test.go
Co-authored-by: Marek Siarkowicz <marek.siarkowicz@protonmail.com>
2022-03-23 23:37:31 +08:00
eval-exec df71f59c0e close idle connections 2022-03-23 23:37:31 +08:00
eval-exec 983ee82c98 add test for transport/tls.go:ValidateSecureEndpoints() 2022-03-23 23:37:31 +08:00
EXEC bd7d09255b Fix panic in etcd validate secure endpoints #13810
`ValidateSecureEndpoints()` should call `t.DialContext()` instead of `t.Dial()`, because `t.Dial` is `nil`
2022-03-23 23:37:31 +08:00
ahrtr 1a3822f2c3 Rename ClientConfig to ConfigSpec
The ClientConfig is a fully declarive configuration, so it makes more
sense to rename it to ConfigSpec. It can also mitigate the confusion
between Config and ClientConfig.
2022-03-13 05:41:49 +08:00
ahrtr 3dcbbf62d9 Move clientconfig into clientv3 so that it can be reused by both etcdctl and v3 discovery 2022-03-12 06:38:41 +08:00
cuishuang b7824b794b fix some typos
Signed-off-by: cuishuang <imcusg@gmail.com>

fix some typos

Signed-off-by: cuishuang <imcusg@gmail.com>
2022-03-07 20:28:53 +08:00
Marek Siarkowicz fb55910500 version: bump up to 3.6.0-alpha.0 2022-02-28 13:20:27 +01:00
Marek Siarkowicz 6af760131e
Merge pull request #13687 from serathius/etcdctl
Add downgrade commands
2022-02-22 17:12:23 +01:00
Piotr Tabor f80f477073
Merge pull request #13644 from Juneezee/refactor/t.TempDir
*: use `T.TempDir` to create temporary test directory
2022-02-21 19:52:37 +01:00
Marek Siarkowicz d0c1c3a1fd client: Alias downgrade action enum 2022-02-21 17:58:20 +01:00
ahrtr 8681888012 fix typo, renamed ErrGPRCNotSupportedForLearner to ErrGRPCNotSupportedForLearner 2022-02-21 14:46:58 +08:00
ahrtr 5fc0092c87 trim the suffix dot from the srv.Target for etcd-client DNS lookup 2022-02-18 18:16:19 +08:00
Eng Zer Jun e9edc5432c
*: use `T.TempDir` to create temporary test directory
The directory created by `T.TempDir()` and is automatically removed when
the test and all its subtests complete.

Reference: https://pkg.go.dev/testing#T.TempDir
Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2022-01-25 00:00:46 +08:00
ahrtr 15568f4c00 add protection code for Range when the sortTarget is an invalid value 2022-01-18 07:46:37 +08:00
Piotr Tabor fdd98477ef Update dep: require gopkg.in/yaml.v2 v2.2.8 -> v2.4.0 due to: CVE-2019-11254 2022-01-17 16:52:58 +01:00
Jille Timmermans 3710083dc3 retry_interceptor.go: Log the method that failed
When getting warnings from the retry_interceptors that RPCs failed, they
provide very little context as to what exactly failed.

I used this patch to try debugging
https://groups.google.com/g/etcd-dev/c/3hxxr9ktdpM.
2022-01-16 16:46:50 +01:00
Piotr Tabor b2f6ffdd81
Merge pull request #13595 from serathius/enable-v2
server: --enable-v2 and --enable-v2v3 is decomissioned
2022-01-15 16:48:29 +01:00
Marek Siarkowicz ee5ef42c5c server: --enable-v2 and --enable-v2v3 is decomissioned 2022-01-14 13:19:30 +01:00
Marek Siarkowicz 4032d4f66a Remove CodeQL errors 2022-01-13 14:29:09 +01:00
Sahdev Zala 96a9fd0a1e
Merge pull request #13574 from cunnie/defer_cancel
Golang Client docs: defer `cancel()`, avoid erroring
2022-01-03 20:03:22 -05:00
Yap Sok Ann 17fd2e7282 Disable auth gracefully without impacting existing watchers
This attempts to fix a special case of the problem described in #12385,
where trying to do `clientv3.Watch` with an expired token would result
in `ErrGRPCPermissionDenied`, due to the failing authorization check in
`isWatchPermitted`. Furthermore, the client can't auto recover, since
`shouldRefreshToken` rightly returns false for the permission denied
error.

In this case, we would like to have a runbook to dynamically disable
auth, without causing any disruption. Doing so would immediately expire
all existing tokens, which would then cause the behavior described
above. This means existing watchers would still work for a period of
time after disabling auth, until they have to reconnect, e.g. due to a
rolling restart of server nodes.

This commit adds a client-side fix and a server-side fix, either of
which is sufficient to get the added test case to pass. Note that it is
an e2e test case instead of an integration one, as the reconnect only
happens if the server node is stopped via SIGINT or SIGTERM.

A generic fix for the problem described in #12385 would be better, as
that shall also fix this special case. However, the fix would likely be
a lot more involved, as some untangling of authn/authz is required.
2021-12-31 14:39:46 +07:00
Brian Cunnie 5620a9c227
Golang Client docs: defer cancel(), avoid erroring
In the sample code demonstrating how to specify a client request
timeout, the `cancel()` is called immediately after the Put, but it
should be deferred instead, giving the Put enough time to complete.

In the canonical Golang context
[docs](https://pkg.go.dev/context#WithTimeout), the sample code sets a
`defer cancel()` immediately after context creation, and with this
commit we adhere to that convention.

fixes:
```json
{
  "level": "warn",
  "ts": "2021-12-29T09:56:42.439-0800",
  "logger": "etcd-client",
  "caller": "v3@v3.5.1/retry_interceptor.go:62",
  "msg": "retrying of unary invoker failed",
  "target": "etcd-endpoints://0xc000213340/localhost:2379",
  "attempt": 0,
  "error": "rpc error: code = Canceled desc = context canceled"
}
```
2021-12-29 14:10:36 -08:00
Piotr Tabor 69279532f4
Merge pull request #13540 from songlh-psu/fixing-3
fixing one panic and two goroutine leaks
2021-12-21 11:03:59 +01:00
Linhai Song 0098dbf350 fixing two goroutine leaks and one panic 2021-12-15 22:38:25 -05:00
Linhai d1194977eb fix potential goroutine leaks in TestTxnPanics 2021-12-15 01:22:56 -05:00
Kushal Pisavadia 71493bde3e *: Upgrade to use go.opentelemetry.io/otel@v1.2.0
Upgrading from v1.0.1.

Upgrading related dependencies
------------------------------

The following dependencies also had to be upgraded:

- go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.26.1
  From v0.25.0. This gets rid of a transitive dependency on go.opentelemetry.io/otel@v1.0.1.
- google.golang.org/genproto@v0.0.0-20211118181313-81c1377c94b1
2021-11-24 16:03:33 +00:00
Chao Chen b5e4c2d3c4 client/v2: remove unsafe json-iterator/reflect2 2021-11-09 11:16:40 -08:00
Piotr Tabor 9c28e07588
Merge pull request #13447 from Juneezee/deprecate-ioutil
*: move from io/ioutil to io and os packages
2021-10-29 23:18:36 +02:00
Sam Batschelet 316e62b4e1 *: fixup go 1.17 bump
Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
2021-10-27 14:26:55 -04:00
Eng Zer Jun 2a151c8982
*: move from io/ioutil to io and os packages
The io/ioutil package has been deprecated as of Go 1.16, see
https://golang.org/doc/go1.16#ioutil. This commit replaces the existing
io/ioutil functions with their new definitions in io and os packages.

Signed-off-by: Eng Zer Jun <engzerjun@gmail.com>
2021-10-28 00:05:28 +08:00
Lili Cosic e6e279a14d *: Bump go to go v1.17.2 2021-10-27 13:44:54 +02:00
Piotr Tabor 8aa6ae9bbb
Merge pull request #13361 from vooon/otel-1.0.0
server: update OpenTelemetry to 1.0.1
2021-10-20 14:14:00 +02:00
Vladimir Ermakov 49a8aa7f49
run fix.sh
To fix dependencies.

Signed-off-by: Vladimir Ermakov <vooon341@gmail.com>
2021-10-13 14:08:27 +03:00
Arda Güçlü d7fa8022e6
Merge branch 'main' into support-zap-console-encoding 2021-10-08 11:00:57 +03:00
Aleksandr Razumov 3644c9d67b
client/pkg/fileutil: add missing logger to {Create,Touch}DirAll
Also populate it to every invocation.
2021-10-07 17:53:59 +03:00
Karen Almog d93b7c8cb1 pkg/types: Support Unix sockets in NewURLS
Resolves #12450
This commits adds support to unix/unixs socket URLs, which currently
fail with the message "URL address does not have the form "host:port".
It also replaces the work started in #11747.
2021-10-07 11:59:32 +02:00
Marek Siarkowicz c929a917b6 client: Use first endpoint as http2 authority header 2021-09-29 14:41:35 +02:00
Piotr Tabor ca5bad0e83
Merge pull request #13261 from njhill/sorting
etcdserver,clientv3: server-side ignore sort-ascend-key for range reqs
2021-09-25 17:47:10 +02:00
Arda Güçlü 3c6e09f932 Handle empty log-format gracefully 2021-09-23 08:17:29 +03:00
Arda Güçlü ec252d06c9 Return error when log-format is invalid 2021-09-22 19:45:08 +03:00
Arda Güçlü e647995a38 Add zap encoding configurable
Json encoding is the default zap encoding value and can not be changeable.
This PR enables configuring zap encoding to console via new flag `log-format`.
2021-09-22 15:48:47 +03:00
Tero Saarni 78463237d4 Merge branch 'main' into etcd-client-solaris-build 2021-09-10 11:31:44 +03:00
leoyang.yl ad69fe0f40 fix IsOptsWithFromKey
fix IsOptsWithFromKey

fix IsOptsWithFromKey

fix IsOptsWithFromKey

fix IsOptsWithFromKey

fix IsOptsWithFromKey
2021-09-10 00:44:52 +08:00
Hitoshi Mitake 2a750a8dba *: implement a retry logic for auth old revision in the client 2021-09-05 01:13:52 +09:00
Tero Saarni af626ebfde client/pkg/v3: fixes Solaris build of transport
Add empty implementation for reuse port socket option since Solaris does not
support SO_REUSEPORT.
2021-08-30 11:44:02 +03:00
Ashish Ranjan 6d300fd38b client/v3: refresh the token when ErrUserEmpty is received while retrying
To fix a bug in the retry logic caused when the auth token is cleared after receiving `ErrInvalidAuthToken` from the server and the subsequent call to `getToken` also fails due to some reason (eg. context deadline exceeded).
This leaves the client without a token and the retry will continue to fail with `ErrUserEmpty` unless the token is refreshed.
2021-08-25 11:27:19 +08:00
nickhill 99182f5404 etcdserver,clientv3: server-side ignore sort-ascend-key for range requests
A client-side optimization was made in #6100 to filter ascending key sorts to avoid an unnecessary re-sort since this is the order already returned by the back-end logic.

It seems to me that this really belongs on the server side since it's tied to the server implementation and should apply for any caller of the kv api (for example non-go clients).

Related, the client/v3 syncer depends on this default sorting which isn't explicit in the kv api contract. So I'm proposing the required sort parameters be included explicitly; it will take the fast path either way.
2021-07-30 15:02:15 -07:00
Yi-Shu Tai 77a5072b69 client: call .Endpoints() in dial() in client/v3/client.go instead of accessing cfg.Endpoints directly
0cdd558361/client/v3/client.go (L299) accesses
endpoints without acquiring lock. Fix it to call Endpoints()

Fix #13201
2021-07-27 15:55:22 -07:00
Piotr Tabor 40da254d68
Merge pull request #13146 from tklauser/filelock-x-sys-windows
client/pkg/fileutil: use `golang.org/x/sys/windows` for `FileLockEx`
2021-07-01 10:44:08 +02:00
Piotr Tabor 7271adecc5
Merge pull request #12992 from awly/client-auth-bundle-overwrite
client/v3: do not overwrite authTokenBundle on dial
2021-07-01 10:40:18 +02:00
Marek Siarkowicz 86f68b9374 *: Add missing file licenses and Fix static analysis prevent skipping them in future 2021-06-29 12:52:02 +02:00
Tobias Klauser c3fe63a658 client/pkg/fileutil: use golang.org/x/sys/windows for FileLockEx
Use the FileLockEx wrapper and the corresponding LOCKFILE_* and error
constants from the golang.org/x/sys/windows package rather than
implementing these in the fileutil package.
2021-06-25 14:44:30 +02:00
Andrew Lytvynov ab8e5a4f8e client/v3: do not overwrite authTokenBundle on dial
`Client.dial` can be called multiple times. For example, from regular
`NewClient` and [from the `Maintenance`
wrapper](6d451ab61d/client/v3/maintenance.go (L81-L84)).

This ends up creating two (if `Client.dial` is called twice) independent
`credentials.Bundle` instances:
- the first one is wired into `PerRPCCredentials` callback in gRPC
  client.
- the second one is in `Client.authTokenBundle` and receives the latest
  auth token updates.

When using username/password authentication and the server is configured
to issue JWTs, token rotation logic breaks because of the above
`credentials.Bundle` confusion.
2021-06-23 09:44:27 -07:00
Lili Cosic df696a7e76 go.mod: Bump etcd to 3.5.0 2021-06-22 12:25:39 +02:00
Piotr Tabor 5564476831
Merge pull request #13073 from serathius/snapshot-version
[Version in Snapshot] SnapshotResponse includes local etcd version
2021-06-21 16:56:00 +02:00
Brandon Philips 1a943350b0
client: README: update to new go.mod paths
The paths in the readme and go.dev docs are out of date after v3.5
2021-06-15 19:47:22 -07:00
Marek Siarkowicz e1b1d93548 *: Snapshot returns local etcd version
Co-authored-by: Lili Cosic <cosiclili@gmail.com>
2021-06-14 16:36:50 +02:00
Lili Cosic 16477a8270 Update client_golang dependency to v1.11.0 2021-06-07 14:49:57 +02:00
Gyuho Lee 23e6bf3671
Merge pull request #13083 from lilic/add-downgrade-client
client/v3/maintenance.go: Add Downgrade support to client
2021-06-04 15:31:48 -07:00
Lili Cosic d563c76e92 client/v3/maintenance.go: Add Downgrade support to client 2021-06-04 13:53:09 +02:00
Piotr Tabor 404efd70f7 Upgrade of prometheous deps.
For now we need to depend on unstable: prometheus/client_golang
2021-06-03 16:07:01 +02:00
Piotr Tabor edcfe575cf Update dependencies:
- github.com/coreos/go-systemd/v22 v22.3.2
  - github.com/google/btree v1.0.1
  - github.com/json-iterator/go v1.1.11
  - github.com/mattn/go-runewidth v0.0.13
  - github.com/prometheus/client_golang v1.10.0
  - github.com/prometheus/common v0.26.0
  - github.com/sirupsen/logrus v1.8.1
2021-06-03 13:03:01 +02:00
Piotr Tabor b240625d21 Update bbolt to v1.3.6.
`./scripts/update_dep.sh go.etcd.io/bbolt v1.3.6`
2021-06-03 12:39:53 +02:00
Piotr Tabor 90cea7d4e7 Update zap to v1.17.0. 2021-06-03 12:37:43 +02:00
Piotr Tabor 573e055cd3
Merge pull request #13069 from lilic/log-errors
client, raft: Log unhandled errors
2021-06-02 18:05:58 +02:00
Lili Cosic 3aae7d7823 client/v3/README.md: Remove warning 2021-06-02 16:35:12 +02:00
Lili Cosic 65cdf43e48 client/v3/lease.go: Log unhandled errors 2021-06-02 11:41:26 +02:00
Piotr Tabor f15e0b8237 integration.BeforeTest can be run without leak-detection. 2021-05-27 20:55:11 +02:00
Piotr Tabor bda32db121 Expose clientv3.CreateDefaultZapLoggerConfig
Need raised in
f3f4259d3f.

BTW: Alligned the implementation of grpclogs with what we use in embed
server, so reduced the client code dependencies.
2021-05-25 22:59:08 +02:00
Pavan BG facba6c663 client: Replace nil value testing with assert.Nil
Multiple errors in previous commit where `nil` is not being detected by `assert.Equal`
2021-05-21 14:18:01 +05:30
Pavan BG 25d3c1b068 *: Fixed go.sum
Ran fix.sh script to correct for differences in go.sum, perhaps due to multiple Golang versions
2021-05-21 13:28:43 +05:30
Pavan BG ce5bb69c28 client: Added missing packages
go.etcd.io/etcd/client/pkg/v3/testutil: package github.com/stretchr/testify/assert imported from implicitly required module; to add missing requirements, run:
2021-05-20 21:44:12 +05:30
Pavan BG 319ef4aa42 client: Removed AssertEqual function
Used github.com/stretchr/testify/assert to replace the AssertEqual function definition. Required the use of copyToInterface to copy a string slice to the empty interface slice
2021-05-20 21:33:07 +05:30
Pavan BG d9c5e1f0a2 *: Replace internal testutil AssertEqual function in favour of github.com/stretchr/testify/assert.Equals
To maintain uniformity in testing methodology, assert.Equals has been used everywhere while replacing testutil.AssertEqual

Fixes #13015
2021-05-20 19:55:45 +05:30
Lili Cosic 7957f4461d server/embed/config_test.go: Add unit test for socket options 2021-05-20 10:00:27 +02:00
Lili Cosic 73c530de83 client,server: Expose reuse-port and reuse-address so they can be
configured via a config file.
2021-05-19 18:41:21 +02:00
Piotr Tabor 16d51d8c26 Fix not retryable error codes from: Unavailable -> FailedPrecondition
- ErrGRPCNotCapable("etcdserver: not capable") -> codes.FailedPrecondition  (it will not autofix, it requires new version of server)
 - ErrGPRCNotSupportedForLearner("etcdserver: rpc not supported for learner") -> codes.FailedPrecondition (as long as its learner, the call will not work)
 - ErrGRPCClusterVersionUnavailable("etcdserver: cluster version not found during downgrade") -> codes.FailedPrecondition (backend does not contain the version (old etcd?) so retry will not help)

https://github.com/etcd-io/etcd/runs/2599598633?check_suite_focus=true

```
{"level":"warn","ts":"2021-05-17T09:55:30.246Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000539880/#initially=[unix://localhost:m30]","attempt":0,"error":"rpc error: code = Unavailable desc = etcdserver: rpc not supported for learner"}
{"level":"warn","ts":"2021-05-17T09:55:30.270Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying
of unary invoker
failed","target":"etcd-endpoints://0xc000539880/#initially=[unix://localhost:m30]","attempt":1,"error":"rpc
error: code = Unavailable desc = etcdserver: rpc not supported for
learner"}`
```
2021-05-19 02:08:53 +02:00
Piotr Tabor 3dd577ec08 Simplify client's logger configuration in tests. 2021-05-18 13:03:16 +02:00
Piotr Tabor 8981afb6f5 Fix unit tests logging config. 2021-05-14 22:45:57 +02:00
Piotr Tabor f5c26814ab Loggers to catch the e2e flake. 2021-05-14 22:42:31 +02:00
Gyuho Lee e2d67f2e3b
Merge pull request #12956 from gyuho/rename-to-main
*: rename "master" branch references to "main" in source code
2021-05-13 08:26:33 -07:00
Gyuho Lee a8e1cf0433 client: rename "master" branch references to "main"
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2021-05-12 10:38:50 -07:00
Marek Siarkowicz 1189ee3f3d client: Add logger argument to NewCtxClient 2021-05-12 16:40:55 +02:00
Piotr Tabor 1929aa0a36
Merge pull request #12948 from serathius/logger
client: Allow setting zap logger in config
2021-05-12 13:49:48 +02:00
Marek Siarkowicz 06afe87b34 client: Allow setting zap logger in config 2021-05-12 12:15:20 +02:00
Piotr Tabor 3b24496c2a
Merge pull request #12925 from yangxuanjia/yxjetcd_check_resuming
check resuming len to avoid panic
2021-05-12 09:18:40 +02:00
yangxuanjia 68b1e9f728 check resuming len to avoid panic 2021-05-12 09:51:59 +08:00
Lili Cosic 1a718a958e Add initial Tracing with OpenTelemetry 2021-05-10 10:44:40 +02:00
刘兵 a8a3eea56d
fix doc demo bug
the service parameter not used.
2021-05-05 13:57:10 +08:00
Piotr Tabor 451f65d661
Merge pull request #12908 from ptabor/20210429-client-retry-logging
Clientv3 (retry interceptor) logs should use the configured logger
2021-04-29 19:25:04 +02:00
Piotr Tabor ffea1537d4 ClientV3 tests use integration.NewClient that configures proper logger. 2021-04-29 18:18:34 +02:00
Piotr Tabor bc8d3f6639 Clientv3 (retry) logs should use the configured logger.
clientv3 logs (especially tests) were poluted with unattributed to testing.T log lines:

```
{"level":"warn","ts":"2021-04-29T12:42:11.055+0200","logger":"etcd-client","caller":"v3/retry_interceptor.go:64","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000fafc0/#initially=[unix://localhost:m10]","attempt":0,"error":"rpc error: code = ResourceExhausted desc = etcdserver: mvcc: database space exceeded"}
```

The reasons were 2 fold:
  - Interceptors were copying logger before "WithLogger" could modify it.
  - We were not propagating the loggers in a few testing contexts.
2021-04-29 12:57:09 +02:00
Piotr Tabor f53b70facb Embed: In case KVStoreHash verification fails, close the backend.
In case of failed verification, the server used to keep opened backend
(so the file was locked on OS level).
2021-04-29 11:51:25 +02:00
Piotr Tabor 11249fdee9
Merge pull request #12874 from ptabor/20210417-go1.16.3
Update go for 3.5: 1.15.x -> 1.16.3
2021-04-19 16:51:49 +02:00
Piotr Tabor 3423a949c0 Update go for 3.5: 1.15 -> 1.16.(3).
https://github.com/etcd-io/etcd/issues/12732
2021-04-19 16:50:54 +02:00
Piotr Tabor 06ba0fc5a2
Merge pull request #12846 from pyiyun/fix-snaptmpfile-bug
etcdserver: remove temp files in snap dir when etcdserver starting
2021-04-17 12:58:46 +02:00
Piotr Tabor 17b982382e Fix TestSnapshotV3RestoreMultiMemberAdd flakes (leaks)
- most important: unix's socket transport should not keep idle
connections. For top-level Transport we close them using:
    f3c518025e/server/etcdserver/api/rafthttp/transport.go (L226)
    but currently we don't have access to close them witing the nest (unix) transport. Short idle deadline is good enough.

  - Use dialContext (instead of dial) to make sure context is passed down the stack
  - Make sure Context is cancelled as soon as the operation is done in pipeline
  - nit: use dedicated method to yeld goroutines.

Tested with:
```
d=$(date +"%Y%m%d_%H%M")
(cd tests && go test --timeout=60m ./integration/snapshot -run TestSnapshotV3RestoreMultiMemberAdd -v --count=180 2>&1 | tee log_${d}.log)
```

There were transports & cmux leaked:

```
   leak.go:118: Test appears to have leaked a Transport:
        internal/poll.runtime_pollWait(0x7f6c5c3784c8, 0x72, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc003296298, 0x72, 0x0, 0x18, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc003296280, 0xc0031f60a8, 0x18, 0x18, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/internal/poll/fd_unix.go:166 +0x1d5
        net.(*netFD).Read(0xc003296280, 0xc0031f60a8, 0x18, 0x18, 0x18, 0xc0009056e2, 0x203000)
        	/usr/lib/google-golang/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc000010258, 0xc0031f60a8, 0x18, 0x18, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/net/net.go:183 +0x91
        github.com/soheilhy/cmux.(*bufferedReader).Read(0xc0003d24e0, 0xc0031f60a8, 0x18, 0x18, 0xc0003d24d0, 0xc0009056e2, 0xc000278400)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/buffer.go:53 +0x12d
        github.com/soheilhy/cmux.hasHTTP2Preface(0x1367e20, 0xc0003d24e0, 0x7f6c5c699f40)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/matchers.go:195 +0x8a
        github.com/soheilhy/cmux.matchersToMatchWriters.func1(0x7f6c5c699f40, 0xc000010258, 0x1367e20, 0xc0003d24e0, 0xc000010258)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:128 +0x39
        github.com/soheilhy/cmux.(*cMux).serve(0xc003228690, 0x138c410, 0xc000010258, 0xc00327f740, 0xc0059ba860)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:192 +0x1e7
        created by github.com/soheilhy/cmux.(*cMux).Serve
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:179 +0x191

        internal/poll.runtime_pollWait(0x7f6c5c60f3f0, 0x72, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/runtime/netpoll.go:222 +0x55
        internal/poll.(*pollDesc).wait(0xc000d53018, 0x72, 0x1000, 0x1000, 0xffffffffffffffff)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:87 +0x45
        internal/poll.(*pollDesc).waitRead(...)
        	/usr/lib/google-golang/src/internal/poll/fd_poll_runtime.go:92
        internal/poll.(*FD).Read(0xc000d53000, 0xc000cfd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/internal/poll/fd_unix.go:166 +0x1d5
        net.(*netFD).Read(0xc000d53000, 0xc000cfd000, 0x1000, 0x1000, 0x3, 0x3, 0x1000000000001)
        	/usr/lib/google-golang/src/net/fd_posix.go:55 +0x4f
        net.(*conn).Read(0xc00031a570, 0xc000cfd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
        	/usr/lib/google-golang/src/net/net.go:183 +0x91
        net/http.(*persistConn).Read(0xc00093b320, 0xc000cfd000, 0x1000, 0x1000, 0x577750, 0x60, 0x0)
        	/usr/lib/google-golang/src/net/http/transport.go:1933 +0x77
        bufio.(*Reader).fill(0xc005702fc0)
        	/usr/lib/google-golang/src/bufio/bufio.go:101 +0x108
        bufio.(*Reader).Peek(0xc005702fc0, 0x1, 0xc00077c660, 0xc003b082a0, 0xc000d08de0, 0x5ae586, 0x11dd6c0)
        	/usr/lib/google-golang/src/bufio/bufio.go:139 +0x4f
        net/http.(*persistConn).readLoop(0xc00093b320)
        	/usr/lib/google-golang/src/net/http/transport.go:2094 +0x1a8
        created by net/http.(*Transport).dialConn
        	/usr/lib/google-golang/src/net/http/transport.go:1754 +0xdaa

        net/http.(*persistConn).writeLoop(0xc00093b320)
        	/usr/lib/google-golang/src/net/http/transport.go:2393 +0xf7
        created by net/http.(*Transport).dialConn
        	/usr/lib/google-golang/src/net/http/transport.go:1755 +0xdcf

        sync.runtime_Semacquire(0xc0059ba868)
        	/usr/lib/google-golang/src/runtime/sema.go:56 +0x45
        sync.(*WaitGroup).Wait(0xc0059ba860)
        	/usr/lib/google-golang/src/sync/waitgroup.go:130 +0x65
        github.com/soheilhy/cmux.(*cMux).Serve.func1(0xc003228690, 0xc0059ba860)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:158 +0x56
        github.com/soheilhy/cmux.(*cMux).Serve(0xc003228690, 0x13698c0, 0xc00377a0f0)
        	/home/ptab/private/golang/pkg/mod/github.com/soheilhy/cmux@v0.1.5/cmux.go:173 +0x115
        go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func1(0xc0007cc360, 0x122b75f)
        	/home/ptab/corp/etcd/server/embed/etcd.go:518 +0x2b9
        go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers.func3(0xc00036d080, 0xc0059330a0)
        	/home/ptab/corp/etcd/server/embed/etcd.go:549 +0x182
        created by go.etcd.io/etcd/server/v3/embed.(*Etcd).servePeers
        	/home/ptab/corp/etcd/server/embed/etcd.go:543 +0x73a
--- FAIL: TestSnapshotV3RestoreMultiMemberAdd (17.74s)
```
2021-04-16 20:17:28 +02:00
pyiyun 28a490b09c etcdserver: remove temp files in snap dir when etcdServer starting
When etcd exits abnormally, tmp files will remain in snap dir, so clean up tmp files in snap dir when etcdserver starting.

Fixes #12837
2021-04-16 20:30:04 +08:00
Piotr Tabor eafbc8c57e Update zap logging dependency.
In particular bring up zapgrpc V2 code:
89e382035d
https://pkg.go.dev/google.golang.org/grpc/grpclog#LoggerV2
2021-04-14 12:15:48 +02:00
大可 5db0070150
Update doc.go 2021-04-13 22:22:42 +08:00
大可 d7e971e8d8
client: fix doc typo 2021-04-13 16:33:24 +08:00
Piotr Tabor 3bb7acc8cf Migrate dependencies pkg/foo -> client/pkg/foo 2021-04-07 00:38:47 +02:00
Piotr Tabor 0b81fdf418 Move pkg/* used by client to client/pkg. 2021-04-05 23:47:27 +02:00
Piotr Tabor f290ab2e60 Update dependecies:
github.com/grpc-ecosystem/grpc-gateway v1.14.6 -> grpc-gateway v1.16.0
  golang.org/x/time v0.0.0-20200630173020-3af7569d3a1e->v0.0.0-20210220033141-f8bda1e9f3ba
2021-03-27 20:48:33 +01:00
Piotr Tabor a60676686b Update dep: grpc: 1.32.0 -> 1.36.0. 2021-03-24 22:27:55 +01:00
Piotr Tabor 45fb7b41d6 Update dep: github.com/golang/protobuf v1.3.5 ->  v1.5.1
Thanks to https://go-review.googlesource.com/c/protobuf/+/300869/ its
feasible now.
2021-03-24 22:27:12 +01:00
Piotr Tabor 18321a0c85
Merge pull request #12786 from ptabor/20210318-debug-env
client: Bring back ETCD_CLIENT_DEBUG variable interpretation.
2021-03-18 21:55:51 +01:00
Piotr Tabor 9312d1b077 clientv3: Bring back ETCD_CLIENT_DEBUG variable interpretation.
env ETCD_CLIENT_DEBUG supports log levels (debug, info, warn, error, dpanic, panic, fatal).
Only when set, overrides application-wide grpc logging settings.
2021-03-18 18:43:07 +01:00
wpedrak dac6e37ea1 *: over 20 staticcheck fixes 2021-03-18 15:06:17 +01:00
Piotr Tabor a47c18d30a Fix 2 remaining 'defer AfterTest' calls. 2021-03-13 23:41:29 +01:00
Piotr Tabor 783e26fcdf Fix gogo to 1.3.2 in api/go.mod. 2021-03-11 19:34:34 +01:00
wpedrak 2c2456bf3d *: partial staticcheck fix 2021-03-10 14:13:38 +00:00
Piotr Tabor fb1d48e98e Integration tests: Use BeforeTest(t) instead of defer AfterTest().
Thanks to this change, a single method BeforeTest(t) can handle
before-test logic as well as registration of cleanup code
(t.Cleanup(func)).
2021-03-09 18:19:51 +01:00
Piotr Tabor a46a358577 --experimental-memory-mlock support
The flag protects etcd memory from being swapped out to disk.
This can happen in memory constrained systems where mmaped bbolt
area is natural condidate for swapping out.

This flag should provide better tail latency on the cost of higher RSS
ram usage. If the experiment is successful, the logic should get moved
into bbolt layer, where we can protect specific bbolt instances
(e.g. avoid protecting both during defragmentation).
2021-03-07 12:32:57 +01:00
Emily Selwood 9ebbf5f38b
Metrics example 404s - fix url
The metrics example link points to a file that appears to have moved. This change points it to what I think is the right place.
2021-03-04 08:17:48 +00:00
David Lanouette 6998c5641c client v2: rename error var for revive
The revive tool complained durring the build.  Error variable has been
renamed.

Fixes #12718

Signed-off-by: David Lanouette <David.Lanouette@GMail.com>
2021-03-01 17:09:16 -05:00
David Lanouette 7d02ce2073 client v2: check for empty request from the context
If the simpleHTTPClient.Do is called and the context has a nil request, return an error early.

Fixes #12718

Signed-off-by: David Lanouette <David.Lanouette@GMail.com>
2021-03-01 16:46:30 -05:00
Piotr Tabor 54b87505a3 Remove dead legacy logger code. 2021-02-26 09:13:09 +01:00
Piotr Tabor 45b1e6b470 ClientV3: Ordering: Fix the ordering test such it does not fail.
The test depended on very subtle timing semantic and on properties of
'copied' clients.

https://travis-ci.com/github/etcd-io/etcd/jobs/486191449

Examplar failure:
```
{"level":"warn","ts":"2021-02-25T12:34:47.894Z","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0000d6fc0/#initially=[unix://localhost:86269902489114839060]","attempt":1,"error":"rpc error: code = Unavailable desc = etcdserver: rpc not supported for learner"}
{"level":"warn","ts":"2021-02-25T12:34:48.163Z","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc00035a000/#initially=[unix://localhost:78285857058450835940]","attempt":0,"error":"rpc error: code = FailedPrecondition desc = etcdserver: not leader"}
{"level":"info","ts":"2021-02-25T12:34:48.255Z","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"warn","ts":"2021-02-25T12:34:48.255Z","caller":"v3/maintenance.go:221","msg":"failed to receive from snapshot stream; closing","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"info","ts":"2021-02-25T12:34:48.255Z","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2021-02-25T12:34:50.255Z","caller":"v3/maintenance.go:219","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2021-02-25T12:34:51.717Z","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"warn","ts":"2021-02-25T12:34:52.017Z","caller":"v3/maintenance.go:221","msg":"failed to receive from snapshot stream; closing","error":"rpc error: code = Canceled desc = context canceled"}
{"level":"info","ts":"2021-02-25T12:34:52.018Z","caller":"v3/maintenance.go:211","msg":"opened snapshot stream; downloading"}
{"level":"warn","ts":"2021-02-25T12:34:53.018Z","caller":"v3/maintenance.go:221","msg":"failed to receive from snapshot stream; closing","error":"rpc error: code = DeadlineExceeded desc = context deadline exceeded"}
--- FAIL: TestEndpointSwitchResolvesViolation (10.12s)
    ordering_util_test.go:81: failed to resolve order violation etcdclient: no cluster members have a revision higher than the previously received revision
```
2021-02-25 22:15:13 +01:00
Piotr Tabor 60d5159091 version: bump up to 3.5.0-alpha.0 2021-02-24 19:55:45 +00:00
Piotr Tabor 1a9c81abda Update grpc dependency to 1.32.
Simplify grpc testing infrastructure to align with upstream changes.
2021-02-23 11:31:50 +01:00
Piotr Tabor 4a1c24556c clientv3: PS: Replace balancer with upstream grpc solution
Addresses comments from: https://github.com/etcd-io/etcd/pull/12671#pullrequestreview-593942302
2021-02-23 10:03:15 +01:00
Piotr Tabor b67ed4e4aa
Merge pull request #12671 from ptabor/20210207-grpc-del-balancer
clientv3: Replace balancer with upstream grpc solution
2021-02-17 22:45:13 +01:00
Piotr Tabor a836a8045b Get rid of legacy client/v3/naming API.
Update grpcproxy to use the new abstractions.
2021-02-09 11:56:28 +01:00
Piotr Tabor 0b75fede64 Replace client/v3/balancer with standard components: resolver + round_robin LB
This commit significantly reduces volume of custom code
in etcd client v3, while preserving full existing functionality.
2021-02-08 18:50:31 +01:00
limeng01 8feb55f65c *: implement Endpoint Watch and new Resolver 2021-02-08 20:05:45 +08:00
Piotr Tabor dae29bb719
Merge pull request #12667 from lemonlinger/implement-endpoint-manager-and-resolver
Implement Update method for EndpointManager
2021-02-04 17:25:51 +01:00
limeng01 571ed502d4 endpoints: implement Update method for EndpointManager.
- Add integration test for endpoints and resolver.
2021-02-04 23:30:22 +08:00
Sahdev Zala 66777cf1e8 Doc: fix inconsistency and broken path
1) Fix the path for rpc errors
2) The README says 2 types of client errors vs doc mentioning 3
types of client errors. grpc erros should be considered single time
as covered in the README.
2021-02-03 12:02:49 -05:00
Danny Hermes ea34f8dbc6 Prepare `balancer` interfaces for `>=google.golang.org/grpc@1.30.0` upgrade. 2021-02-02 15:54:33 -06:00
Piotr Tabor 5d7c1db3a9 Introduce grpc-1.30+ compatible client/v3/naming API.
This is not yet implementation, just API and tests to be filled
with implementation in next CLs,
tracked by: https://github.com/etcd-io/etcd/issues/12652

We propose here 3 packages:
 - clientv3/naming/endpoints ->
    That is abstraction layer over etcd that allows to write, read &
    watch Endpoints information. It's independent from GRPC API. It hides
    the storage details.

 - clientv3/naming/endpoints/internal ->
    That contains the grpc's compatible Update class to preserve the
    internal JSON mashalling format.

 - clientv3/naming/resolver ->
   That implements the GRPC resolver API, such that etcd can be
   used for connection.Dial in grpc.

Please see the grpc_naming.md document changes & grpcproxy/cluster.go
new integration, to see how the new abstractions work.
2021-01-30 12:32:19 +01:00
Piotr Tabor 92458228e1
Merge pull request #12423 from Murphylu1993/master
client: replace dial with dialContext
2021-01-25 20:28:40 +01:00
Piotr Tabor 84b5b87fb2 Make test.sh scripts OSX/BSD compatible:
- build & test scripts deprecated. Call *.sh variants.
  This will avoid delete the symlinks and get rid of
  subtle dependency on 'sed --follow-symlinks' on OsX/BSD sed.

- Fix parameters to mktemp
2021-01-12 16:29:00 +01:00
Sahdev Zala c632042bb9
Merge pull request #12489 from zhijianli88/cleanup-tmpfiles
Cleanup tmpfiles
2021-01-07 12:26:04 -05:00
Sahdev Zala a4570a60e7
Merge pull request #12549 from bbiao/bugfix/refresh_token
client/v3: clear auth token when encounter ErrInvalidAuthToken
2020-12-29 22:08:14 -05:00
bbiao af4ef4ec04
client/v3: clear auth token when encounter ErrInvalidAuthToken
Old etcdserver which have not apply pr of #12165 will check auth token
even if the request is an Authenticate request.
If the client has a invalid auth token, it will not able to update it's
token, since the Authenticate has a invalid auth token.
This fix clear the auth token when encounter an ErrInvalidAuthToken to
talk with old version etcd servers.

Fix #12385 with #12165 and #12264
2020-12-27 11:08:22 +08:00
ianwoolf fc5b5fa31e fix godoc link in clientv3 readme 2020-12-23 10:59:13 +08:00
琚致远 bd97953de3
Update README.md 2020-12-21 17:40:40 +08:00
琚致远 549bff5980
docs: update client v3's README 2020-12-20 00:06:10 +08:00
Piotr Tabor 4d173231d1 Documentation: Developer's docs about etcd-3.5.x modules 2020-12-02 10:00:09 +01:00
Li Zhijian fb3f91c141 test: cleanup temporary file for TestConfigFromFile
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
2020-11-25 17:40:43 +08:00
Piotr Tabor eeafcef0d2 Use "v3.5.0-pre" to reference within-etcd modules
instead of v3.0.0-000101010000000-00000000000,
that might be misleading as we don't develop etcd v3.0.0 any longer.

This version is a virtual version and is not supposed to be tagged
within the repository. We should tag real versions like: 3.5.0-alpha.0.

Please notice that go.etcd.io/etcd/client/v2 will be versioned as `v2.305.0-pre`.
The reason is that client v2 must have v2 version. I propose a
convention to envode the major version as 100x in minor version to make
the association to the underlying repository clear, staying within v2
version family.

The change was generated using:
```
DRY_RUN=false TARGET_VERSION="v3.5.0-pre" ./scripts/release_mod.sh update_versions
```
2020-11-04 18:28:43 +01:00
meadlai ec37e15caf
Update README.md 2020-10-28 14:34:03 +08:00
luqi ed81d2e2db client: replace dial with dialContext 2020-10-27 12:43:14 +08:00
Piotr Tabor 45b007b8b4 contrib,clientv3: Move contrib/recipies to clientv3/experimental/recipies/...
Recipies is set of patterns / primitives implementation on top of clientv3.
It's used by integration tests. It shouldn't be considered "server" code.
2020-10-22 11:10:07 +02:00
Piotr Tabor e33c6dd9df client/v3: Rename of imports 2020-10-20 10:13:06 +02:00
Piotr Tabor cf795c0ae6 client/v3: Create and update go.mod 2020-10-20 10:09:12 +02:00
Piotr Tabor 8a7f15511c clientv3: Move to client/v3 (just file move)
Mechanical:
% git mv clientv3 client/v3
% git mv client/mock client/v3/
2020-10-20 10:08:35 +02:00
Piotr Tabor b7f0f52a16
*: Refresh of dependencies (#12399)
This PR focuses on dependencies that have new stable versions.
2020-10-15 15:32:00 -07:00
Piotr Tabor 45e4306508 client: Move client to client/v2 as a module.
We make v2 client code a module go.etcd.io/etcd/client/v2.

Pretty mechanical change that can be summarized as:

  mkdir client/v2
  cd client/v2 && git mod init go.etcd.io/etcd/client/v2

  git mv client/*.go client/v2/
  find -name '*.go' | xargs sed -i --follow-symlinks 's|/v3/client["]|/client/v2\"|g'

  + fixing changelog, bom, go.mod etc.
2020-10-15 14:39:43 +02:00
Piotr Tabor de55bb6331 pkg: Rename imports after making 'pkg' a module
find -name '*.go' | xargs sed --follow-symlinks -i 's|go.etcd.io/etcd/v3/pkg/|go.etcd.io/etcd/pkg/v3/|g'
go fmt ./...
2020-10-13 00:09:27 +02:00
Piotr Tabor 28036db6f0 Mechanical: Move mock packages out of ./pkg
Mechanical execution of:

```
git mv ./pkg/mock/mockserver ./client/pkg/mock
git mv ./pkg/mock/{mockstorage,mockstore,mockwait} ./server/pkg/mock
```

The packages depend on etcd API / protos - so they are NOT etcd-dependencies.
In such situation thay should be placed in 'pkg' folder.
2020-10-12 23:58:09 +02:00
Piotr Tabor c5ccebf792 tests/integration: Simplify the testMain for examples.
We introduce a LazyCluster abstraction (instead of copy-pasted logic)
that makes clusters to be created only if there are runnable tests
in need for the infrastructure.
2020-10-07 15:42:35 +02:00
Piotr Tabor 73b92fe688 tests: Make examples (for not client) to be both: documentation and integration-runnable
This CL tries to connect 2 objectives:
  - Examples should be close (the same package) to the original code,
    such that they can participate in documentation.
  - Examples should be runnable - such that they are not getting out of
    sync with underlying API/implementation.

In case of etcd-client, the examples are assuming running 'integration'
style, i.e. thay do connect to fully functional etcd-server.
That would lead to a cyclic dependencies between modules:
  - server depends on client (as client need to be lightweight)
  - client (for test purposes) depend on server.
Go modules does not allow to distingush testing dependency from
prod-code dependency.

Thus to meet the objective:
  - The examples are getting executed within testing/integration packages against real etcd
  - The examples are symlinked to 'unit' tests, such that they included in documentation.
  - Long-term the unit examples should get rewritten to use 'mocks' instead of real integration tests.
2020-10-07 15:38:40 +02:00
Piotr Tabor 3153038ffb tests/integration: Moving integration tests to tests/integration directory
git mv integration ./tests

git mv client/integration/* ./tests/integration/client
git mv clientv3/integration/* ./tests/integration/clientv3
git mv client/example_keys_test.go client/main_test.go tests/integration/client/examples
git mv clientv3/concurrency/*_test.go tests/integration/clientv3/concurrency

git mv etcdserver/api/v2store/store_v2v3_test.go etcdserver/api/v2v3/*_test.go tests/integration/v2store
git mv tests/integration/v2store/store_v2v3_test.go tests/integration/v2store/store_v2v3.go
git mv tests/integration/v2store/store_test.go tests/integration/v2store/store_v2v3_test.go
git mv etcdserver/api/v2store/store_test.go tests/integration/v2store
git mv etcdserver/api/v2store/store_v2_test.go tests/integration/v2store

git mv proxy/grpcproxy/*_test.go tests/integration/proxy/grpcproxy

git mv ./clientv3/snapshot/testdata ./clientv3/snapshot/*_test.go ./tests/integration/snapshot
2020-10-07 15:36:21 +02:00
Piotr Tabor 28f2b07623 *: Update references to code moved to the api/ dir.
Follow up to file-moves done in the previous commit.

The commit contains purely mechanical consequences of execution (apart
of scripts/genproto.sh):

  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/etcdserver/api/v3rpc/rpctypes|v3/api/v3rpc/rpctypes|g'
  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/version|v3/api/version|g'
  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/mvcc/mvccpb|v3/api/mvccpb|g'
  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/etcdserver/etcdserverpb|v3/api/etcdserverpb|g'
  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/etcdserver/api/membership/membershippb|v3/api/membershippb|g'
  % find ./ -name '*.go'  | xargs sed --follow-symlinks -i 's|v3/auth/authpb|v3/api/authpb|g'

  % find ./ -name '*.proto' -o -name '*.md'  | xargs -L 1 sed --follow-symlinks -i 's|/mvcc/mvccpb/kv.proto|/api/mvccpb/kv.proto|g'
  % find ./ -name '*.proto' -o -name '*.md'  | xargs -L 1 sed --follow-symlinks -i 's|/auth/authpb/auth.proto|/api/authpb/auth.proto|g'
  % find ./ -name '*.proto' -o -name '*.md'  | xargs -L 1 sed --follow-symlinks -i 's|/etcdserver/api/membership/membershippb/membership.proto|/api/membershippb/membership.proto|g'

  I also modified manually paths in scripts/genproto.sh.

  % go fmt ./...
2020-10-06 11:56:16 +02:00
Piotr Tabor 528f5315d6 auth: Fix "race" - auth unit tests leaking goroutines
- We were leaking goroutines in auth-test
  - The go-routines were depending / modifying global test environment
variables (simpleTokenTTLDefault) leading to races

Removed the leaked go-routines, and expanded 'auth' package to
be covered we leaked go-routines detection.
2020-10-03 19:38:30 +02:00
Piotr Tabor fb2e96e651 vendor: Get rid of ./vendor cleanup
Updated scripts and documentation to not recommend vendoring.
Implemented best practices for tools installation.

Performed multiple tests to confirm its not breaking any workflows and
has no negative performance impact. Rather see 3x speedup.

1. PASSES="fmt unit integration e2e functional" ./test
2. ./scripts/updatebom.sh
3. ./scripts/updatedep.sh
4. ./scripts/genproto.sh - works - ca be simplified - in follow up PR
5. Installation without explicit GOPATH:

```
% unset GOPATH
% [sudo] rm -rf ~/go
% git clone https://github.com/etcd-io/etcd.git
% time ./build
go: downloading google.golang.org/grpc v1.26.0
go: downloading github.com/jonboulle/clockwork v0.1.0
go: downloading github.com/prometheus/client_golang v1.0.0
go: downloading github.com/soheilhy/cmux v0.1.4
go: downloading github.com/gogo/protobuf v1.2.1
go: downloading sigs.k8s.io/yaml v1.1.0
go: downloading golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2
go: downloading github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903
go: downloading go.etcd.io/bbolt v1.3.5
go: downloading go.uber.org/zap v1.15.0
go: downloading golang.org/x/crypto v0.0.0-20191002192127-34f69633bfdc
go: downloading github.com/golang/protobuf v1.3.2
go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8
go: downloading github.com/beorn7/perks v1.0.0
go: downloading github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4
go: downloading github.com/coreos/go-systemd/v22 v22.0.0
go: downloading gopkg.in/yaml.v2 v2.2.2
go: downloading github.com/coreos/go-semver v0.2.0
go: downloading github.com/sirupsen/logrus v1.4.2
go: downloading golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7
go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: downloading github.com/google/uuid v1.0.0
go: downloading github.com/modern-go/reflect2 v1.0.1
go: downloading github.com/prometheus/common v0.4.1
go: downloading github.com/spf13/cobra v0.0.3
go: downloading github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
go: downloading github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c
go: downloading github.com/spf13/pflag v1.0.1
go: downloading github.com/json-iterator/go v1.1.7
go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: downloading github.com/google/btree v1.0.0
go: downloading go.uber.org/atomic v1.6.0
go: downloading github.com/prometheus/procfs v0.0.2
go: downloading go.uber.org/multierr v1.5.0
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.9.5
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4
go: downloading github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4
go: downloading google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading golang.org/x/text v0.3.3
go: downloading github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5
go: downloading github.com/bgentry/speakeasy v0.1.0
go: downloading gopkg.in/cheggaaa/pb.v1 v1.0.25
go: downloading github.com/urfave/cli v1.20.0
go: downloading github.com/mattn/go-runewidth v0.0.2
./build  8.22s user 2.31s system 117% cpu 8.961 total
```

Before:
```
% git clone https://github.com/etcd-io/etcd.git && cd etcd && time ./build
Cloning into 'etcd'...
remote: Enumerating objects: 97872, done.
remote: Total 97872 (delta 0), reused 0 (delta 0), pack-reused 97872
Receiving objects: 100% (97872/97872), 58.97 MiB | 19.85 MiB/s, done.
Resolving deltas: 100% (63091/63091), done.

./build  34.97s user 4.15s system 236% cpu 16.555 total
```

6. Rebuild without changes:

```
% time ./build
./build  1.43s user 0.83s system 168% cpu 1.336 total
```

7. Instantation of vendor directory (assuming ./build loaded them to
$GOPATH/pkg):

```
time go mod vendor
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa
go: downloading github.com/creack/pty v1.1.11
go: downloading github.com/etcd-io/gofail v0.0.0-20190801230047-ad7f989257ca
go: downloading github.com/konsorten/go-windows-terminal-sequences v1.0.1
go mod vendor  0.51s user 0.44s system 110% cpu 0.861 total
```

8. Fresh instantation of vendor:

```
% rm -rf vendor
% [sudo] rm -rf ~/go

% time go mod vendor
go: downloading github.com/coreos/go-systemd/v22 v22.0.0
go: downloading github.com/spf13/cobra v0.0.3
go: downloading github.com/prometheus/client_golang v1.0.0
go: downloading golang.org/x/net v0.0.0-20190813141303-74dc4d7220e7
go: downloading github.com/dustin/go-humanize v0.0.0-20171111073723-bb3d318650d4
go: downloading github.com/gogo/protobuf v1.2.1
go: downloading sigs.k8s.io/yaml v1.1.0
go: downloading google.golang.org/grpc v1.26.0
go: downloading github.com/urfave/cli v1.20.0
go: downloading go.uber.org/zap v1.15.0
go: downloading github.com/spf13/pflag v1.0.1
go: downloading github.com/soheilhy/cmux v0.1.4
go: downloading github.com/json-iterator/go v1.1.7
go: downloading github.com/coreos/go-semver v0.2.0
go: downloading github.com/prometheus/common v0.4.1
go: downloading github.com/prometheus/procfs v0.0.2
go: downloading go.uber.org/atomic v1.6.0
go: downloading github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5
go: downloading github.com/golang/protobuf v1.3.2
go: downloading github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa
go: downloading github.com/grpc-ecosystem/go-grpc-middleware v1.0.1-0.20190118093823-f849b5445de4
go: downloading github.com/modern-go/reflect2 v1.0.1
go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd
go: downloading go.uber.org/multierr v1.5.0
go: downloading github.com/creack/pty v1.1.11
go: downloading github.com/mattn/go-runewidth v0.0.2
go: downloading github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0
go: downloading golang.org/x/crypto v0.0.0-20191002192127-34f69633bfdc
go: downloading golang.org/x/sys v0.0.0-20200202164722-d101bd2416d5
go: downloading github.com/jonboulle/clockwork v0.1.0
go: downloading gopkg.in/yaml.v2 v2.2.2
go: downloading github.com/etcd-io/gofail v0.0.0-20190801230047-ad7f989257ca
go: downloading github.com/grpc-ecosystem/grpc-gateway v1.9.5
go: downloading github.com/google/btree v1.0.0
go: downloading google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55
go: downloading github.com/beorn7/perks v1.0.0
go: downloading github.com/dgrijalva/jwt-go v3.2.0+incompatible
go: downloading github.com/google/uuid v1.0.0
go: downloading golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2
go: downloading github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8
go: downloading go.etcd.io/bbolt v1.3.5
go: downloading golang.org/x/text v0.3.3
go: downloading gopkg.in/cheggaaa/pb.v1 v1.0.25
go: downloading github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2
go: downloading github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4
go: downloading github.com/inconshreveable/mousetrap v1.0.0
go: downloading github.com/gorilla/websocket v0.0.0-20170926233335-4201258b820c
go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.1
go: downloading github.com/bgentry/speakeasy v0.1.0
go: downloading github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903
go: downloading github.com/sirupsen/logrus v1.4.2
go: downloading github.com/konsorten/go-windows-terminal-sequences v1.0.1
go mod vendor  3.62s user 1.30s system 127% cpu 3.854 total
```

9. Size of the repository - before: 39M, after: 18M

Before:

```
% time git clone https://github.com/etcd-io/etcd.git
Cloning into 'etcd'...
remote: Enumerating objects: 97872, done.
remote: Total 97872 (delta 0), reused 0 (delta 0), pack-reused 97872
Receiving objects: 100% (97872/97872), 58.97 MiB | 20.53 MiB/s, done.
Resolving deltas: 100% (63091/63091), done.
git clone https://github.com/etcd-io/etcd.git  4.66s user 1.02s system 93% cpu 6.068 total

% du -h --exclude .git -d 1
944K	./clientv3
108K	./etcdmain
5.4M	./Documentation
384K	./security
384K	./mvcc
28K	./.github
8.0K	./version
144K	./contrib
240K	./proxy
2.5M	./etcdserver
112K	./embed
536K	./integration
332K	./tools
116K	./lease
108K	./logos
896K	./tests
960K	./raft
216K	./client
52K	./scripts
100K	./hack
464K	./etcdctl
3.0M	./pkg
620K	./functional
136K	./wal
152K	./auth
21M	./vendor
39M
```

After:
```
% time git clone https://github.com/ptabor/etcd.git -b 20200908-no-vendor
Cloning into 'etcd'...
remote: Enumerating objects: 38, done.
remote: Counting objects: 100% (38/38), done.
remote: Compressing objects: 100% (37/37), done.
remote: Total 98489 (delta 10), reused 8 (delta 1), pack-reused 98451
Receiving objects: 100% (98489/98489), 59.23 MiB | 21.26 MiB/s, done.
Resolving deltas: 100% (63572/63572), done.
git clone https://github.com/ptabor/etcd.git -b 20200908-no-vendor  5.56s user 1.05s system 105% cpu 6.260 total

% du -h --exclude .git -d 1
944K	./clientv3
108K	./etcdmain
5.4M	./Documentation
384K	./security
384K	./mvcc
28K	./.github
8.0K	./version
144K	./contrib
240K	./proxy
2.5M	./etcdserver
112K	./embed
536K	./integration
332K	./tools
116K	./lease
108K	./logos
896K	./tests
960K	./raft
216K	./client
56K	./scripts
100K	./hack
464K	./etcdctl
3.0M	./pkg
620K	./functional
136K	./wal
152K	./auth
19M	.
```
2020-09-15 14:38:54 +02:00
Brandon Philips d88d765ba4 Documentation, CHANGELOG: use new go.etcd.io/etcd/v3 pkg
Use the new package path in the docs and announce it in the CHANGELOG
2020-04-28 22:02:19 +00:00
Brandon Philips 96cce208c2 go.mod: use go.etcd.io/etcd/v3 versioning
This change makes the etcd package compatible with the existing Go
ecosystem for module versioning.

Used this tool to update package imports:
  https://github.com/KSubedi/gomove
2020-04-28 00:57:35 +00:00
sfzhu93 467e08c32a
in multiple packages: fixed goroutine leak bugs in tests (cont.d) (#11570) 2020-01-30 10:55:59 -08:00
Gyuho Lee 1caaa9ed4a test: test update for Go 1.12.5 and related changes
Update to Go 1.12.5 testing. Remove deprecated unused and gosimple
pacakges, and mask staticcheck 1006. Also, fix unconvert errors related
to unnecessary type conversions and following staticcheck errors:
- remove redundant return statements
- use for range instead of for select
- use time.Since instead of time.Now().Sub
- omit comparison to bool constant
- replace T.Fatal and T.Fatalf in tests with T.Error and T.Fatalf respectively because the goroutine calls T.Fatal must be called in the same goroutine as the test
- fix error strings that should not be capitalized
- use sort.Strings(...) instead of sort.Sort(sort.StringSlice(...))
- use he status code of Canceled instead of grpc.ErrClientConnClosing which is deprecated
- use use status.Errorf instead of grpc.Errorf which is deprecated

Related #10528 #10438
2019-06-05 17:02:05 -04:00
Gyuho Lee 34bd797e67 *: revert module import paths
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2019-05-28 15:39:35 -07:00
Xiang Li caee28a88e
Merge pull request #10666 from mkumatag/fix_tests
Fix tests for latest golang
2019-05-03 11:18:58 -07:00
shivaramr 9150bf52d6 go modules: Fix module path version to include version number 2019-04-26 15:29:50 -07:00
Davanum Srinivas daee668b75
client: Switch to case sensitive unmarshalling to be compatible with ugorji
Using lessons learned from k8s changes:
https://github.com/kubernetes/kubernetes/pull/65034

Change-Id: Ia17a8f94ae6ed00c5af2595c2b48d3c9a0344427
2019-04-23 16:54:44 -04:00
Davanum Srinivas 86e3481ba2
scripts: Remove generated code and script
Change-Id: Iac4601443bcad71920fd96b97bfe21c16116577a
2019-04-23 16:54:44 -04:00
Davanum Srinivas 90108a2e61
client: Replace ugorji/codec with json-iterator/go
We need to use the stdlib-compatible one that is case-sensitive, etc

Change-Id: Id0df573a70e09967ac7d8c0a63d99d6a49ce82f1
2019-04-23 16:54:44 -04:00
Manjunath A Kumatagi 867b45d865 client: Fix tests for latest golang 2019-04-22 08:14:10 -05:00
zhoulin xie a943ad0ee4 client/keys_bench_test.go: Fix some misspells
Signed-off-by: zhoulin xie <zhoulin.xie@daocloud.io>
2019-02-28 14:36:06 -05:00
Sophos 6543273666 client: generate new keys and remove yynn2 = 0 2019-02-25 12:07:40 +08:00
Sam Batschelet f3fbedc88f client: update generated ugorji codec, manual remove "yynn2=0"
Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
2018-12-21 18:16:55 -05:00
Sam Batschelet fa35126ef8
*: add client support for discovery-srv-name
Signed-off-by: Sam Batschelet <sbatsche@redhat.com>
2018-11-09 10:13:04 -05:00
vcaesar fc7ef659cc client,clientv3: update client docs to "go.etcd.io" 2018-08-30 19:26:12 -04:00
Gyuho Lee 0ef9ef3c74 *: rerun "gofmt"
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2018-08-28 18:25:39 -07:00
Gyuho Lee d37f1521b7 *: update import paths to "go.etcd.io/etcd"
Signed-off-by: Gyuho Lee <leegyuho@amazon.com>
2018-08-28 17:47:55 -07:00
Gyuho Lee 42bef8460c *: clean up code format
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-07-21 16:34:56 -07:00
Gyuho Lee 91664ec5ac client: manually delete "yynn2 = 0"
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-09 13:42:50 -07:00
Gyuho Lee 13c440dea4 client: regenerate with f3cacc17c85ecb7f1b6a9e373ee85d1480919868
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-09 13:42:50 -07:00
Gyuho Lee ebba9ed29e client: regenerate
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-04-09 13:42:50 -07:00
Gyuho Lee 0850ccbf45 *: revert "internal/version" change
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-02-26 17:11:40 -08:00
Gyuho Lee ebfba72715 client: fix "ineffassign" warning
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-02-26 10:07:18 -08:00
Gyuho Lee 37546f74ab *: move "version" to "internal/version"
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-01-29 10:00:20 -08:00
Gyuho Lee 97cd2a41c8 client,clientv3: remove "cmd/vendor" in README.md
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
2018-01-24 15:26:17 -08:00
Gyu-Ho Lee 75110dd839 *: fix naked returns
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-11-10 18:46:15 -08:00
Gyu-Ho Lee 81ca10f991 client/keys.generated.go: remove ineffassign yynn2 = 0
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-10-20 08:52:48 -07:00
Gyu-Ho Lee 1b2a62d9d0 client/keys.generated.go: remove redundant and: x.Expiration != nil
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-10-20 08:47:01 -07:00
Andrew Meissner 12a6efb74b update: client
Updating the codec required codecgen to be reran on the client/keys.go file.  This is the result of that run.
2017-10-20 09:23:23 -06:00
Gyu-Ho Lee 0199bdc266 *: fix 'ineffassign' issues
Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-10-03 10:14:33 -07:00
Gyu-Ho Lee fbc7acde95 client: permute endpoints manually (for Go 1.9>)
To keep backward compatibility, use old algorithm of
rand.Rand.Perm.

Reference: caae0917bf (diff-d4a72c5ba8515eae95a093e0aec62635).

Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
2017-09-22 10:19:30 +09:00
Alexandre Beslic 8dc4833a3e
client: regenerate sources for etcd/client with new codec version
Major updates to ugorji/go changed the signature of some
methods, resulting in the build failing for etcd/client
with default installation of the codec.

We regenerate the sources using codecgen with the new version
to reflect on the new changes.

Fixes #8573

Signed-off-by: Alexandre Beslic <abeslic@abronan.com>
2017-09-19 15:14:58 +02:00