It specifies request timeout error possibly caused by connection lost,
and print out better log for user to understand.
It handles two cases:
1. the leader cannot connect to majority of cluster.
2. the connection between follower and leader is down for a while,
and it losts proposals.
log format:
```
20:04:19 etcd3 | 2015-08-25 20:04:19.368126 E | etcdhttp: etcdserver:
request timed out, possibly due to connection lost
20:04:19 etcd3 | 2015-08-25 20:04:19.368227 E | etcdhttp: etcdserver:
request timed out, possibly due to connection lost
```
The original workflow may fail to cancel if stop() cancels the finished
request just before dial() assigning a new cancel. This commit checks
streamReader status before setting cancel to avoid this problem.
It is tested at travis for 300 times. go 1.5 always works well, while
go 1.4 fails to stop once.
This problem is totally fixed at 1.5.
go1.5 adds a Request.Cancel channel, which allows for "race free"
cancellation
(8b4278ffb7).
Our implementation relies on it to always cancel in-flight request.
The original test code before fb4b0b5cf0
doesn't work because reader side may update the
stream, while writer side writes message to the old stream and fails.
This PR removes unnecessary call to set term, and avoids this problem to
happen on term > 1 in the future.
Update term when AddPeer, or the term in peer will not be updated until
the term is changed. This fixes the log flood happended when the v2.1
follower applies the snapshot from v2.0 leader:
```
rafthttp: cannot attach out of data stream server [0 / 17]
```
or
```
rafthttp: server streaming to 6e3bd23ae5f1eae0 at term 0 has been
stopped
```
So it ensures that the message will not be discarded because the receive
side has not been ready, which happens easily in multiple core test.
Use log.fatal instead of log.error. The test exits when there is
something wrong because the error may affect following test cases.
before:
```
2015/06/12 20:06:19 rafthttp: dropped MsgApp from %!s(uint64=2) since
receiving buffer is full
```
after:
```
2015/06/12 13:51:38 rafthttp: dropped MsgProp from 2 since receiving
buffer is full
```
Refactor the fake cancel implementation.
The old one may cancel other in-flight message in random, which leaves
the original target message blocked forever.
ForceGosched() performs bad when GOMAXPROCS>1. When GOMAXPROCS=1, it
could promise that other goroutines run long enough
because it always yield the processor to other goroutines. But it cannot
yield processor to goroutine running on other processors. So when
GOMAXPROCS>1, the yield may finish when goroutine on the other
processor just runs for little time.
Here is a test to confirm the case:
```
package main
import (
"fmt"
"runtime"
"testing"
)
func ForceGosched() {
// possibility enough to sched up to 10 go routines.
for i := 0; i < 10000; i++ {
runtime.Gosched()
}
}
var d int
func loop(c chan struct{}) {
for {
select {
case <-c:
for i := 0; i < 1000; i++ {
fmt.Sprintf("come to time %d", i)
}
d++
}
}
}
func TestLoop(t *testing.T) {
c := make(chan struct{}, 1)
go loop(c)
c <- struct{}{}
ForceGosched()
if d != 1 {
t.Fatal("d is not incremented")
}
}
```
`go test -v -race` runs well, but `GOMAXPROCS=2 go test -v -race` fails.
Change the functionality to waiting for schedule to happen.
This PR sets etcd version and min cluster version in request header,
and let server check version compatibility. rafthttp server
will reject any message from peer with incompatible version(too low
version or too high version), and print out warning logs.
Because etcd 2.1 will build stream to any existing peers and etcd 2.0
requires the remote to provide most updated term, it is
necessary for streamReader to know the latest term.
This helps etcd 2.1 connect to msgappV1 handler when the remote member
doesn't support msgappV2. And it doesn't print out unsupported handler
error to make log clean.
Fix#2841.
From Prometheus developer:
```
the recommended way for etcd as an open source project and under
consideration of its size would be etcd_<subsystem>_<name>.
```
We made the naming change accordingly.
Jonathan Boulle