1. setting environment variable cannot be in quote
2. "--race" testing for unit tests is supposed to be part of linux-amd64-unit-4-cpu-race config.
3. 'run' function in test script should log_error in case of failed
command (wrong operator for ints comparison in bash).
* ./tests: Remove legacy coverage collection code
The legacy tests/cover.test.bash script was not ./test script
compatible for a long time.
The following method of coverage collection works (also across
packages) and does not make all the test execution slower.
```
COVERDIR=coverage PASSES="build build_cov cov" ./test
go tool cover -html ./coverage/cover.out
```
* CI: Reduce duplicated coverage between different variants on Travis
We used to execute unit tests in 3 different jobs,
every time with --race detection and every time in 3 variants:1,2,4
CPUS.
The proposed change makes each of the jobs use different variant of
CPUS, and only 4-cpu variant is running with --race detection
(as the more-parallel variant is more likely to experience races),
This refactoring offers following benefits:
- Unified way how go test commands are being called (in terms of flags intepretation)
- Uses standard go mechanisms (like go lists) to find files/packages that are subject for test. The mechanism are module aware.
- Added instruction how to install tools needed for the tests/checkers.
- Added colors to the output to make it easier to spot any failure.
Confirmed to work using:
- COVERDIR="./coverage" CPU="4" RACE=false COVER=false PASSES="build build_cov cov" ./test
- CPU="4" RACE=false COVER=false PASSES="e2e functional integration" ./test
- COVERDIR="./coverage" COVER="false" CPU="4" RACE="false" PASSES="fmt build unit build_cov integration e2e integration_e2e grpcproxy cov" ./test
- PASSES=unit PKG=./wal TIMEOUT=1m ./test
- PASSES=integration PKG=./clientv3 TIMEOUT=1m ./test
- PASSES=unit PKG=./wal TESTCASE=TestNew TIMEOUT=1m ./test
- PASSES=unit PKG=./wal TESTCASE="\bTestNew\b" TIMEOUT=1m ./test
- PASSES=integration PKG=./client/integration TESTCASE="\bTestV2NoRetryEOF\b" TIMEOUT=1m ./test
- COVERDIR=coverage PASSES="build_cov cov" ./test
Added check that ensures that go.mod & go.sum files are up-to-date.
The check verifies whether 'go mod tidy' does not generate any mutations
in these files.
The check can be run on its own:
PASSES="mod_tidy" ./test
Or as part of "fmt" pass:
PASSES="fmt" ./test
Examplar outputs:
```
% PASSES="fmt" ./test
Running with TEST_CPUS: 1,2,4
Starting 'fmt' pass at Fri 11 Sep 2020 11:07:54 PM CEST
'shellcheck' started at Fri 11 Sep 2020 11:07:54 PM CEST
'shellcheck' completed at Fri 11 Sep 2020 11:07:54 PM CEST
'markdown_you' started at Fri 11 Sep 2020 11:07:54 PM CEST
'markdown_you' completed at Fri 11 Sep 2020 11:07:54 PM CEST
'goword' started at Fri 11 Sep 2020 11:07:54 PM CEST
'goword' completed at Fri 11 Sep 2020 11:07:54 PM CEST
'gofmt' started at Fri 11 Sep 2020 11:07:54 PM CEST
'gofmt' completed at Fri 11 Sep 2020 11:07:55 PM CEST
'govet' started at Fri 11 Sep 2020 11:07:55 PM CEST
'govet' completed at Fri 11 Sep 2020 11:07:57 PM CEST
'revive' started at Fri 11 Sep 2020 11:07:57 PM CEST
Skipping revive...
'revive' completed at Fri 11 Sep 2020 11:07:57 PM CEST
'license_header' started at Fri 11 Sep 2020 11:07:57 PM CEST
'license_header' completed at Fri 11 Sep 2020 11:07:58 PM CEST
'receiver_name' started at Fri 11 Sep 2020 11:07:58 PM CEST
'receiver_name' completed at Fri 11 Sep 2020 11:07:58 PM CEST
'commit_title' started at Fri 11 Sep 2020 11:07:58 PM CEST
'commit_title' completed at Fri 11 Sep 2020 11:07:58 PM CEST
'mod_tidy' started at Fri 11 Sep 2020 11:07:58 PM CEST
*** /tmp/fileiALKRA_go.mod 2020-09-11 23:07:58.838010716 +0200
--- ./go.mod 2020-09-11 23:07:58.974010922 +0200
***************
*** 29,39 ****
github.com/mattn/go-runewidth v0.0.2 // indirect
github.com/modern-go/reflect2 v1.0.1
github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5
github.com/prometheus/client_golang v1.0.0
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4
- github.com/prometheus/common v0.4.1
github.com/sirupsen/logrus v1.4.2 // indirect
github.com/soheilhy/cmux v0.1.4
github.com/spf13/cobra v0.0.3
github.com/spf13/pflag v1.0.1
github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8
--- 29,38 ----
./go.mod is not in sync with 'go mod tidy'
```
```
% PASSES="mod_tidy" ./test
Running with TEST_CPUS: 1,2,4
Starting 'mod_tidy' pass at Fri 11 Sep 2020 11:09:21 PM CEST
*** /tmp/file9gy4so_go.mod 2020-09-11 23:09:21.166133290 +0200
--- ./go.mod 2020-09-11 23:09:21.286133466 +0200
***************
*** 29,39 ****
github.com/mattn/go-runewidth v0.0.2 // indirect
github.com/modern-go/reflect2 v1.0.1
github.com/olekukonko/tablewriter v0.0.0-20170122224234-a0225b3f23b5
github.com/prometheus/client_golang v1.0.0
github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4
- github.com/prometheus/common v0.4.1
github.com/sirupsen/logrus v1.4.2 // indirect
github.com/soheilhy/cmux v0.1.4
github.com/spf13/cobra v0.0.3
github.com/spf13/pflag v1.0.1
github.com/tmc/grpc-websocket-proxy v0.0.0-20170815181823-89b8d40f7ca8
--- 29,38 ----
./go.mod is not in sync with 'go mod tidy'
```
This run *should* certainly pass, but it's consistently the one that
fails with a regularity that essentially blocks the CI pipeline.
Someone needs to take a look at #10700, but in the meantime, the show
must go on.
We only get 5 concurrent workers in Travis.
Save time by fetching test image only once
for all build tests.
Signed-off-by: Gyuho Lee <gyuhox@gmail.com>
Piotr Tabor