409 lines
14 KiB
Plaintext
409 lines
14 KiB
Plaintext
# Run a V2 membership change that adds two voters at once and auto-leaves the
|
|
# joint configuration. (This is the same as specifying an explicit transition
|
|
# since more than one change is being made atomically).
|
|
|
|
# Bootstrap n1.
|
|
add-nodes 1 voters=(1) index=2
|
|
----
|
|
INFO 1 switched to configuration voters=(1)
|
|
INFO 1 became follower at term 0
|
|
INFO newRaft 1 [peers: [1], term: 0, commit: 2, applied: 2, lastindex: 2, lastterm: 1]
|
|
|
|
campaign 1
|
|
----
|
|
INFO 1 is starting a new election at term 0
|
|
INFO 1 became candidate at term 1
|
|
INFO 1 received MsgVoteResp from 1 at term 1
|
|
INFO 1 became leader at term 1
|
|
|
|
propose-conf-change 1 transition=auto
|
|
v2 v3
|
|
----
|
|
ok
|
|
|
|
# Add two "empty" nodes to the cluster, n2 and n3.
|
|
add-nodes 2
|
|
----
|
|
INFO 2 switched to configuration voters=()
|
|
INFO 2 became follower at term 0
|
|
INFO newRaft 2 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
|
|
INFO 3 switched to configuration voters=()
|
|
INFO 3 became follower at term 0
|
|
INFO newRaft 3 [peers: [], term: 0, commit: 0, applied: 0, lastindex: 0, lastterm: 0]
|
|
|
|
# n1 immediately gets to commit & apply the conf change using only itself. We see that
|
|
# it starts transitioning out of that joint configuration (though we will only see that
|
|
# proposal in the next ready handling loop, when it is emitted). We also see that this
|
|
# is using joint consensus, which it has to since we're carrying out two additions at
|
|
# once.
|
|
process-ready 1
|
|
----
|
|
Ready MustSync=true:
|
|
Lead:1 State:StateLeader
|
|
HardState Term:1 Vote:1 Commit:4
|
|
Entries:
|
|
1/3 EntryNormal ""
|
|
1/4 EntryConfChangeV2 v2 v3
|
|
CommittedEntries:
|
|
1/3 EntryNormal ""
|
|
1/4 EntryConfChangeV2 v2 v3
|
|
INFO 1 switched to configuration voters=(1 2 3)&&(1) autoleave
|
|
INFO initiating automatic transition out of joint configuration voters=(1 2 3)&&(1) autoleave
|
|
|
|
# n1 immediately probes n2 and n3.
|
|
stabilize 1
|
|
----
|
|
> 1 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/5 EntryConfChangeV2
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
|
|
1->3 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
|
|
|
|
# First, play out the whole interaction between n1 and n2. We see n1's probe to
|
|
# n2 get rejected (since n2 needs a snapshot); the snapshot is delivered at which
|
|
# point n2 switches to the correct config, and n1 catches it up. This notably
|
|
# includes the empty conf change which gets committed and applied by both and
|
|
# which transitions them out of their joint configuration into the final one (1 2 3).
|
|
stabilize 1 2
|
|
----
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
|
|
INFO 2 [term: 0] received a MsgApp message with higher term from 1 [term: 1]
|
|
INFO 2 became follower at term 1
|
|
DEBUG 2 [logterm: 0, index: 3] rejected MsgApp [logterm: 1, index: 3] from 1
|
|
> 2 handling Ready
|
|
Ready MustSync=true:
|
|
Lead:1 State:StateFollower
|
|
HardState Term:1 Commit:0
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
|
|
DEBUG 1 received MsgAppResp(MsgApp was rejected, lastindex: 0) from 2 for index 3
|
|
DEBUG 1 decreased progress of 2 to [StateProbe match=0 next=1]
|
|
DEBUG 1 [firstindex: 3, commit: 4] sent snapshot[index: 4, term: 1] to 2 [StateProbe match=0 next=1]
|
|
DEBUG 1 paused sending replication messages to 2 [StateSnapshot match=0 next=1 paused pendingSnap=4]
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
Messages:
|
|
1->2 MsgSnap Term:1 Log:0/0 Snapshot: Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
|
|
> 2 receiving messages
|
|
1->2 MsgSnap Term:1 Log:0/0 Snapshot: Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
|
|
INFO log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 4, term: 1]
|
|
INFO 2 switched to configuration voters=(1 2 3)&&(1) autoleave
|
|
INFO 2 [commit: 4, lastindex: 4, lastterm: 1] restored snapshot [index: 4, term: 1]
|
|
INFO 2 [commit: 4] restored snapshot [index: 4, term: 1]
|
|
> 2 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Commit:4
|
|
Snapshot Index:4 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[1] Learners:[] LearnersNext:[] AutoLeave:true
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/4
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/4
|
|
DEBUG 1 recovered from needing snapshot, resumed sending replication messages to 2 [StateSnapshot match=4 next=5 paused pendingSnap=4]
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/4 Commit:4 Entries:[1/5 EntryConfChangeV2]
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/4 Commit:4 Entries:[1/5 EntryConfChangeV2]
|
|
> 2 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/5 EntryConfChangeV2
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/5
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/5
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Vote:1 Commit:5
|
|
CommittedEntries:
|
|
1/5 EntryConfChangeV2
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/5 Commit:5
|
|
INFO 1 switched to configuration voters=(1 2 3)
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/5 Commit:5
|
|
> 2 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Commit:5
|
|
CommittedEntries:
|
|
1/5 EntryConfChangeV2
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/5
|
|
INFO 2 switched to configuration voters=(1 2 3)
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/5
|
|
|
|
# n3 immediately receives a snapshot in the final configuration.
|
|
stabilize 1 3
|
|
----
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/3 Commit:4 Entries:[1/4 EntryConfChangeV2 v2 v3]
|
|
INFO 3 [term: 0] received a MsgApp message with higher term from 1 [term: 1]
|
|
INFO 3 became follower at term 1
|
|
DEBUG 3 [logterm: 0, index: 3] rejected MsgApp [logterm: 1, index: 3] from 1
|
|
> 3 handling Ready
|
|
Ready MustSync=true:
|
|
Lead:1 State:StateFollower
|
|
HardState Term:1 Commit:0
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
|
|
> 1 receiving messages
|
|
3->1 MsgAppResp Term:1 Log:0/3 Rejected (Hint: 0)
|
|
DEBUG 1 received MsgAppResp(MsgApp was rejected, lastindex: 0) from 3 for index 3
|
|
DEBUG 1 decreased progress of 3 to [StateProbe match=0 next=1]
|
|
DEBUG 1 [firstindex: 3, commit: 5] sent snapshot[index: 5, term: 1] to 3 [StateProbe match=0 next=1]
|
|
DEBUG 1 paused sending replication messages to 3 [StateSnapshot match=0 next=1 paused pendingSnap=5]
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
Messages:
|
|
1->3 MsgSnap Term:1 Log:0/0 Snapshot: Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
|
|
> 3 receiving messages
|
|
1->3 MsgSnap Term:1 Log:0/0 Snapshot: Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
|
|
INFO log [committed=0, applied=0, unstable.offset=1, len(unstable.Entries)=0] starts to restore snapshot [index: 5, term: 1]
|
|
INFO 3 switched to configuration voters=(1 2 3)
|
|
INFO 3 [commit: 5, lastindex: 5, lastterm: 1] restored snapshot [index: 5, term: 1]
|
|
INFO 3 [commit: 5] restored snapshot [index: 5, term: 1]
|
|
> 3 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Commit:5
|
|
Snapshot Index:5 Term:1 ConfState:Voters:[1 2 3] VotersOutgoing:[] Learners:[] LearnersNext:[] AutoLeave:false
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/5
|
|
> 1 receiving messages
|
|
3->1 MsgAppResp Term:1 Log:0/5
|
|
DEBUG 1 recovered from needing snapshot, resumed sending replication messages to 3 [StateSnapshot match=5 next=6 paused pendingSnap=5]
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
Messages:
|
|
1->3 MsgApp Term:1 Log:1/5 Commit:5
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/5 Commit:5
|
|
> 3 handling Ready
|
|
Ready MustSync=false:
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/5
|
|
> 1 receiving messages
|
|
3->1 MsgAppResp Term:1 Log:0/5
|
|
|
|
# Nothing else happens.
|
|
stabilize
|
|
----
|
|
ok
|
|
|
|
# Now remove two nodes. What's new here is that the leader will actually have
|
|
# to go to a quorum to commit the transition into the joint config.
|
|
|
|
propose-conf-change 1
|
|
r2 r3
|
|
----
|
|
ok
|
|
|
|
# n1 sends out MsgApps.
|
|
stabilize 1
|
|
----
|
|
> 1 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
|
1->3 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
|
|
|
# n2, n3 ack them.
|
|
stabilize 2 3
|
|
----
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/5 Commit:5 Entries:[1/6 EntryConfChangeV2 r2 r3]
|
|
> 2 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/6
|
|
> 3 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/6
|
|
|
|
# n1 gets some more proposals. This is part of a regression test: There used to
|
|
# be a bug in which these proposals would prompt the leader to transition out of
|
|
# the same joint state multiple times, which would cause a panic.
|
|
propose 1 foo
|
|
----
|
|
ok
|
|
|
|
propose 1 bar
|
|
----
|
|
ok
|
|
|
|
# n1 switches to the joint config, then initiates a transition into the final
|
|
# config.
|
|
stabilize 1
|
|
----
|
|
> 1 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
|
1->3 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
|
1->2 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
|
1->3 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/6
|
|
3->1 MsgAppResp Term:1 Log:0/6
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Vote:1 Commit:6
|
|
CommittedEntries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/8 Commit:6
|
|
1->3 MsgApp Term:1 Log:1/8 Commit:6
|
|
INFO 1 switched to configuration voters=(1)&&(1 2 3) autoleave
|
|
INFO initiating automatic transition out of joint configuration voters=(1)&&(1 2 3) autoleave
|
|
> 1 handling Ready
|
|
Ready MustSync=true:
|
|
Entries:
|
|
1/9 EntryConfChangeV2
|
|
|
|
# n2 and n3 also switch to the joint config, and ack the transition out of it.
|
|
stabilize 2 3
|
|
----
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
|
1->2 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
|
1->2 MsgApp Term:1 Log:1/8 Commit:6
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/6 Commit:5 Entries:[1/7 EntryNormal "foo"]
|
|
1->3 MsgApp Term:1 Log:1/7 Commit:5 Entries:[1/8 EntryNormal "bar"]
|
|
1->3 MsgApp Term:1 Log:1/8 Commit:6
|
|
> 2 handling Ready
|
|
Ready MustSync=true:
|
|
HardState Term:1 Commit:6
|
|
Entries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
CommittedEntries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/7
|
|
2->1 MsgAppResp Term:1 Log:0/8
|
|
2->1 MsgAppResp Term:1 Log:0/8
|
|
INFO 2 switched to configuration voters=(1)&&(1 2 3) autoleave
|
|
> 3 handling Ready
|
|
Ready MustSync=true:
|
|
HardState Term:1 Commit:6
|
|
Entries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
CommittedEntries:
|
|
1/6 EntryConfChangeV2 r2 r3
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/7
|
|
3->1 MsgAppResp Term:1 Log:0/8
|
|
3->1 MsgAppResp Term:1 Log:0/8
|
|
INFO 3 switched to configuration voters=(1)&&(1 2 3) autoleave
|
|
|
|
# n2 and n3 also leave the joint config and the dust settles. We see at the very
|
|
# end that n1 receives some messages from them that it refuses because it does
|
|
# not have them in its config any more.
|
|
stabilize
|
|
----
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/7
|
|
2->1 MsgAppResp Term:1 Log:0/8
|
|
2->1 MsgAppResp Term:1 Log:0/8
|
|
3->1 MsgAppResp Term:1 Log:0/7
|
|
3->1 MsgAppResp Term:1 Log:0/8
|
|
3->1 MsgAppResp Term:1 Log:0/8
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Vote:1 Commit:8
|
|
CommittedEntries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
|
1->3 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
|
1->2 MsgApp Term:1 Log:1/9 Commit:8
|
|
1->3 MsgApp Term:1 Log:1/9 Commit:8
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
|
1->2 MsgApp Term:1 Log:1/9 Commit:8
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/8 Commit:7 Entries:[1/9 EntryConfChangeV2]
|
|
1->3 MsgApp Term:1 Log:1/9 Commit:8
|
|
> 2 handling Ready
|
|
Ready MustSync=true:
|
|
HardState Term:1 Commit:8
|
|
Entries:
|
|
1/9 EntryConfChangeV2
|
|
CommittedEntries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
> 3 handling Ready
|
|
Ready MustSync=true:
|
|
HardState Term:1 Commit:8
|
|
Entries:
|
|
1/9 EntryConfChangeV2
|
|
CommittedEntries:
|
|
1/7 EntryNormal "foo"
|
|
1/8 EntryNormal "bar"
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
> 1 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Vote:1 Commit:9
|
|
CommittedEntries:
|
|
1/9 EntryConfChangeV2
|
|
Messages:
|
|
1->2 MsgApp Term:1 Log:1/9 Commit:9
|
|
1->3 MsgApp Term:1 Log:1/9 Commit:9
|
|
INFO 1 switched to configuration voters=(1)
|
|
> 2 receiving messages
|
|
1->2 MsgApp Term:1 Log:1/9 Commit:9
|
|
> 3 receiving messages
|
|
1->3 MsgApp Term:1 Log:1/9 Commit:9
|
|
> 2 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Commit:9
|
|
CommittedEntries:
|
|
1/9 EntryConfChangeV2
|
|
Messages:
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
INFO 2 switched to configuration voters=(1)
|
|
> 3 handling Ready
|
|
Ready MustSync=false:
|
|
HardState Term:1 Commit:9
|
|
CommittedEntries:
|
|
1/9 EntryConfChangeV2
|
|
Messages:
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
INFO 3 switched to configuration voters=(1)
|
|
> 1 receiving messages
|
|
2->1 MsgAppResp Term:1 Log:0/9
|
|
raft: cannot step as peer not found
|
|
3->1 MsgAppResp Term:1 Log:0/9
|
|
raft: cannot step as peer not found
|