The suggested pattern for Raft proposals is that they be retried
periodically until they succeed. This turns out to be an issue
when a leader cannot commit entries because the leader will continue
to append re-proposed entries to its log without committing anything.
This can result in the uncommitted tail of a leader's log growing
without bound until it is able to commit entries.
This change add a safeguard to protect against this case where a
leader's log can grow without bound during loss of quorum scenarios.
It does so by introducing a new, optional ``MaxUncommittedEntriesSize
configuration. This config limits the max aggregate size of uncommitted
entries that may be appended to a leader's log. Once this limit
is exceeded, proposals will begin to return ErrProposalDropped
errors.
See cockroachdb/cockroach#27772
The error message is different to the condition it checks for. This PR
modifies the error message accordingly.
Closes#9717.
Signed-off-by: Kostas Christidis <kostas@christidis.io>
Save the snapshot index to the WAL before saving the snapshot to the
filesystem. This ensures that we'll only ever call wal.Open with a
snapshot that was previously saved to the WAL.
The ConfChange fix crashes WAL replay because it assumed the node
always exists. Additionally, restart on a single node cluster
would deadlock because it had the wrong raft HardState.
To fix, replay path now goes through the regular raft loop.
Fixes#4474
Add a channel over which we can propose cluster config changes to
raft. In an upcoming commit we'll add an HTTP endpoint that sends config
changes over this channel.
deadlock if no leader; node selects on propc=nil and writes to Ready,
client blocks on propC in same select as Ready reader, and so progress
of raft state machine deadlocks.
Gyuho Lee