I am getting errors in the debug logs where Replication attempts to leader have failed repeatedly and is causing hung queries.
2021-03-27 20:08:00.423+0000 INFO [o.n.c.c.r.RaftReplicator] Replication attempt 82 to leader MemberId{xleaderx}: DistributedOperation{content=TransactionRepresentationReplicatedTransaction{tx=PhysicalTransactionRepresentation[masterId:-1,authorId:-1,timeStarted:1616871019605,latestCommittedTxWhenStarted:79746,timeCommitted:1616871019605,lockSession:2,additionalHeader:[]commands.length:1}, globalSession=GlobalSession{sessionId=___, owner=MemberId{xleaderx}}, operationId=LocalOperationId{localSessionId=0, sequenceNumber=6999}}
2021-03-27 20:08:01.129+0000 INFO [o.n.c.c.c.s.RaftLogShipper] MemberId{xFOL1x}[matchIndex: 84176, lastSentIndex: 84240, localAppendIndex: 84260, mode: CATCHUP]: timed out, moving to MISMATCH mode
2021-03-27 20:08:01.132+0000 WARN [o.n.c.c.c.s.RaftLogShipper] MemberId{xFOL1x}[matchIndex: 84176, lastSentIndex: 84177, localAppendIndex: 84260, mode: MISMATCH]: match index not progressing. This should be transient.
2021-03-27 20:08:01.308+0000 INFO [o.n.c.c.c.s.RaftLogShipper] MemberId{xFOL1x}[matchIndex: 84176, lastSentIndex: 84240, localAppendIndex: 84260, mode: MISMATCH]: starting catch up after mismatch, moving to CATCHUP mode
2021-03-27 20:08:05.734+0000 INFO [o.n.c.c.c.s.RaftLogShipper] MemberId{xFOL2x}[matchIndex: 84176, lastSentIndex: 84240, localAppendIndex: 84260, mode: CATCHUP]: timed out, moving to MISMATCH mode
2021-03-27 20:08:05.737+0000 WARN [o.n.c.c.c.s.RaftLogShipper] MemberId{xFOL2x}[matchIndex: 84176, lastSentIndex: 84177, localAppendIndex: 84260, mode: MISMATCH]: match index not progressing. This should be transient.
2021-03-27 20:08:05.935+0000 INFO [o.n.c.c.c.s.RaftLogShipper] MemberId{xFOL2x}[matchIndex: 84176, lastSentIndex: 84240, localAppendIndex: 84260, mode: MISMATCH]: starting catch up after mismatch, moving to CATCHUP mode
2021-03-27 20:08:08.132+0000 INFO [o.n.c.c.c.s.RaftLogShipper] MemberId{xFOL1x}[matchIndex: 84176, lastSentIndex: 84240, localAppendIndex: 84260, mode: CATCHUP]: timed out, moving to MISMATCH mode
I am at a loss on how to bring the cluster back into sync. There are no running transactions or queries. Has anyone come across this issue before? What options do I have here?