Hello,
I am trying to form a cluster with 2 primaries (I dont need fault tolerance) and later add some secondary servers as well, but to begin only with 2 primaries. I have the following config in neo4j.conf:
server.directories.import=import
server.bolt.enabled=true
server.http.enabled=true
server.https.enabled=false
server.metrics.csv.rotation.compression=zip
server.jvm.additional=-XX:+UseG1GC
server.jvm.additional=-XX:-OmitStackTraceInFastThrow
server.jvm.additional=-XX:+AlwaysPreTouch
server.jvm.additional=-XX:+UnlockExperimentalVMOptions
server.jvm.additional=-XX:+TrustFinalNonStaticFields
server.jvm.additional=-XX:+DisableExplicitGC
server.jvm.additional=-XX:-RestrictContended
server.jvm.additional=-Djdk.nio.maxCachedBufferSize=1024
server.jvm.additional=-Dio.netty.tryReflectionSetAccessible=true
server.jvm.additional=-Djdk.tls.ephemeralDHKeySize=2048
server.jvm.additional=-Djdk.tls.rejectClientInitiatedRenegotiation=true
server.jvm.additional=-XX:FlightRecorderOptions=stackdepth=256
server.jvm.additional=-XX:+UnlockDiagnosticVMOptions
server.jvm.additional=-XX:+DebugNonSafepoints
server.jvm.additional=--add-opens=java.base/java.nio=ALL-UNNAMED
server.jvm.additional=--add-opens=java.base/java.io=ALL-UNNAMED
server.jvm.additional=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED
server.jvm.additional=-Dlog4j2.disable.jmx=true
server.windows_service_name=neo4j
server.default_listen_address=172.16.31.120
server.default_advertised_address=172.16.31.120
dbms.cluster.discovery.endpoints=172.16.30.66:5000,172.16.31.120:5000
initial.dbms.default_primaries_count=2
server.discovery.advertised_address=172.16.31.120:5000
server.cluster.advertised_address=172.16.31.120:6000
server.bolt.listen_address=0.0.0.0:7687
server.bolt.advertised_address=172.16.31.120:7687
The service starts fine on both nodes, however, debug.log shows the following error message on loop. The VM running neo4j are ubuntu 20.04 LTS with no sudo apt upgrade performed. While setting up the same config on an ubuntu 18 LTS the cluster can form fine and I can connect. Firewall is not in discussion as it's not present (nor VM level (firewalld) nor network restrictions)
2024-05-20 20:21:52.243+0000 INFO [c.n.c.d.a.AkkaTopologyService] Restarting discovery system after probable network partition
2024-05-20 20:21:53.044+0000 INFO [c.n.c.d.a.AkkaTopologyService] System bootstrapper selection is removed; was: '6d1484eb-ad8a-4435-a1e1-e46d28a0d15a'
2024-05-20 20:21:53.045+0000 INFO [c.n.c.d.d.TopologyState] Server {6d1484eb} is removed with version 5.19.0 in state AVAILABLE with connectors [BOLT=172.16.30.66:7687,CLUSTER=172.16.30.66:6000,RAFT=172.16.30.66:7000], initial settings ['initial.server.allowed_databases'='','initial.server.denied_databases'='','initial.server.mode_constraint'='PRIMARY','initial.server.tags'='','server.cluster.initial_mode_constraint'='PRIMARY'], component versions [cluster-discovery=4,community-topology-graph=1,dbms-runtime=17,fabric-database=1,security-privileges=18,security-users=4,topology-graph=3]
2024-05-20 20:21:53.045+0000 INFO [c.n.c.d.d.TopologyState] Database {00000000}/system on {6d1484eb} is leaving and removing raft mapping (RaftMemberId{6d1484eb})
2024-05-20 20:21:53.046+0000 INFO [c.n.c.d.d.TopologyState] Database {00000000}/system on {6d1484eb} is removed
2024-05-20 20:21:53.046+0000 INFO [c.n.c.c.c.m.RaftTargetMembershipAggregator] [system/00000000] Discovery source update {RaftMemberId{63e7037b}=VOTER}
2024-05-20 20:21:53.046+0000 INFO [Raft] [system/00000000] [Event] Target membership changed [targetMembers={RaftMemberId{63e7037b}=VOTER}]
2024-05-20 20:21:53.104+0000 WARN [a.e.DummyClassForStringSources] Using serializer [com.neo4j.causalclustering.discovery.akka.marshal.UniqueAddressSerializer] for message [akka.cluster.UniqueAddress]. Note that this serializer is not implemented by Akka. It's not recommended to replace serializers for messages provided by Akka.
2024-05-20 20:21:53.107+0000 INFO [c.n.c.d.a.AkkaTopologyService] Publishing initial data into actor system PublishInitialData[registeredConnectors={RAFT=172.16.31.120:7000, CLUSTER=172.16.31.120:6000, BOLT=172.16.31.120:7687}, localDatabaseInfos={DatabaseId{00000000}=DatabaseInfo[databaseServer=DatabaseServer[databaseId=DatabaseId{00000000}, serverId=ServerId{63e7037b}], operatorState=STARTING, statusMessage=null, mode=RAFT, raftMemberId=RaftMemberId{63e7037b}, databaseAccess=READ_WRITE]}, localLeaderInfos={DatabaseId{00000000}=LeaderInfo{term -1, not yet elected}}, localVersions=ComponentVersionVector[cluster-discovery=4,community-topology-graph=1,dbms-runtime=17,fabric-database=1,security-privileges=18,security-users=4,topology-graph=3], systemBootstrapper=Optional.empty, serializerVersion=Optional.empty]
2024-05-20 20:21:53.108+0000 INFO [c.n.c.d.a.AkkaTopologyService] Successfully restarted discovery system
2024-05-20 20:21:55.186+0000 INFO [c.n.c.d.d.TopologyState] Server {6d1484eb} is added with version 5.19.0 in state AVAILABLE with connectors [BOLT=172.16.30.66:7687,CLUSTER=172.16.30.66:6000,RAFT=172.16.30.66:7000], initial settings ['initial.server.allowed_databases'='','initial.server.denied_databases'='','initial.server.mode_constraint'='PRIMARY','initial.server.tags'='','server.cluster.initial_mode_constraint'='PRIMARY'], component versions
2024-05-20 20:21:55.188+0000 INFO [c.n.c.d.d.TopologyState] Component versions for server {6d1484eb} changed as [added: fabric-database=1, added: cluster-discovery=4, added: topology-graph=3, added: security-privileges=18, added: dbms-runtime=17, added: community-topology-graph=1, added: security-users=4]
2024-05-20 20:21:55.189+0000 INFO [c.n.c.d.a.AkkaTopologyService] System bootstrapper selected; is: '6d1484eb-ad8a-4435-a1e1-e46d28a0d15a'
2024-05-20 20:21:55.190+0000 INFO [c.n.c.d.d.TopologyState] Database {00000000}/system on {6d1484eb} is discoverable in mode RAFT and publishing RaftMemberId{6d1484eb}
2024-05-20 20:21:55.191+0000 INFO [c.n.c.c.c.m.RaftTargetMembershipAggregator] [system/00000000] Discovery source update {RaftMemberId{63e7037b}=VOTER, RaftMemberId{6d1484eb}=VOTER}
2024-05-20 20:21:55.191+0000 INFO [Raft] [system/00000000] [Event] Target membership changed [targetMembers={RaftMemberId{63e7037b}=VOTER, RaftMemberId{6d1484eb}=VOTER}]
2024-05-20 20:21:55.192+0000 INFO [c.n.c.p.CheckConnectivityTopologyListener] Checking connectivity to port RAFT for server 6d1484eb-ad8a-4435-a1e1-e46d28a0d15a, mode PRIMARY. Address 172.16.30.66:7000, result Success
2024-05-20 20:21:55.196+0000 INFO [c.n.c.p.CheckConnectivityTopologyListener] Checking connectivity to port CLUSTER for server 6d1484eb-ad8a-4435-a1e1-e46d28a0d15a, mode PRIMARY. Address 172.16.30.66:6000, result Success
2024-05-20 20:22:23.786+0000 INFO [c.n.c.d.d.TopologyState] Database {00000000}/system on {6d1484eb} is leaving and removing raft mapping (RaftMemberId{6d1484eb})
2024-05-20 20:22:23.787+0000 INFO [c.n.c.d.d.TopologyState] Database {00000000}/system on {6d1484eb} is removed
2024-05-20 20:22:23.787+0000 INFO [c.n.c.c.c.m.RaftTargetMembershipAggregator] [system/00000000] Discovery source update {RaftMemberId{63e7037b}=VOTER}
2024-05-20 20:22:23.788+0000 INFO [Raft] [system/00000000] [Event] Target membership changed [targetMembers={RaftMemberId{63e7037b}=VOTER}]
2024-05-20 20:22:23.789+0000 INFO [c.n.c.d.d.TopologyState] Server {6d1484eb} is removed with version 5.19.0 in state AVAILABLE with connectors [BOLT=172.16.30.66:7687,CLUSTER=172.16.30.66:6000,RAFT=172.16.30.66:7000], initial settings ['initial.server.allowed_databases'='','initial.server.denied_databases'='','initial.server.mode_constraint'='PRIMARY','initial.server.tags'='','server.cluster.initial_mode_constraint'='PRIMARY'], component versions [cluster-discovery=4,community-topology-graph=1,dbms-runtime=17,fabric-database=1,security-privileges=18,security-users=4,topology-graph=3]
2024-05-20 20:22:25.346+0000 INFO [c.n.c.d.d.TopologyState] Database {00000000}/system on {6d1484eb} is discoverable in mode RAFT and publishing RaftMemberId{6d1484eb}
2024-05-20 20:22:25.347+0000 INFO [c.n.c.c.c.m.RaftTargetMembershipAggregator] [system/00000000] Discovery source update {RaftMemberId{63e7037b}=VOTER, RaftMemberId{6d1484eb}=VOTER}
2024-05-20 20:22:25.347+0000 INFO [Raft] [system/00000000] [Event] Target membership changed [targetMembers={RaftMemberId{63e7037b}=VOTER, RaftMemberId{6d1484eb}=VOTER}]
2024-05-20 20:22:25.349+0000 INFO [c.n.c.d.d.TopologyState] Server {6d1484eb} is added with version 5.19.0 in state AVAILABLE with connectors [BOLT=172.16.30.66:7687,CLUSTER=172.16.30.66:6000,RAFT=172.16.30.66:7000], initial settings ['initial.server.allowed_databases'='','initial.server.denied_databases'='','initial.server.mode_constraint'='PRIMARY','initial.server.tags'='','server.cluster.initial_mode_constraint'='PRIMARY'], component versions [cluster-discovery=4,community-topology-graph=1,dbms-runtime=17,fabric-database=1,security-privileges=18,security-users=4,topology-graph=3]
2024-05-20 20:22:25.354+0000 INFO [c.n.c.p.CheckConnectivityTopologyListener] Checking connectivity to port RAFT for server 6d1484eb-ad8a-4435-a1e1-e46d28a0d15a, mode PRIMARY. Address 172.16.30.66:7000, result Success
2024-05-20 20:22:25.358+0000 INFO [c.n.c.p.CheckConnectivityTopologyListener] Checking connectivity to port CLUSTER for server 6d1484eb-ad8a-4435-a1e1-e46d28a0d15a, mode PRIMARY. Address 172.16.30.66:6000, result Success
2024-05-20 20:22:25.765+0000 INFO [c.n.c.p.h.InboundChannelLogger] [raft-server] Initializing server channel [id: 0x34e86c70, L:/172.16.31.120:7000 - R:/172.16.30.66:39690]
2024-05-20 20:22:25.767+0000 INFO [c.n.c.p.h.InboundChannelLogger] [catchup-server] Initializing server channel [id: 0x38e8c2b4, L:/172.16.31.120:6000 - R:/172.16.30.66:39798]
2024-05-20 20:22:25.767+0000 INFO [c.n.c.p.h.InboundChannelLogger] [catchup-server] Initializing server channel [id: 0x88467ede, L:/172.16.31.120:6000 - R:/172.16.30.66:39800]
2024-05-20 20:22:25.768+0000 INFO [c.n.c.p.h.InboundChannelLogger] [catchup-server] Initializing server channel [id: 0xf9587b53, L:/172.16.31.120:6000 - R:/172.16.30.66:39802]
2024-05-20 20:22:45.126+0000 INFO [c.n.c.d.a.AkkaTopologyService] Restarting discovery system after probable network partition