I have a Neo4j deployment in K8s with a replica set of 3, in cluster mode. My configurations are following the general guidelines of the cluster deployment:
echo -e " dbms.mode=CORE dbms.default_advertised_address=$MY_IP causal_clustering.minimum_core_cluster_size_at_formation=$REPLICAS causal_clustering.minimum_core_cluster_size_at_runtime=$((REPLICAS / 2 + 1)) dbms.routing.enabled=true " >> "$NEO4J_HOME/conf/neo4j.conf"
Please note that there is indentation in the conf file as shown below:
dbms.ssl.policy.bolt.enabled=true dbms.ssl.policy.bolt.base_directory=certificates path dbms.ssl.policy.bolt.private_key=server.key dbms.ssl.policy.bolt.public_certificate=server.crt dbms.ssl.policy.bolt.client_auth=OPTIONAL dbms.ssl.policy.bolt.verify_hostname=false dbms.mode=CORE dbms.default_advertised_address=my_ip causal_clustering.minimum_core_cluster_size_at_formation=3 causal_clustering.minimum_core_cluster_size_at_runtime=2 dbms.routing.enabled=true causal_clustering.initial_discovery_members=add1:5000,add2:5000,add3:5000 gds.enterprise.license_file=/home/sandbox/neo/key/key.txt
Even then when I try to connect to a follower using neo4j scheme, it fails and falls back to bolt scheme and I am unable to write, I get the standard error of "this is a follower, write not possible".
How do I connect to Neo4j using neo scheme?
Answer/questions from the team:
What MY_IP here? is that the pod’s address or the address of a load balancer service?
And where is the client connecting from? is it outside the k8s deployment?
it looks to me like they’re trying to use client-side routing through a single LB service, which is not going to work.
If they’re on 4.3 they should use server-side routing (SSR). Otherwise, they should try to follow the example of the Labs Helm chart and create a LB service per cluster member.
Basically you have to use bolt:// on 4.2 and will run into problems with errors if not on leader
There is a 4.3 PR for the labs helm chart that puts SSR in place and with that this problem will go away
Let me restate what you have here. You have setup a Neo4j Cluster of 3 Servers/Pods. You have no read replicas. Please tell us what K8 you are using? Google, AWS, Azure, On-Prem, Ranger?
There will be a 4.3 Helm Chart soon that will do Server Side Routing and that will allow a single IP address to be used.
Since you are (probably) on 4.2.x you need to look at the External Exposure chapter in the doc. Your three pods have addr1,addr2,addr3 inside the k8 cluster. You need a route for each pod outside the k8 cluster, and three IP addresses outside of the cluster to route with. It's just the way this helm chart works. Hopefully you are on one of the cloud providers and as such you will create a load balancer (I know - read the doc) for each pod. AND you will advertise a route for each pod to go to its corresponding load balancer service.
I see you have a gds license (trial maybe) - which means you should be working with someone in the Sales Engineering or Customer Success (if you are client already). It would be helpful if I knew whom you were working with (if any).
Port forwarding - IMO - is not the right option for a cluster in 4.2 unless you get very creative and deviate from the docs.