GKE K8s Deployment not forming a cluster

Integrations Orchestration & Kubernetes
1

We've trying to deploy Neo4j cluster (5.24.1 enterprise) on GKE using the following values.yaml via helm:

neo4j:
  name: "neo4j-cluster"
  minimumClusterSize: 3
  edition: "enterprise"
  passwordFromSecret: "neo4j-auth"  # This should be the name of your existing secret
  config:
    dbms.memory.transaction.total.max: '0'
  acceptLicenseAgreement: "yes"
  resources:
    requests:
      cpu: "1"
      memory: "2Gi"
    limits:
      cpu: "2"
      memory: "4Gi"
env:
  NEO4J_PLUGINS: '["apoc"]'

config:
  dbms.security.procedures.unrestricted: "apoc.*"
  dbms.security.procedures.allowlist: "apoc.*"

volumes:
  data:
    mode: "dynamic"
    dynamic:
      storageClassName: premium-rwo

As defined in the docs,
deploying the same values.yaml named server1/2/3 for each server would form a cluster. We're able to successfully deploy the same.

k get pods -n neo4j-cluster

NAME READY STATUS RESTARTS AGE
server-1-0 1/1 Running 0 49m
server-2-0 1/1 Running 0 49m
server-3-0 1/1 Running 0 49m

Checking out the logs for all three deployed server, it shows that each server is able to discover other nodes.

server-1 logs:

2024-10-08 11:22:21.522 IST
2024-10-08 05:52:21.522+0000 INFO Starting...
2024-10-08 11:22:25.762 IST
2024-10-08 05:52:25.762+0000 INFO ======== Neo4j 5.24.1 ========
2024-10-08 11:22:25.796 IST
2024-10-08 05:52:25.795+0000 INFO This instance is ServerId{c0d5ecb3} (c0d5ecb3-8de5-48e7-b872-6b47a002ea15)
2024-10-08 11:22:26.580 IST
2024-10-08 05:52:26.580+0000 INFO Resolved endpoints with K8S{address:'kubernetes.default.svc:443',portName:'tcp-discovery',labelSelector:'app=neo4j-cluster,helm.neo4j.com/service=internals,helm.neo4j.com/clustering=true',clusterDomain:'cluster.local'} to '[server-1-internals.neo4j-cluster.svc.cluster.local:5000, server-2-internals.neo4j-cluster.svc.cluster.local:5000, server-3-internals.neo4j-cluster.svc.cluster.local:5000]'
2024-10-08 11:22:28.380 IST
2024-10-08 05:52:28.379+0000 INFO Resolved endpoints with K8S{address:'kubernetes.default.svc:443',portName:'tcp-discovery',labelSelector:'app=neo4j-cluster,helm.neo4j.com/service=internals,helm.neo4j.com/clustering=true',clusterDomain:'cluster.local'} to '[server-1-internals.neo4j-cluster.svc.cluster.local:5000, server-2-internals.neo4j-cluster.svc.cluster.local:5000, server-3-internals.neo4j-cluster.svc.cluster.local:5000]'
2024-10-08 11:22:49.489 IST
2024-10-08 05:52:49.489+0000 INFO Default database 'neo4j' is created
2024-10-08 11:22:51.078 IST
2024-10-08 05:52:51.078+0000 INFO Connected to server-2-internals.neo4j-cluster.svc.cluster.local/34.118.226.249:7000 [Category[name=RAFT] version:2.0]
2024-10-08 11:22:51.080 IST
2024-10-08 05:52:51.080+0000 INFO Connected to server-3-internals.neo4j-cluster.svc.cluster.local/34.118.231.108:7000 [Category[name=RAFT] version:2.0]
2024-10-08 11:22:51.265 IST
2024-10-08 05:52:51.265+0000 INFO Connected to server-3-internals.neo4j-cluster.svc.cluster.local/34.118.231.108:7000 [Category[name=RAFT] version:2.0]
2024-10-08 11:22:51.275 IST
2024-10-08 05:52:51.274+0000 INFO Connected to server-2-internals.neo4j-cluster.svc.cluster.local/34.118.226.249:7000 [Category[name=RAFT] version:2.0]
2024-10-08 11:22:52.051 IST
2024-10-08 05:52:52.033+0000 INFO Anonymous Usage Data is being sent to Neo4j, see https://neo4j.com/docs/usage-data/
2024-10-08 11:22:52.056 IST
2024-10-08 05:52:52.056+0000 INFO Sending metrics to CSV file at /metrics
2024-10-08 11:22:52.179 IST
2024-10-08 05:52:52.179+0000 INFO Bolt enabled on 0.0.0.0:7687.
2024-10-08 11:22:52.188 IST
2024-10-08 05:52:52.187+0000 INFO Bolt (Routing) enabled on 0.0.0.0:7688.
2024-10-08 11:22:53.895 IST
2024-10-08 05:52:53.895+0000 INFO HTTP enabled on 0.0.0.0:7474.
2024-10-08 11:22:53.897 IST
2024-10-08 05:52:53.896+0000 INFO Remote interface available at http://localhost:7474/
2024-10-08 11:22:53.952 IST
2024-10-08 05:52:53.952+0000 INFO id: F9DEEA541CF8F887CABF1CCA5C39C02F364032076650B0BD0A414FE986F26E39
2024-10-08 11:22:53.953 IST
2024-10-08 05:52:53.953+0000 INFO name: system
2024-10-08 11:22:53.953 IST
2024-10-08 05:52:53.953+0000 INFO creationDate: 2024-10-08T05:52:48.221Z
2024-10-08 11:22:53.954 IST
2024-10-08 05:52:53.954+0000 INFO Started.
2024-10-08 11:23:01.603 IST
2024-10-08 05:53:01.602+0000 INFO Connected to server-3-internals.neo4j-cluster.svc.cluster.local/34.118.231.108:7000 [Category[name=RAFT] version:2.0]
2024-10-08 11:23:01.623 IST
2024-10-08 05:53:01.622+0000 INFO Connected to server-2-internals.neo4j-cluster.svc.cluster.local/34.118.226.249:7000 [Category[name=RAFT] version:2.0]

server-2 logs:

2024-10-08 11:22:18.866 IST
2024-10-08 05:52:18.866+0000 INFO Starting...
2024-10-08 11:22:22.747 IST
2024-10-08 05:52:22.747+0000 INFO ======== Neo4j 5.24.1 ========
2024-10-08 11:22:22.773 IST
2024-10-08 05:52:22.772+0000 INFO This instance is ServerId{a00df9f9} (a00df9f9-d31a-48a8-b3e8-27ee675e6263)
2024-10-08 11:22:23.533 IST
2024-10-08 05:52:23.533+0000 INFO Resolved endpoints with K8S{address:'kubernetes.default.svc:443',portName:'tcp-discovery',labelSelector:'app=neo4j-cluster,helm.neo4j.com/service=internals,helm.neo4j.com/clustering=true',clusterDomain:'cluster.local'} to '[server-1-internals.neo4j-cluster.svc.cluster.local:5000, server-2-internals.neo4j-cluster.svc.cluster.local:5000, server-3-internals.neo4j-cluster.svc.cluster.local:5000]'
2024-10-08 11:22:25.133 IST
2024-10-08 05:52:25.132+0000 INFO Resolved endpoints with K8S{address:'kubernetes.default.svc:443',portName:'tcp-discovery',labelSelector:'app=neo4j-cluster,helm.neo4j.com/service=internals,helm.neo4j.com/clustering=true',clusterDomain:'cluster.local'} to '[server-1-internals.neo4j-cluster.svc.cluster.local:5000, server-2-internals.neo4j-cluster.svc.cluster.local:5000, server-3-internals.neo4j-cluster.svc.cluster.local:5000]'
2024-10-08 11:22:51.160 IST
2024-10-08 05:52:51.160+0000 INFO Connected to server-1-internals.neo4j-cluster.svc.cluster.local/34.118.236.244:7000 [Category[name=RAFT] version:2.0]
2024-10-08 11:22:57.649 IST
2024-10-08 05:52:57.648+0000 INFO Anonymous Usage Data is being sent to Neo4j, see https://neo4j.com/docs/usage-data/
2024-10-08 11:22:57.659 IST
2024-10-08 05:52:57.656+0000 INFO Sending metrics to CSV file at /metrics
2024-10-08 11:22:57.761 IST
2024-10-08 05:52:57.760+0000 INFO Bolt enabled on 0.0.0.0:7687.
2024-10-08 11:22:57.765 IST
2024-10-08 05:52:57.765+0000 INFO Bolt (Routing) enabled on 0.0.0.0:7688.
2024-10-08 11:22:59.872 IST
2024-10-08 05:52:59.872+0000 INFO HTTP enabled on 0.0.0.0:7474.
2024-10-08 11:22:59.873 IST
2024-10-08 05:52:59.873+0000 INFO Remote interface available at http://localhost:7474/
2024-10-08 11:22:59.918 IST
2024-10-08 05:52:59.918+0000 INFO id: F9DEEA541CF8F887CABF1CCA5C39C02F364032076650B0BD0A414FE986F26E39
2024-10-08 11:22:59.919 IST
2024-10-08 05:52:59.919+0000 INFO name: system
2024-10-08 11:22:59.919 IST
2024-10-08 05:52:59.919+0000 INFO creationDate: 2024-10-08T05:52:48.221Z
2024-10-08 11:22:59.919 IST
2024-10-08 05:52:59.919+0000 INFO Started.

server-3 logs:

2024-10-08 11:22:37.267 IST
2024-10-08 05:52:37.267+0000 INFO ======== Neo4j 5.24.1 ========
2024-10-08 11:22:37.295 IST
2024-10-08 05:52:37.294+0000 INFO This instance is ServerId{b6a35149} (b6a35149-6167-4415-b764-4a5ce89cc281)
2024-10-08 11:22:38.052 IST
2024-10-08 05:52:38.052+0000 INFO Resolved endpoints with K8S{address:'kubernetes.default.svc:443',portName:'tcp-discovery',labelSelector:'app=neo4j-cluster,helm.neo4j.com/service=internals,helm.neo4j.com/clustering=true',clusterDomain:'cluster.local'} to '[server-1-internals.neo4j-cluster.svc.cluster.local:5000, server-2-internals.neo4j-cluster.svc.cluster.local:5000, server-3-internals.neo4j-cluster.svc.cluster.local:5000]'
2024-10-08 11:22:39.746 IST
2024-10-08 05:52:39.745+0000 INFO Resolved endpoints with K8S{address:'kubernetes.default.svc:443',portName:'tcp-discovery',labelSelector:'app=neo4j-cluster,helm.neo4j.com/service=internals,helm.neo4j.com/clustering=true',clusterDomain:'cluster.local'} to '[server-1-internals.neo4j-cluster.svc.cluster.local:5000, server-2-internals.neo4j-cluster.svc.cluster.local:5000, server-3-internals.neo4j-cluster.svc.cluster.local:5000]'
2024-10-08 11:22:51.155 IST
2024-10-08 05:52:51.155+0000 INFO Connected to server-1-internals.neo4j-cluster.svc.cluster.local/34.118.236.244:7000 [Category[name=RAFT] version:2.0]
2024-10-08 11:22:58.065 IST
2024-10-08 05:52:58.065+0000 INFO Anonymous Usage Data is being sent to Neo4j, see https://neo4j.com/docs/usage-data/
2024-10-08 11:22:58.069 IST
2024-10-08 05:52:58.069+0000 INFO Sending metrics to CSV file at /metrics
2024-10-08 11:22:58.144 IST
2024-10-08 05:52:58.144+0000 INFO Bolt enabled on 0.0.0.0:7687.
2024-10-08 11:22:58.147 IST
2024-10-08 05:52:58.147+0000 INFO Bolt (Routing) enabled on 0.0.0.0:7688.
2024-10-08 11:23:00.351 IST
2024-10-08 05:53:00.351+0000 INFO HTTP enabled on 0.0.0.0:7474.
2024-10-08 11:23:00.352 IST
2024-10-08 05:53:00.352+0000 INFO Remote interface available at http://localhost:7474/
2024-10-08 11:23:00.356 IST
2024-10-08 05:53:00.356+0000 INFO id: F9DEEA541CF8F887CABF1CCA5C39C02F364032076650B0BD0A414FE986F26E39
2024-10-08 11:23:00.357 IST
2024-10-08 05:53:00.357+0000 INFO name: system
2024-10-08 11:23:00.358 IST
2024-10-08 05:53:00.357+0000 INFO creationDate: 2024-10-08T05:52:48.221Z
2024-10-08 11:23:00.397 IST
2024-10-08 05:53:00.397+0000 INFO Started.

I logged into Neo4j browser via its LB address:

SHOW SERVER output:
│name │address │state │health │hosting │
╞══════════════════════════════════════╪═══════════════════════════════════════════════╪═════════╪═══════════╪═══════════════════╡
│"a00df9f9-d31a-48a8-b3e8-27ee675e6263"│"server-2.neo4j-cluster.svc.cluster.local:7687"│"Enabled"│"Available"│["neo4j", "system"]│
├──────────────────────────────────────┼───────────────────────────────────────────────┼─────────┼───────────┼───────────────────┤
│"b6a35149-6167-4415-b764-4a5ce89cc281"│"server-3.neo4j-cluster.svc.cluster.local:7687"│"Enabled"│"Available"│["neo4j", "system"]│
├──────────────────────────────────────┼───────────────────────────────────────────────┼─────────┼───────────┼───────────────────┤
│"c0d5ecb3-8de5-48e7-b872-6b47a002ea15"│"server-1.neo4j-cluster.svc.cluster.local:7687"│"Enabled"│"Available"│["neo4j", "system"]│

:sysinfo output:

Store Size
Total 17.48 MiB
Database 9.67 MiB

Id Allocation
Node ID 881
Property ID -1
Relationship ID 384
Relationship Type ID 4

Page Cache
Hits 48816
Page Faults 1426
Hit Ratio 100.00%
Usage Ratio 0.70%

Transactions
Last Tx Id 46
Current Read 1
Current Write 0
Peak Transactions 1
Committed Read 70
Committed Write 0

Databases
Name Address Role Status Default Error
neo4j server-3.neo4j-cluster.svc.cluster.local:7687 primary online true -
neo4j server-1.neo4j-cluster.svc.cluster.local:7687 primary online true -
neo4j server-2.neo4j-cluster.svc.cluster.local:7687 primary online true -
system server-1.neo4j-cluster.svc.cluster.local:7687 primary online - -
system server-2.neo4j-cluster.svc.cluster.local:7687 primary online - -
system server-3.neo4j-cluster.svc.cluster.local:7687 primary online - -

CALL dbms.cluster.overview() output:

Neo.ClientError.Procedure.ProcedureNotFound
There is no procedure with the name `dbms.cluster.overview` registered for this database instance. Please ensure you've spelled the procedure name correctly and that the procedure is properly deployed.

I verified it too via connecting the cluster to the Halin dashboard (https://halin.graphapp.io/),
and it shows that cluster is not formed:

(Pasteboard - Uploaded Image)

What are we missing? how to debug this out?

2

@priyanshu

It appears you are using Neo4j 5.24? correct?

Regarding

CALL dbms.cluster.overview() output:

Neo.ClientError.Procedure.ProcedureNotFound
There is no procedure with the name `dbms.cluster.overview` registered for this

and per the 5.x documentation at Procedures - Operations Manual this reports

Name                                   Community     Enterprise       Replaced By
                                              Edition             Edition

dbms.cluster.overview()    No                    Yes                   SHOW SERVERS

as to

I verified it too via connecting the cluster to the Halin dashboard (https://halin.graphapp.io/),

Halin has ceased development for some 2+ years so might not be a great option with 5x