Troubleshooting Connection Issues to Neo4j

O.K. David, thank you very much indeed. I'll set up a new environment without the loadbalancer and check how it works.

Thanks for you support

Issue can resolved by doing 2 steps.

  1. Add route 53 in AWS of Neo4j instance with ip address - copy that hostname.
  2. Edit the neo4j.config ... Added one more line item as
  3. Stop the DB and restart again. it will automatically appear bolt connection details, then login using credentials. You will able to access DB.


Finally I've got the solution :laughing: Maybe it was too simple to be mentioned somewhere, but for me the key: I opened the address "https://domain:7687" and there I added the certificate once again manually and voilà! Now everything is working fine.
I hope this would also help others not to struggle so long with this issue.

1 Like

It surely is a matter of WebSocket timeout in my case, this WebSocket connection failure pops out only when I run a heavy query through the Neo4j Browser.

Here's the situation, my graph contains ~145000 nodes and ~152000 relationships, everything goes fine when I run a query through cypher-shell:

MATCH (a:Device {dID:"441076300000001025283863"}), (z:Device {dID:"441076300000001025271029"})
CALL apoc.path.expandConfig(a, {relationshipFilter:"CABLE_SEG|PIGTAIL", limit:500, terminatorNodes:[z]}) yield path as p
WHERE all(n IN relationships(p) where n.kx_count>0)
AND apoc.coll.containsAll([n IN nodes(p) | n.dID], ["441076300000001025283550"])
AND NOT ANY(n IN nodes(p) WHERE n.dID IN ["441076300000001025277680","441076300000001025277884"])
WITH p, apoc.coll.sum([x IN relationships(p) | x.stpCount]) as stpSum
RETURN p, stpSum

I checked the query plan it took ~100000ms to get 14 rows of result from my graph, it worked:

AND it also went fine when I lowered the parameter limit to 200 in the apoc.path.expandConfig() call on the Neo4j Browser:

So I believe it's a matter of timeout issue on the WebSocket, my concern is whether or not this issue will happen when I use the HTTP API to run a heavy query? what is the threshold? at where I can adjust it?

I get that all sorts of fancy security measures are needed while attempting to run neo4j on AWS or complex configurations with containers and all that.

Is it asking too much for the docs and distribution to provide a step-by-step recipe for a dirt-simple hello-world smoke test?

I'm trying to get neo4j running on a reasonably vanilla-flavored CentOS 7 guest VM running in a vmware hypervisor on a Windows 10 Pro host. The underlying Windows 10 system is more than adequate.

I want to access neo4j using a standard web browser (Chrome or Firefox) running on the host.

This configuration, other than neo4j, has been working just fine for a long time. The guest has an httpd instance on port 80 that is robust and reliable. I shell into the guest with no issues. I use various host-based IDEs that use various sockets to communicate with the guest and all is fine.

I can't make neo4j work. I've been able to get exactly ONE request to produce a login screen -- that subsequently failed because the WS connection to bolt doesn't work. That's it. I attempted the config file change to make that work, and everything stopped working. I reverted it, and still everything is broken.

I've used systemctl to show that the neo4j service is up and running (it is). I've used firewall-cmd to show that ports 7474 and 7687 are open. They are. I've used netstat to confirm that the open ports have listeners. They do. I've confirmed the selinux is disabled. And yet it does nothing.

When I use curl from a shell connected to the guest, it appears that something is on 7474:

$ curl
  "bolt_routing" : "neo4j://",
  "transaction" : "{databaseName}/tx",
  "bolt_direct" : "bolt://",
  "neo4j_version" : "4.0.0",
  "neo4j_edition" : "community"

I haven't found any useful logs -- the "debug.log" in /var/log/neo4j has lots of useless startup and shutdown information and nothing of interest. The journalctl command is similarly unhelpful:

$journalctl -e -u neo4j
-- Logs begin at Wed 2020-02-26 11:56:58 EST, end at Wed 2020-02-26 12:25:09 EST. --
Feb 26 11:57:24 localhost.localdomain systemd[1]: Started Neo4j Graph Database.
Feb 26 11:57:28 localhost.localdomain neo4j[1381]: Directories in use:
Feb 26 11:57:28 localhost.localdomain neo4j[1381]: home:         /var/lib/neo4j
Feb 26 11:57:28 localhost.localdomain neo4j[1381]: config:       /etc/neo4j
Feb 26 11:57:28 localhost.localdomain neo4j[1381]: logs:         /var/log/neo4j
Feb 26 11:57:28 localhost.localdomain neo4j[1381]: plugins:      /var/lib/neo4j/plugins
Feb 26 11:57:28 localhost.localdomain neo4j[1381]: import:       /var/lib/neo4j/import
Feb 26 11:57:28 localhost.localdomain neo4j[1381]: data:         /var/lib/neo4j/data
Feb 26 11:57:28 localhost.localdomain neo4j[1381]: certificates: /var/lib/neo4j/certificates
Feb 26 11:57:28 localhost.localdomain neo4j[1381]: run:          /var/run/neo4j
Feb 26 11:57:28 localhost.localdomain neo4j[1381]: Starting Neo4j.
Feb 26 11:57:38 localhost.localdomain neo4j[1381]: 2020-02-26 16:57:38.701+0000 WARN  Use of deprecated setting dbms.connectors.default_listen_address. It is replaced by dbms.default_listen_
Feb 26 11:57:38 localhost.localdomain neo4j[1381]: 2020-02-26 16:57:38.704+0000 WARN  Use of deprecated setting dbms.directories.certificates. Legacy ssl policy is no longer supported.
Feb 26 11:57:38 localhost.localdomain neo4j[1381]: 2020-02-26 16:57:38.786+0000 INFO  ======== Neo4j 4.0.0 ========
Feb 26 11:57:38 localhost.localdomain neo4j[1381]: 2020-02-26 16:57:38.791+0000 INFO  Starting...
Feb 26 11:58:05 localhost.localdomain neo4j[1381]: 2020-02-26 16:58:05.606+0000 INFO  Bolt enabled on
Feb 26 11:58:05 localhost.localdomain neo4j[1381]: 2020-02-26 16:58:05.607+0000 INFO  Started.
Feb 26 11:58:11 localhost.localdomain neo4j[1381]: 2020-02-26 16:58:11.455+0000 INFO  Remote interface available at http://localhost:7474/

Is it possible for somebody to provide a simple and straightforward recipe for performing a straightforward hello-world roundtrip to demonstrate that neo4j is installed and running on a local guest VM? If it can be done with Apache, the various IDEs, nodejs, react, angular, mongodb, and mysql, is it so very hard to do with neo4j?

@tms you're using 4.0 -- and I think you're running into an issue discussed / solved here: Cypher-shell certificates - #2 by david.allen

Is it easier to use an earlier version of neo4j?

I'm trying very hard to avoid getting sucked down the rathole of certs, domain names, and all that. I get that I need all that for a production server facing the jungle.

I'm running a guest VM on my own physical computer with several firewalls between it and the outside world. None of these communications require any complex authentication technology.

I appreciate the quick response, I'm not trying to bust your chops. I'm just looking for the quickest dirtiest path to doing some local experimentation with neo4j. I may even just punt this altogether and try connecting from running python code using the python drivers.

@tms the absolute quickest and dirtiest way to try Neo4j is, nothing to install or use.

After that, if you want to set it up yourself, I'd ask you to consider using one of the Cloud Marketplace launches. After that, if you don't want a cluster, you should consider something like this: How to Automate Neo4j Deploys on Google Cloud Platform (GCP) | by David Allen | Neo4j Developer Blog | Medium (there's an equivalent article for AWS and Azure if you're there). Most of those options (as of this writing) will be 3.5 series.

Setting up and configuring everything yourself, on a VM of your own creation, using 4.0.0 - your best option is to follow the guidance I linked above.

Ok, got it.

I note the following from an elevated command prompt on the Windows 10 pro host:

<h1>Hello world</h1>

curl: (7) Failed to connect to port 7474: Connection refused

The first shows that the host is able to communicate with the guest on port 80. The second shows (I think) that the guest won't respond on port 7474.

I guess this is consistent with the certificate issue you describe. I guess I'll make and install a self-signed certificate -- I've done that before, but in a different VM and environment.

This turned out to be a configuration issue in neo4j.conf.

For some reason, the following line was commented out:


Not surprisingly, the server ignored 7474 while configured that way. I don't remember doing that, I wonder if perhaps the default distribution comes that way?

Anyway, once I turned it on the neo browser started working. I'm still at v4.0.0.0, by the way. I've left authentication turned off:

Both Chrome and Firefox seem to be doing just fine, neither is complaining.

@david_allen I have a single node neo4j:4.0.1-enterprise in a Kubernetes cluster. It is behind a load balancer and we are using Ingress to expose the browser and the bolt connection via the following configuration:

apiVersion: extensions/v1beta1
kind: Ingress
  name: alma-ingress
    - host:
          - path: /
              serviceName: neo4j
              servicePort: 7474
    - host:
          - path: /
              serviceName: neo4j
              servicePort: 7687

This type of configuration had worked fine for us on neo4j:3.5-enterprise, so we could connect to the browser just fine. We are using Amazon certs so SSL/TLS is legit and not a problem.

When we upgraded to 4.0 this broke. Our load balancer exposes only two ports: 80, 443. Our Ingress redirects all 80 to 443 and our cert is valid, and the load balancer terminates the TLS for us.

We have encryption turned off on the Neo4j server and we have HTTPS also turned off.

When I connect to the browser, I'll use the address like and the browser loads. For the bolt address, then I will use with user/pass.

What happens next is we do connect and get the 101 UPGRADE, and there are some websocket frames exchanged. The client issues the command dbms.routing.getRoutingTable it seems:

Screen Shot 2020-03-04 at 10.23.39 AM

But the server responds back with address and this is not routable, so the browser tries to connect to that and fails, and this repeats ad infinitum.

Screen Shot 2020-03-04 at 10.24.55 AM

I have tried to disable this with the settings:


as per Configuration settings - Operations Manual

I don't want the server to run causal clustering, but we want some of the other enterprise features. We want to run in single mode, and I'm unsure how to get the WebSocket connection back working again.

Can you please advise?

Thanks in advance,

@davisford there were a number of config changes in 4.0, and the site has a migration guide from 3.5 -> 4.0. Could you share your config, and show some logs from the pod, preferrably a debug.log dump?

If the server is responding back with that is indeed not routable -- and this suggests that the Neo4j pod is incorrectly configured with respect to its default_advertised_address. I don't remember this detail off hand but be sure to very carefully check your connector settings, as some configuration key names changed in 4.0. So if you copied the config you were using from 3.5, almost certainly that's your problem.

@david_allen attached is debug.log, also neo4j.conf and I also copy the file overrides.conf into the /conf dir, b/c the docs state that the server should pick up any other conf files and apply them as overrides. Not sure if it is working?

Let me know if you see anything. One thing of note, I see the docker boot shell script does some manipulation to the conf at startup. Something else is also adding these properties like:

PORT=tcp:// runtime. The server logs complain it doesn't understand these, but I'm not sure how/why they are getting added. They are not in the default config I am using...debug.log.txt (110.5 KB) neo4j.conf.txt (36.6 KB) overrides.conf.txt (1.5 KB)

@davisford grep your neo4j.conf for advertised_address, and I see your problem, it has a number of entries like this:


There's your advertisement right there (this also holds in your file for http/https). That isn't routable outside of kubernetes, so you should change that to whatever the externally valid/addressable address should be.

1 Like

Will that affect my internal k8s pods that use a service though? I define a k8s service with labels/selectors and that is how my pods find neo4j. If I fix that property to a DNS entry like, will the server reject requests from internal k8s IPs?

No. The advertised address is about how the server advertises to the world, it isn't about what connections it will accept. If you advertise an externally routable address, it will still accept connections from anywhere, subject to the network interface you bind to internally and your local firewall rules. For Neo4j in Kubernetes -- I really recommend having a look at this: Neo4j Considerations in Orchestration Environments | by David Allen | Neo4j Developer Blog | Medium

Thanks for that. We are using a storage orchestrator (STORK), thus we don't need a cluster. It replicates the data volumes for us and ensures hyper-convergence.

Hi @david_allen I'm still having a problem with this. I have tried to override these values with environment variables in the deployment / pod spec, but it seems like the docker sh script that is embedded in the container overrides my environment values.

Here's a look at the deployed pod spec with a few things redacted -- note the fqdn-here represents a real DNS fully qualified domain name that I've redacted here.

It is receiving some of my environment variables (e.g. I enable prometheus monitoring and those stick), but it just always seems to overwrite the advertised address to be no matter what I do.

apiVersion: v1
kind: Pod
  annotations: eks.privileged "2004" "true"
  creationTimestamp: "2020-03-31T22:08:59Z"
  generateName: neo4j-6d6585bcbf-
    app: neo4j
    pod-template-hash: 6d6585bcbf
  name: neo4j-6d6585bcbf-fl8pw
  namespace: alma
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: ReplicaSet
    name: neo4j-6d6585bcbf
    uid: 38aefaa9-739c-11ea-8fd4-0aa6c32e78f9
  resourceVersion: "43523590"
  selfLink: /api/v1/namespaces/alma/pods/neo4j-6d6585bcbf-fl8pw
  uid: 38afec27-739c-11ea-8fd4-0aa6c32e78f9
  - env:
      value: "yes"
    - name: NEO4J_AUTH
      value: neo4j/Salido4u-2.78
    - name: NEO4J_dbms_mode
      value: single
    - name: NEO4J_metrics_prometheus_enabled
      value: "true"
    - name: NEO4J_metrics_prometheus_endpoint
    - name: NEO4J_dbms_connectors_default_listen_address
    - name: NEO4J_dbms_logs_query_threshold
      value: 2s
    - name: NEO4J_dbms_logs_query_rotation_size
      value: 20m
    - name: NEO4J_dbms_logs_query_rotation_keep_number
      value: "7"
    - name: NEO4J_dbms_logs_query_time_logging_enabled
      value: "true"
    - name: NEO4J_dbms_logs_query_page_logging_enabled
      value: "true"
    - name: NEO4J_dbms_connector_bolt_address
      value: :7687
    - name: NEO4J_dbms_connector_https_advertised_address
      value: fqdn-here:7473
    - name: NEO4J_dbms_connector_http_advertised_address
      value: fqdn-here:7474
    - name: NEO4J_dbms_connector_bolt_advertised_address
      value: fqdn-here:7687
    image: neo4j:4.0.2-enterprise
    imagePullPolicy: IfNotPresent
    name: neo4j
    - containerPort: 7474
      name: browser
      protocol: TCP
    - containerPort: 7687
      name: bolt
      protocol: TCP
    - containerPort: 2004
      name: metrics
      protocol: TCP
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    - mountPath: /var/lib/neo4j/data/
      name: neo4jdata
    - mountPath: /var/run/secrets/
      name: default-token-2t5rf
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  nodeName: ip-192-168-174-9.ec2.internal
  nodeSelector: m4.large
  priority: 0
  restartPolicy: Always
  schedulerName: stork
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  - effect: NoExecute
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    operator: Exists
    tolerationSeconds: 300
  - name: neo4jdata
      claimName: px-neo4j-pvc
  - name: default-token-2t5rf
      defaultMode: 420
      secretName: default-token-2t5rf
  - lastProbeTime: null
    lastTransitionTime: "2020-03-31T22:08:59Z"
    status: "True"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2020-03-31T22:09:00Z"
    status: "True"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2020-03-31T22:09:00Z"
    status: "True"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2020-03-31T22:08:59Z"
    status: "True"
    type: PodScheduled
  - containerID: docker://1223f6779066aa0dab7c8c1f482d9f04584ceb623700594ca0095ef8e4a197fa
    image: neo4j:4.0.2-enterprise
    imageID: docker-pullable://neo4j@sha256:a090c2ed169a68bdbf7dd2f1e5b0c47891530d489dc7f5a5f43c8d719b5a32e4
    lastState: {}
    name: neo4j
    ready: true
    restartCount: 0
        startedAt: "2020-03-31T22:09:00Z"
  phase: Running
  qosClass: BestEffort
  startTime: "2020-03-31T22:08:59Z"

When I shell into the pod itself and cat logs/debug.log I can see it resets these addresses back to, and indeed when I try the WebSocket it again responds with address.

Here's a snipped from that log note the bolt advertised address is now reset again to -- what am I missing here?

2020-03-31 22:09:08.198+0000 INFO [o.n.i.d.DiagnosticsManager] --------------------------------------------------------------------------------
2020-03-31 22:09:08.198+0000 INFO [o.n.i.d.DiagnosticsManager]                                 [ DBMS config ]
2020-03-31 22:09:08.198+0000 INFO [o.n.i.d.DiagnosticsManager] --------------------------------------------------------------------------------
2020-03-31 22:09:08.200+0000 INFO [o.n.i.d.DiagnosticsManager] DBMS provided settings:
2020-03-31 22:09:08.209+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.discovery_advertised_address=neo4j-6d6585bcbf-fl8pw:5000
2020-03-31 22:09:08.209+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.discovery_listen_address=
2020-03-31 22:09:08.209+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.raft_advertised_address=neo4j-6d6585bcbf-fl8pw:7000
2020-03-31 22:09:08.210+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.raft_listen_address=
2020-03-31 22:09:08.210+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.transaction_advertised_address=neo4j-6d6585bcbf-fl8pw:6000
2020-03-31 22:09:08.210+0000 INFO [o.n.i.d.DiagnosticsManager] causal_clustering.transaction_listen_address=
2020-03-31 22:09:08.210+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.bolt.advertised_address=
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.bolt.enabled=true
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.http.advertised_address=
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.http.enabled=true
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.https.advertised_address=
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.connector.https.enabled=false
2020-03-31 22:09:08.211+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.default_listen_address=
2020-03-31 22:09:08.212+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.directories.import=/var/lib/neo4j/import
2020-03-31 22:09:08.212+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.directories.logs=/logs
2020-03-31 22:09:08.212+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.directories.neo4j_home=/var/lib/neo4j
2020-03-31 22:09:08.212+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.jvm.additional=-Djdk.tls.rejectClientInitiatedRenegotiation=true
2020-03-31 22:09:08.212+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.logs.query.rotation.size=20971520
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.logs.query.threshold=2s
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.memory.pagecache.size=512M
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.mode=SINGLE
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.tx_log.rotation.retention_policy=100M size
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] dbms.windows_service_name=neo4j
2020-03-31 22:09:08.213+0000 INFO [o.n.i.d.DiagnosticsManager] metrics.prometheus.enabled=true
2020-03-31 22:09:08.214+0000 INFO [o.n.i.d.DiagnosticsManager] metrics.prometheus.endpoint=
2020-03-31 22:09:08.214+0000 INFO [o.n.i.d.DiagnosticsManager]

Nevermind, I got it. I see I specified the environment variable incorrectly (need two underscores). I use a kustomize patch and this works now (I have Ingress that routes bolt / 443 to 7687 on the service:

- op: add
  path: /spec/template/spec/containers/0/env/-
    name: NEO4J_dbms_connector_bolt_advertised__address
    value: ""
1 Like

Dear nithin,

I want to change configuration of Neo4j DB but they are all under neo4j os user. AWS configuration provide ubuntu user to connect to EC instance.
So how can you restart the server as well as change the DB configuration?
Do you need to log in to EC2 instance using neo4j user?
Do you have neo4j password default?

Thanks in advanced