First time Cluster in AWS : Connection Refused

mike_r_black · August 25, 2019, 4:51am

I'm trying to set up my first cluster in AWS. I have 3 EC2 instances. Following the manual and it appears as if the cluster is established, but when I try to cypher-shell from one of the instances I get a Connection Refused. Any tips where to go about looking to try and figure out how to get this cluster going? I'll paste some what I've found that I think could be relevant. Some AWS setup notes:

Each EC2 instance is in the same VPC and Security group
All traffic is allowed within the Security Group so all ports between EC2s for the cluster to communicate should be allowed
Cluster definitions are referencing the private IPv4 addresses

Service Status of Node 1

ubuntu@ip-172-31-24-72:~$ sudo service neo4j status
● neo4j.service - Neo4j Graph Database
   Loaded: loaded (/lib/systemd/system/neo4j.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-08-25 04:03:07 UTC; 4min 29s ago
 Main PID: 6189 (java)
    Tasks: 53 (limit: 1152)
   CGroup: /system.slice/neo4j.service
           └─6189 /usr/bin/java -cp /var/lib/neo4j/plugins:/etc/neo4j:/usr/share/neo4j/lib/*:/var/lib/neo4j/plugins/* -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFina

Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]: 2019-08-25 04:03:12.972+0000 INFO  My connection info: [
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]:         Discovery:   listen=0.0.0.0:5000, advertised=localhost:5000,
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]:         Transaction: listen=0.0.0.0:6000, advertised=localhost:6000,
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]:         Raft:        listen=0.0.0.0:7000, advertised=localhost:7000,
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]:         Client Connector Addresses: bolt://localhost:7687,http://localhost:7474,https://localhost:7473
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]: ]
Aug 25 04:03:12 ip-172-31-24-72 neo4j[6189]: 2019-08-25 04:03:12.976+0000 INFO  Discovering other core members in initial members set: [172.31.23.197:5000, 172.31.29.208:5000]
Aug 25 04:03:44 ip-172-31-24-72 neo4j[6189]: 2019-08-25 04:03:44.297+0000 INFO  Bound to cluster with id 13a49e26-3218-4a54-9974-0662e7e530e1
Aug 25 04:03:44 ip-172-31-24-72 neo4j[6189]: 2019-08-25 04:03:44.303+0000 INFO  Discovered core member at 172.31.29.208:5000
Aug 25 04:03:49 ip-172-31-24-72 neo4j[6189]: 2019-08-25 04:03:49.212+0000 INFO  Discovered core member at 172.31.23.197:5000

Service Status Node 2

ubuntu@ip-172-31-29-208:~$ sudo service neo4j status
● neo4j.service - Neo4j Graph Database
   Loaded: loaded (/lib/systemd/system/neo4j.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-08-25 04:03:33 UTC; 34min ago
 Main PID: 2667 (java)
    Tasks: 53 (limit: 1152)
   CGroup: /system.slice/neo4j.service
           └─2667 /usr/bin/java -cp /var/lib/neo4j/plugins:/etc/neo4j:/usr/share/neo4j/lib/*:/var/lib/neo4j/plugins/* -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFina

Aug 25 04:03:38 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:03:38.658+0000 INFO  Discovering other core members in initial members set: [172.31.23.197:5000, 172.31.24.72:5000]
Aug 25 04:03:48 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:03:48.572+0000 INFO  Bound to cluster with id 13a49e26-3218-4a54-9974-0662e7e530e1
Aug 25 04:03:48 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:03:48.635+0000 INFO  Discovered core member at 172.31.24.72:5000
Aug 25 04:03:49 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:03:49.211+0000 INFO  Discovered core member at 172.31.23.197:5000
Aug 25 04:14:04 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:14:04.353+0000 WARN  Lost core member at 172.31.23.197:5000
Aug 25 04:14:14 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:14:14.647+0000 INFO  Discovered core member at 172.31.23.197:5000
Aug 25 04:24:29 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:24:29.167+0000 WARN  Lost core member at 172.31.23.197:5000
Aug 25 04:24:39 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:24:39.741+0000 INFO  Discovered core member at 172.31.23.197:5000
Aug 25 04:34:53 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:34:53.722+0000 WARN  Lost core member at 172.31.23.197:5000
Aug 25 04:35:04 ip-172-31-29-208 neo4j[2667]: 2019-08-25 04:35:04.235+0000 INFO  Discovered core member at 172.31.23.197:5000

Service Status Node 3

ubuntu@ip-172-31-23-197:~$ sudo service neo4j status
● neo4j.service - Neo4j Graph Database
   Loaded: loaded (/lib/systemd/system/neo4j.service; disabled; vendor preset: enabled)
   Active: active (running) since Sun 2019-08-25 04:34:53 UTC; 3min 45s ago
 Main PID: 4062 (java)
    Tasks: 68 (limit: 1152)
   CGroup: /system.slice/neo4j.service
           └─4062 /usr/bin/java -cp /var/lib/neo4j/plugins:/etc/neo4j:/usr/share/neo4j/lib/*:/var/lib/neo4j/plugins/* -server -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFina

Aug 25 04:35:09 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:35:09.419+0000 INFO  Bound to cluster with id 13a49e26-3218-4a54-9974-0662e7e530e1
Aug 25 04:35:09 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:35:09.479+0000 INFO  Discovered core member at 172.31.24.72:5000
Aug 25 04:35:09 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:35:09.496+0000 INFO  Discovered core member at 172.31.29.208:5000
Aug 25 04:35:38 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:35:38.954+0000 INFO  Waiting to hear from leader...
Aug 25 04:36:06 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:36:06.955+0000 INFO  Waiting to hear from leader...
Aug 25 04:36:34 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:36:34.956+0000 INFO  Waiting to hear from leader...
Aug 25 04:37:02 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:37:02.957+0000 INFO  Waiting to hear from leader...
Aug 25 04:37:30 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:37:30.959+0000 INFO  Waiting to hear from leader...
Aug 25 04:37:58 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:37:58.960+0000 INFO  Waiting to hear from leader...
Aug 25 04:38:26 ip-172-31-23-197 neo4j[4062]: 2019-08-25 04:38:26.961+0000 INFO  Waiting to hear from leader...

I don't know why the 3rd node is waiting to hear from the leader?

stefan.armbruster · August 25, 2019, 6:51am

I guess the advertised address settings are missing. Amend to your neo4j.conf file:

dbms.connectors.default_advertised_address=<internal_ip_address_of_this_host>

Note: if your clients (e.g. neo4j browser) are sitting outside on the public interface you might add as well:

dbms.connector.bolt.advertised_address=<external_ip_of_this_host>
dbms.connector.http.advertised_address=<external_ip_of_this_host>
dbms.connector.https.advertised_address=<external_ip_of_this_host>

mike_r_black · August 25, 2019, 6:46pm

Thanks! That solved it!

Topic		Replies	Views
Neo4j on EC2: "Could not connect with the "neo4j://" scheme to this Neoj server." Server browser	3	996	August 22, 2020
Remote connection to a AWS ECS EC2 Neo4j docker installation General migrated	0	133	November 23, 2022
Need help with AWS Cluster SSL setup Cluster	0	90	December 21, 2023
Errors trying to use AWS images of an EC2 instance using Neo4j in Docker Operations aws , ec2 , docker	2	758	May 2, 2020
Neo4j refuses connection after delete Neo4j Graph Platform operations	2	510	February 5, 2021

First time Cluster in AWS : Connection Refused

Service Status of Node 1

Service Status Node 2

Service Status Node 3

Related Topics