I have a new Neo4j ami installed on an Amazon EC2 linux instance.
I have SSL set up fine and can reach it :7473 with no issue.
I would like to use :443 and not have to specify a port in the browser.
the problem is if I use systemctl start neo4j i get an error saying that port is already in use...
2024-04-11 21:39:53.656+0000 INFO Starting...
2024-04-11 21:39:55.964+0000 INFO This instance is ServerId{8b75a2dd} (8b75a2dd-e98d-4842-93d3-a720014dbd54)
2024-04-11 21:39:57.469+0000 INFO ======== Neo4j 5.18.1 ========
2024-04-11 21:40:04.632+0000 INFO Bolt enabled on ip-10-10-71-100.ec2.internal:7687.
2024-04-11 21:40:04.658+0000 INFO Mounted unmanaged extension [com.neo4j.bloom.server] at [/bloom]
2024-04-11 21:40:04.659+0000 INFO Mounted unmanaged extension [semantics.extension] at [/rdf]
2024-04-11 21:40:05.010+0000 ERROR Failed to start Neo4j on 0.0.0.0:443: Address 0.0.0.0:443 is already in use, cannot bind to it.
But as far as I can tell nothing is using 443. It is a brand new install using a vanilla Neo4j ami image. The only other software I installed on it was certbot for a Let's Encrypt cert.
[root@ip-10-10-71-100 neo4j]# netstat -anltp | grep LISTEN
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 2254/sshd
tcp 0 0 127.0.0.1:38169 0.0.0.0:* LISTEN 5725/code-5c3e652f6
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 2193/master
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1747/rpcbind
tcp6 0 0 :::22 :::* LISTEN 2254/sshd
tcp6 0 0 :::111 :::* LISTEN 1747/rpcbind
Even stranger is if restart using neo4j-admin server start instead of systemctl it works fine. It starts up and I can access on :443. So while that works manually the database won't automatically boot on restart.
I see somebody posted this back in 2020. https://community.neo4j.com/t/failed-to-start-neo4j-instance-deployed-on-amazon-ec2/29271
Any tips or suggestions greatly appreciated.
-josh
@josh.blackwell
I can reproduced this with 5.14.1 and simply by changing
/etc/neo4j/neo4j.conf and
# HTTP Connector. There can be zero or one HTTP connectors.
server.http.enabled=true
server.http.listen_address=:80
server.http.advertised_address=:80
and then upon running systemctl neo4j start , running journalctl -u neo4j reports
Apr 12 01:18:27 ubuntu22dockerdebjava17 neo4j[1754]: 2024-04-12 01:18:27.337+0000 INFO Bolt enabled on localhost:7687.
Apr 12 01:18:27 ubuntu22dockerdebjava17 neo4j[1754]: 2024-04-12 01:18:27.343+0000 INFO Bolt (Routing) enabled on localhost:7688.
Apr 12 01:18:27 ubuntu22dockerdebjava17 neo4j[1754]: 2024-04-12 01:18:27.830+0000 ERROR Failed to start Neo4j on localhost:80: Address localhost:80 is alr>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: 2024-04-12 01:18:35.143+0000 ERROR Failed to start Neo4j on localhost:80.
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: java.lang.RuntimeException: Error starting Neo4j database server at /var/lib/neo4j/data/databases
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseMana>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceF>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at com.neo4j.server.enterprise.EnterpriseDBMSProvider.createManagementService(EnterpriseDBMSP>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at com.neo4j.server.enterprise.EnterpriseDBMSProvider.<init>(EnterpriseDBMSProvider.java:29) >
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at com.neo4j.server.enterprise.EnterpriseDBMSFactory.createManagementService(EnterpriseDBMSFa>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at com.neo4j.server.enterprise.EnterpriseBootstrapper.createNeo(EnterpriseBootstrapper.java:2>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:187) [neo4j-5.14.0.jar:5.14.0]
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:99) [neo4j-5.14.0.jar:5.14.0]
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at com.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:19) [neo4j>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'com.neo4j.server.enterprise.Ente>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:364) ~[neo>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:91) ~[neo4j-common-5.14.0.ja>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseMana>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: ... 8 more
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: Caused by: org.neo4j.server.ServerStartupException: Starting Neo4j failed: Address localhost:80 is al>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.neo4j.server.AbstractNeoWebServer.start(AbstractNeoWebServer.java:195) ~[neo4j-server->
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:347) ~[neo>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:91) ~[neo4j-common-5.14.0.ja>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseMana>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: ... 8 more
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: Caused by: org.neo4j.configuration.helpers.PortBindException: Address localhost:80 is already in use,>
Apr 12 01:18:35 ubuntu22dockerdebjava17 neo4j[1754]: at org.eclipse.jetty.server.ServerConnector.openAcceptChannel(ServerConnector.java:344) ~[jet
and the resolution and by resolution this far exceeds my linux skills but per linux - Allow non-root process to bind to port 80 and 443? - Super User
I then ran
sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/lib/jvm/zulu17/bin/java
and now systemctl start neo4j and corresponding journalctl -u neo4j reports
Apr 12 01:22:35 ubuntu22dockerdebjava17 neo4j[2069]: SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Apr 12 01:22:37 ubuntu22dockerdebjava17 neo4j[2069]: 2024-04-12 01:22:37.209+0000 INFO ======== Neo4j 5.14.0 ========
Apr 12 01:22:37 ubuntu22dockerdebjava17 neo4j[2069]: 2024-04-12 01:22:37.235+0000 INFO This instance is ServerId{8afc1cea} (8afc1cea-c229-4eb2-b140-4eb54>
Apr 12 01:22:44 ubuntu22dockerdebjava17 neo4j[2069]: 2024-04-12 01:22:44.085+0000 INFO Sending metrics to CSV file at /var/lib/neo4j/metrics
Apr 12 01:22:44 ubuntu22dockerdebjava17 neo4j[2069]: 2024-04-12 01:22:44.123+0000 INFO Bolt enabled on localhost:7687.
Apr 12 01:22:44 ubuntu22dockerdebjava17 neo4j[2069]: 2024-04-12 01:22:44.156+0000 INFO Bolt (Routing) enabled on localhost:7688.
Apr 12 01:22:45 ubuntu22dockerdebjava17 neo4j[2069]: 2024-04-12 01:22:45.532+0000 INFO HTTP enabled on localhost:80.
Apr 12 01:22:45 ubuntu22dockerdebjava17 neo4j[2069]: 2024-04-12 01:22:45.534+0000 INFO Remote interface available at http://localhost/
Apr 12 01:22:45 ubuntu22dockerdebjava17 neo4j[2069]: 2024-04-12 01:22:45.539+0000 INFO id: 2F5C7D0C66B48ADF2E5A66CED84E2288A3B019AFFA0447DEA6C978B8F5C272>
Apr 12 01:22:45 ubuntu22dockerdebjava17 neo4j[2069]: 2024-04-12 01:22:45.540+0000 INFO name: system
Apr 12 01:22:45 ubuntu22dockerdebjava17 neo4j[2069]: 2024-04-12 01:22:45.540+0000 INFO creationDate: 2023-11-30T17:38:09.3Z
Apr 12 01:22:45 ubuntu22dockerdebjava17 neo4j[2069]: 2024-04-12 01:22:45.542+0000 INFO Started.
and thus success.
I will caution ( and cant stress this enough) , I'm no linux 
I'm no linux SA. Did said sudo setcap CAP_NET_BIND_SERVICE=+eip /usr/lib/jvm/zulu17/bin/java open the door elsewhere and cause issue?
linux - Allow non-root process to bind to port 80 and 443? - Super User provides more commentary on whether this is a good / bad thing etc but in short the failure you encounter does not appear be native to Neo but more so native to Linux and its implementation specifics
Ha! thank you @dana_canzano that was a good lead.
Given the error message saying it was a port in use issue, it didn't occur to me it was a permissions issue. That explains why I could start it using neo4j-admin server start as I am running it as root but systemctl is using neo4j user.
Not sure what the right solution is but at least I know what is going on. Thanks!
[root@ip-10-10-71-100 systemd]# cat /usr/lib/systemd/system/neo4j.service
[Unit]
Description=Neo4j Graph Database
After=network-online.target
Wants=network-online.target
[Service]
ExecStart=/usr/share/neo4j/bin/neo4j console
Restart=on-abnormal
User=neo4j
Group=neo4j
Environment="NEO4J_CONF=/etc/neo4j" "NEO4J_HOME=/var/lib/neo4j"
LimitNOFILE=60000
TimeoutSec=120
[Install]
WantedBy=multi-user.target
BTW: that error is coming from Neo4j so if you can relay that to the Eng team and get them to distinguish between a permission error and a port in-use issue might be helpful to others.
org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.server.CommunityNeoWebServer@3a38f122' was successfully initialized, but failed to start. Please see the attached cause exception "Address 0.0.0.0:443 is already in use, cannot bind to it.".
@josh.blackwell
i will relay to engineering.
in a weird way had it had reported Permission denied on port ### my 1st reaction would have been ... oh you can set permissions on a port?
i guess either way its still a 'binding' issue