Unable to get Docker + HTTPS working

Neo4j Graph Platform
1

Hi,

We want to use docker to run neo4j 4.1.1 and have SSL enabled. However, neo4j will not run with these settings. The specific error we get is:

ERROR Failed to start Neo4j on dbms.connector.http.listen_address, a socket address. If missing port or hostname it is acquired from dbms.default_listen_address. Error starting Neo4j database server at /data/databases

The following command is used to spin up the docker container (the image comes from docker hub):

#!/bin/bash
groups=( $( id --real --groups neo4j ) )
docker run \
        --interactive \
        --detach \
        --name=neo4j411 \
        --publish=7687:7687 \
        --publish=7473:7473 \
        --publish=7474:7474 \
        --volume=/appl/neo4j/conf:/conf \
        --volume=/appl/neo4j/plugins:/plugins \
        --volume=/appl/neo4j/logs:/logs \
        --volume=/appl/neo4j/data:/data \
        --volume=/appl/mnt/:/import \
        --volume=/appl/neo4j/export:/export \
        --volume=/appl/neo4j/ssl:/ssl \
        --user="$(id -u neo4j):$(id -g neo4j)" \
        --group-add=$groups \
        --env TZ='Europe/Amsterdam' \
        --env NEO4J_dbms_security_procedures_unrestricted=apoc.*,gds.* \
        --env NEO4J_dbms_security_procedures_whitelist=gds.*,apoc.* \
        --env NEO4J_dbms_memory_heap_initial__size=31G \
        --env NEO4J_dbms_memory_heap_max__size=31G \
        --env NEO4J_dbms_memory_pagecache_size=135G \
        --env NEO4J_dbms_ssl_policy_bolt_enabled=true \
        --env NEO4J_dbms_ssl_policy_bolt_base__directory=/ssl/bolt \
        --env NEO4J_dbms_ssl_policy_bolt_private__key=private.key \
        --env NEO4J_dbms_ssl_policy_bolt_public__certificate=certificate.csr \
        --env NEO4J_dbms_ssl_policy_bolt_client__auth=NONE \
        --env NEO4J_dbms_ssl_policy_https_enabled=true \
        --env NEO4J_dbms_ssl_policy_https_base__directory=/ssl/https \
        --env NEO4J_dbms_ssl_policy_https_private__key=private.key \
        --env NEO4J_dbms_ssl_policy_https_public__certificate=certificate.csr \
        --env NEO4J_dbms_ssl_policy_https_client__auth=NONE \
        --env NEO4J_dbms_connector_http_enabled=true \
        --env NEO4J_dbms_connector_https_enabled=true \
        --env NEO4J_dbms_connector_bolt_tls__level=REQUIRED \
        neo4j:latest /bin/bash

Which results in the following log entries in neo4j.log:

2020-11-27 14:13:19.274+0000 INFO  Starting...
2020-11-27 14:13:21.817+0000 INFO  ======== Neo4j 4.1.1 ========
2020-11-27 14:13:31.478+0000 ERROR Failed to start Neo4j on dbms.connector.http.listen_address, a socket address. If missing port or hostname it is acquired from dbms.default_listen_address. Error starting Neo4j database server at /data/databases
java.lang.RuntimeException: Error starting Neo4j database server at /data/databases
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198)
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:158)
        at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:36)
        at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:117)
        at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:87)
        at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.procedure.impl.GlobalProceduresRegistry@536d97f8' was successfully initialized, but failed to start. Please see the attached cause exception "Unable to set up injection for procedure `GraphGenerateProc`, the field `tracker` has type `interface org.neo4j.graphalgo.core.utils.mem.AllocationTracker` which is not a known injectable component.".
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:463)
        at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110)
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:189)
        ... 5 more
Caused by: org.neo4j.kernel.api.exceptions.ComponentInjectionException: Unable to set up injection for procedure `GraphGenerateProc`, the field `tracker` has type `interface org.neo4j.graphalgo.core.utils.mem.AllocationTracker` which is not a known injectable component.
        at org.neo4j.procedure.impl.FieldInjections.createInjector(FieldInjections.java:98)
        at org.neo4j.procedure.impl.FieldInjections.setters(FieldInjections.java:81)
        at org.neo4j.procedure.impl.ProcedureCompiler.compileProcedure(ProcedureCompiler.java:264)
        at org.neo4j.procedure.impl.ProcedureCompiler.compileProcedure(ProcedureCompiler.java:226)
        at org.neo4j.procedure.impl.ProcedureJarLoader.loadProcedures(ProcedureJarLoader.java:114)
        at org.neo4j.procedure.impl.ProcedureJarLoader.loadProceduresFromDir(ProcedureJarLoader.java:85)
        at org.neo4j.procedure.impl.GlobalProceduresRegistry.start(GlobalProceduresRegistry.java:342)
        at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442)
        ... 7 more
2020-11-27 14:13:31.479+0000 INFO  Neo4j Server shutdown initiated by request

Without SSL, using the following docker command, we are able to get everything up and running:

docker run \
        --interactive \
        --detach \
        --name=neo4j411 \
        --publish=7687:7687 \
        --publish=7473:7473 \
        --publish=7474:7474 \
        --volume=/appl/neo4j/conf:/conf \
        --volume=/appl/neo4j/plugins:/plugins \
        --volume=/appl/neo4j/logs:/logs \
        --volume=/appl/neo4j/data:/data \
        --volume=/appl/mnt/:/import \
        --volume=/appl/neo4j/export:/export \
        --volume=/appl/neo4j/ssl:/ssl \
        --user="$(id -u neo4j):$(id -g neo4j)" \
        --group-add=$groups \
        --env TZ='Europe/Amsterdam' \
        --env NEO4J_dbms_security_procedures_unrestricted=apoc.*,gds.* \
        --env NEO4J_dbms_security_procedures_whitelist=gds.*,apoc.* \
        --env NEO4J_dbms_memory_heap_initial__size=31G \
        --env NEO4J_dbms_memory_heap_max__size=31G \
        --env NEO4J_dbms_memory_pagecache_size=135G \
        neo4j:latest /bin/bash

Your help is much appreciated. Thanks!

2

Hi @michael.the

I see your question is open for too long, sorry about that.
Welcome to the Neo4j Community (say hi to Mario for me ;) )

I haven't experienced this before, looking at your stacktrace my first try would be starting without all the plugins. The stack points to the GraphGenerateProc as cause of this issue.

Can you please check this:

groups=( $( id --real --groups neo4j ) )
docker run \
        --interactive \
        --detach \
        --name=neo4j411 \
        --publish=7687:7687 \
        --publish=7473:7473 \
        --publish=7474:7474 \
        --volume=/appl/neo4j/conf:/conf \
        --volume=/appl/neo4j/logs:/logs \
        --volume=/appl/neo4j/data:/data \
        --volume=/appl/mnt/:/import \
        --volume=/appl/neo4j/export:/export \
        --volume=/appl/neo4j/ssl:/ssl \
        --user="$(id -u neo4j):$(id -g neo4j)" \
        --group-add=$groups \
        --env TZ='Europe/Amsterdam' \
        --env NEO4J_dbms_memory_heap_initial__size=31G \
        --env NEO4J_dbms_memory_heap_max__size=31G \
        --env NEO4J_dbms_memory_pagecache_size=135G \
        --env NEO4J_dbms_ssl_policy_bolt_enabled=true \
        --env NEO4J_dbms_ssl_policy_bolt_base__directory=/ssl/bolt \
        --env NEO4J_dbms_ssl_policy_bolt_private__key=private.key \
        --env NEO4J_dbms_ssl_policy_bolt_public__certificate=certificate.csr \
        --env NEO4J_dbms_ssl_policy_bolt_client__auth=NONE \
        --env NEO4J_dbms_ssl_policy_https_enabled=true \
        --env NEO4J_dbms_ssl_policy_https_base__directory=/ssl/https \
        --env NEO4J_dbms_ssl_policy_https_private__key=private.key \
        --env NEO4J_dbms_ssl_policy_https_public__certificate=certificate.csr \
        --env NEO4J_dbms_ssl_policy_https_client__auth=NONE \
        --env NEO4J_dbms_connector_http_enabled=true \
        --env NEO4J_dbms_connector_https_enabled=true \
        --env NEO4J_dbms_connector_bolt_tls__level=REQUIRED \
        neo4j:latest /bin/bash

Cheers,
Ron

3

It is unbelievable but moving the 2 plugins from the head of the --env list in the command to the tail, it worked out!!!

#!/bin/bash
groups=( $( id --real --groups neo4j ) )
docker run \
        --interactive \
        --detach \
        --name=neo4j411 \
        --publish=7687:7687 \
        --publish=7473:7473 \
        --publish=7474:7474 \
        --volume=/appl/neo4j/conf:/conf \
        --volume=/appl/neo4j/plugins:/plugins \
        --volume=/appl/neo4j/logs:/logs \
        --volume=/appl/neo4j/data:/data \
        --volume=/appl/mnt/:/import \
        --volume=/appl/neo4j/export:/export \
        --volume=/appl/neo4j/ssl:/ssl \
        --user="$(id -u neo4j):$(id -g neo4j)" \
        --group-add=$groups \
        --env TZ='Europe/Amsterdam' \
        --env NEO4J_dbms_memory_heap_initial__size=31G \
        --env NEO4J_dbms_memory_heap_max__size=31G \
        --env NEO4J_dbms_memory_pagecache_size=135G \
        --env NEO4J_dbms_ssl_policy_bolt_enabled=true \
        --env NEO4J_dbms_ssl_policy_bolt_base__directory=/ssl/bolt \
        --env NEO4J_dbms_ssl_policy_bolt_private__key=private.key \
        --env NEO4J_dbms_ssl_policy_bolt_public__certificate=certificate.csr \
        --env NEO4J_dbms_ssl_policy_bolt_client__auth=NONE \
        --env NEO4J_dbms_ssl_policy_https_enabled=true \
        --env NEO4J_dbms_ssl_policy_https_base__directory=/ssl/https \
        --env NEO4J_dbms_ssl_policy_https_private__key=private.key \
        --env NEO4J_dbms_ssl_policy_https_public__certificate=certificate.csr \
        --env NEO4J_dbms_ssl_policy_https_client__auth=NONE \
        --env NEO4J_dbms_connector_http_enabled=true \
        --env NEO4J_dbms_connector_https_enabled=true \
        --env NEO4J_dbms_connector_bolt_tls__level=REQUIRED \
        --env NEO4J_dbms_security_procedures_unrestricted=apoc.*,gds.* \
        --env NEO4J_dbms_security_procedures_whitelist=gds.*,apoc.* \
         neo4j:latest /bin/bash
1 Like
4

Good to hear that Mario, although I cannot think of a reason why that would make a difference.

5

@MarioLiebreks @michael.the I'm trying to use your docker run command and it works nicely for the browser, but when I get inside the container and try to invoke cypher-shell command, it doesn't work :(
[root@6d1516546a82 neo4j]# cypher-shell
Connection to the database terminated. Please ensure that your database is listening on the correct host and port and that you have compatible encryption settings both on Neo4j server and driver. Note that the default encryption setting has changed in Neo4j 4.0.
[root@6d1516546a82 neo4j]# cypher-shell --encryption=true
Failed to establish secured connection with the server
Error message in debug.log is
2021-09-01 10:59:34.009+0000 ERROR [o.n.b.t.TransportSelectionHandler] Fatal error occurred when initialising pipeline: [id: 0xdbb6e8bd, L:/127.0.0.1:7687 ! R:/127.0.0.1:57400]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: certificate_unknown

Did you guys managed to get it working?

6

There's a helpful note about this error on this blog post here Getting Certificates for Neo4j with LetsEncrypt | by David Allen | Neo4j Developer Blog | Medium

Basically, the error is telling you that the cypher-shell command failed SSL validation.
This is because cypher-shell will default to neo4j://localhost:7687 when --address is not supplied.
To use cypher-shell after enabling SSL, you have to do "cypher-shell -u neo4j -a neo4j+s://{dns-from-your-certification}:7687.

Hope this help some other people in the future :slight_smile: