Cypher-shell certificates

mike_blum_neo4j · February 13, 2020, 7:57pm

With Bolt configured like this cypher-shell with Neo4j 4.0 seems to not like the self-signed certificates anymore:

./bin/cypher-shell --database system -u neo4j -p $secret --encryption true --debug
org.neo4j.driver.exceptions.SecurityException: Failed to establish secured connection with the server
	at org.neo4j.driver.internal.util.Futures.blockingGet(Futures.java:143)
	at org.neo4j.driver.internal.InternalSession.run(InternalSession.java:69)
	at org.neo4j.driver.internal.InternalSession.run(InternalSession.java:51)
	at org.neo4j.driver.internal.AbstractQueryRunner.run(AbstractQueryRunner.java:37)
	at org.neo4j.driver.internal.AbstractQueryRunner.run(AbstractQueryRunner.java:55)
	at org.neo4j.shell.state.BoltStateHandler.reconnect(BoltStateHandler.java:196)
	at org.neo4j.shell.state.BoltStateHandler.reconnect(BoltStateHandler.java:173)
	at org.neo4j.shell.state.BoltStateHandler.connect(BoltStateHandler.java:161)
	at org.neo4j.shell.CypherShell.connect(CypherShell.java:143)
	at org.neo4j.shell.Main.connectMaybeInteractively(Main.java:129)
	at org.neo4j.shell.Main.startShell(Main.java:92)
	at org.neo4j.shell.Main.main(Main.java:41)
	Suppressed: org.neo4j.driver.internal.util.ErrorUtil$InternalExceptionCause
		at org.neo4j.driver.internal.async.connection.HandshakeHandler.transformError(HandshakeHandler.java:195)
		at org.neo4j.driver.internal.async.connection.HandshakeHandler.exceptionCaught(HandshakeHandler.java:96)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:276)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:268)
		at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.exceptionCaught(SslHandler.java:1095)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:297)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.notifyHandlerException(AbstractChannelHandlerContext.java:831)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:376)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at org.neo4j.driver.internal.shaded.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
		at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
		at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
		at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
		at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
		at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
		at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
		at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
		at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:131)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:320)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:263)
	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:258)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:645)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.onCertificate(CertificateMessage.java:464)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.consume(CertificateMessage.java:360)
	at java.base/sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:392)
	at java.base/sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:443)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1061)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1048)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at java.base/sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:995)
	at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1499)
	at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1513)
	at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1397)
	at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1224)
	at org.neo4j.driver.internal.shaded.io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1271)
	at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:505)
	at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:444)
	at org.neo4j.driver.internal.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:283)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at org.neo4j.driver.internal.shaded.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352)
	at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1422)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374)
	at org.neo4j.driver.internal.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360)
	at org.neo4j.driver.internal.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:931)
	at org.neo4j.driver.internal.shaded.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
	at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:700)
	at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:635)
	at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:552)
	at org.neo4j.driver.internal.shaded.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:514)
	at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$6.run(SingleThreadEventExecutor.java:1044)
	at org.neo4j.driver.internal.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at org.neo4j.driver.internal.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:439)
	at java.base/sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:306)
	at java.base/sun.security.validator.Validator.validate(Validator.java:264)
	at java.base/sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:313)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:276)
	at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:141)
	at java.base/sun.security.ssl.CertificateMessage$T12CertificateConsumer.checkServerCerts(CertificateMessage.java:623)
	... 36 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
	at java.base/sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
	at java.base/java.security.cert.CertPathBuilder.build(CertPathBuilder.java:297)
	at java.base/sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:434)
	... 42 more

Here are the relevant sections from the neo4j.conf:

# Bolt connector
dbms.connector.bolt.enabled=true
dbms.connector.bolt.tls_level=REQUIRED
dbms.connector.bolt.listen_address=0.0.0.0:7687
dbms.connector.bolt.advertised_address=10.132.13.45:7687

# SSL settings (dbms.ssl.policy.<scope>.*)
#  .base_directory       Base directory for SSL policies paths. All relative paths within the
#                        SSL configuration will be resolved from the base dir.
#
#  .private_key          A path to the key file relative to the '.base_directory'.
#
#  .private_key_password The password for the private key.
#
#  .public_certificate   A path to the public certificate file relative to the '.base_directory'.
#
#  .trusted_dir          A path to a directory containing trusted certificates.
#
#  .revoked_dir          Path to the directory with Certificate Revocation Lists (CRLs).
#
#  .verify_hostname      If true, the server will verify the hostname that the client uses to connect with. In order
#                        for this to work, the server public certificate must have a valid CN and/or matching
#                        Subject Alternative Names.
#
#  .client_auth          How the client should be authorized. Possible values are: 'none', 'optional', 'require'.
#
#  .tls_versions         A comma-separated list of allowed TLS versions. By default only TLSv1.2 is allowed.
#
#  .trust_all            Setting this to 'true' will ignore the trust truststore, trusting all clients and servers.
#                        Use of this mode is discouraged. It would offer encryption but no security.
#
#  .ciphers              A comma-separated list of allowed ciphers. The default ciphers are the defaults of
#                        the JVM platform.

# Bolt SSL configuration
dbms.ssl.policy.bolt.enabled=true
dbms.ssl.policy.bolt.base_directory=/etc/certs
dbms.ssl.policy.bolt.private_key=private.key
dbms.ssl.policy.bolt.public_certificate=public.crt

# Https SSL configuration
dbms.ssl.policy.https.enabled=true
dbms.ssl.policy.https.base_directory=/etc/certs
dbms.ssl.policy.https.private_key=private.key
dbms.ssl.policy.https.public_certificate=public.crt

# Cluster SSL configuration
dbms.ssl.policy.cluster.enabled=true
dbms.ssl.policy.cluster.base_directory=/etc/certs
dbms.ssl.policy.cluster.private_key=private.key
dbms.ssl.policy.cluster.public_certificate=public.crt

# Backup SSL configuration
dbms.ssl.policy.backup.enabled=true
dbms.ssl.policy.backup.base_directory=/etc/certs
dbms.ssl.policy.backup.private_key=private.key
dbms.ssl.policy.backup.public_certificate=public.crt

Looking at the cypher-shell docs (this is on the same box as the database) there doens't appear to be an easy way to tell cypher-shell about the certificate. Is there a way to enable encrypted Bolt connections without needing to have all clients (cypher-shell, java apps, etc) needing to know the certificate via JKS or other PKI implementations?

david_allen · February 14, 2020, 12:08pm

Due to a change in 4.0, the server doesn't come packaged with a self-signed cert anymore, so the reason this isn't working is because you don't have a cert at all on the server side.

There's a PR open right now to add the ability to generate these certs for 4.0 in docker. I'm pointing you to this just because it has a bit of shell code showing you how you can generate your own.

mike_blum_neo4j · February 14, 2020, 3:22pm

Interesting! Thanks for the bash script. Right now I have this:

# Bolt SSL configuration
dbms.ssl.policy.bolt.enabled=true
dbms.ssl.policy.bolt.base_directory=/etc/certs
dbms.ssl.policy.bolt.private_key=private.key
dbms.ssl.policy.bolt.public_certificate=public.crt

And here is the directory structure for my self-signed cert that i generated and loaded onto the box:

ls -la /etc/certs/
total 24
drwxr-xr-x  4 neo4j neo4j 4096 Feb 13 22:21 .
drwxr-xr-x 98 root  root  4096 Feb 13 23:22 ..
-rw-r--r--  1 neo4j neo4j 1257 Feb 13 22:21 fullchain.pem
lrwxrwxrwx  1 root  root    22 Feb 13 22:21 private.key -> /etc/certs/privkey.pem
-rw-r--r--  1 neo4j neo4j 1704 Feb 13 22:21 privkey.pem
lrwxrwxrwx  1 root  root    24 Feb 13 22:21 public.crt -> /etc/certs/fullchain.pem
drwxr-xr-x  2 neo4j neo4j 4096 Feb 13 22:21 revoked
drwxr-xr-x  2 neo4j neo4j 4096 Feb 13 22:21 trusted

ls -lar /etc/certs/trusted/
total 8
lrwxrwxrwx 1 root  root    24 Feb 13 22:21 public.crt -> /etc/certs/fullchain.pem

I can load the browser at https://addr:7473 and login from there after accepting the 7473 and 7687 danger notices in firefox. But for some reason cypher-shell has a difference of opinion.

david_allen · February 14, 2020, 3:46pm

I'm at the end of my knowledge here, but I believe some separate toggle may be needed to make cypher-shell specifically accept self-signed certificates. At the driver level that is certainly the case, but sorry I don't know how the driver options map to cypher-shell flags.

mike_blum_neo4j · March 6, 2020, 10:05pm

This still seems to be an issue in neo4j 4.0.1. Opened a github issue to track it: 4.0.1 Cypher Shell - Self-Signed Certificates · Issue #209 · neo4j/cypher-shell · GitHub

Graham_Lea · April 28, 2020, 1:45am

Is this possibly because browsers give the option of accepting a dodgy certificate (i.e. out of date or mismatching Common Name) to establish an encrypted (but potentially MITM'd) connection, but cypher-shell doesn't provide a similar option, so its TLS stack always aborts if the certificate isn't perfect?

bawa.deepak · July 1, 2021, 10:38pm

Bit old post now.
But solution that worked for me: I had to add the cert to cacerts of Java used by neo4j as that is the one used by scripts such as cypher-shell.
Cheers.

tms · August 20, 2021, 9:49pm

I'm running my Neo4J (enterprise edition) servers on a Linux platform (CentOS 7, soon to be RockyLinux), so my approach may not work for those in other platforms.

After arguing with self-signed certs for years, about a month ago I switched to LetsEncrypt + Certbot.

It's dirt simple -- download the zipfile for your platform, unzip it, and follow the directions. Just a heads-up from bitter first-hand experience -- be sure to open the https port with firewall-cmd (run firewall-cmd --permanent --add-service https from a root shell and reboot), and if you're on AWS EC2 like me, be sure to open 443 in your security group(s).

Certbot automates all the headaches of renewing and installing DV certificates (so far as I know it only works for DV). Each cert you get has a 90-day lifetime, and Certbot installs a cron job to automagically renew it. That helps limit any vulnerability to bad guys.

Once installed, run sudo certbot certificates to get the path information needed for various config files. I also had to fiddle with ownership and permissions so that non-root services can read what they need. Nothing too hard, though -- in my opinion, MUCH easier than all the self-signed malarky.

So far as I know, the browsers all happily accept these certificates and you'll end up with a green padlock on any website.

I'm here because I'm about to try moving my Neo4J connections from http to https.

Topic		Replies	Views
Cypher-shell certificates Neo4j Graph Platform migrated	1	183	August 3, 2022
Cypher-shell - Connection to the database terminated Installation	6	955	July 22, 2021
Neo4j Browser won't establish a secure connection Neo4j Graph Platform browser , helm	1	512	August 19, 2021
Windows Desktop App 1.2.7 not connecting encrypted to Server 4.0.3 Desktop	4	984	June 28, 2020
Problem with connect to bolt in docker Neo4j Graph Platform	2	603	November 16, 2020

Cypher-shell certificates

Related topics