Hi, don't know if here or on Github is the place for this, however we've come across an issue while using the official neo4j docker that I'm not entirely certain how to place. Here's the setup:
- Neo4j Version: We run neo4j:latest container
- Language: Python 3.10 w/neo4j:latest python driver lib
- OS: docker in AWS Linux
- Not on a cluster
This is a fatal error that occurs awhile after database startup on connection request by our application. It bubbles up to our driver that the connection was lost because of a fatal DB error, so into the debug logs I went, to find this:
2022-06-30 18:43:43.368+0000 ERROR [o.n.b.t.p.ProtocolHandshaker] Fatal error occurred during protocol handshaking: [id: 0x477f0a03, L:/172.19.0.2:7687 - R:/172.19.0.1:57778]
java.lang.NullPointerException: null
at io.netty.buffer.Unpooled.copiedBuffer(Unpooled.java:363) ~[netty-buffer-4.1.75.Final.jar:4.1.75.Final]
at org.neo4j.bolt.transport.DiscoveryResponseHandler.channelRead(DiscoveryResponseHandler.java:60) ~[neo4j-bolt-4.4.8.jar:4.4.8]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.handler.codec.ByteToMessageDecoder.handlerRemoved(ByteToMessageDecoder.java:255) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:517) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:449) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:279) [netty-codec-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at org.neo4j.bolt.transport.pipeline.AuthenticationTimeoutHandler.channelRead(AuthenticationTimeoutHandler.java:58) [neo4j-bolt-4.4.8.jar:4.4.8]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) [netty-transport-classes-epoll-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:480) [netty-transport-classes-epoll-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378) [netty-transport-classes-epoll-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
2022-06-30 18:52:36.788+0000 ERROR [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection: [id: 0xeb7069d4, L:/172.19.0.2:7687 ! R:/172.19.0.1:48598]
org.neo4j.bolt.runtime.BoltConnectionFatality: Terminated connection '[id: 0xeb7069d4, L:/172.19.0.2:7687 ! R:/172.19.0.1:48598]' as the client failed to authenticate within 30000 ms.
at org.neo4j.bolt.transport.pipeline.AuthenticationTimeoutHandler.channelIdle(AuthenticationTimeoutHandler.java:78) ~[neo4j-bolt-4.4.8.jar:4.4.8]
at io.netty.handler.timeout.IdleStateHandler$ReaderIdleTimeoutTask.run(IdleStateHandler.java:506) [netty-handler-4.1.75.Final.jar:4.1.75.Final]
at io.netty.handler.timeout.IdleStateHandler$AbstractIdleTask.run(IdleStateHandler.java:478) [netty-handler-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:170) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:384) [netty-transport-classes-epoll-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.75.Final.jar:4.1.75.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
I saw another community post (here) that mentioned a connection pool issue, however this doesn't happen on timeout, rather right away after a connection is attempted. To be clear, this doesn't happen on every connection, rather it happens relatively spontaneously and a restart has cleared up the problem. I checked memory and storage to make sure that there weren't any issues there and so far as I saw there weren't:
Container Memory/Storage:
Storage Host:
Any help or pointers on this would be appreciated, even pointers to another ticket that may already be opened or documentation that talks about this.