Debug.log shows "The client is unauthorized due to authentication failure" eventhough database connection is created

Neo4j Graph Platform
,
1

While running Neo4j enterprise edition version 3.5.12 the following error occurs in debug.log after connecting from a browser session:

2020-01-08 15:48:14.388+0000 WARN [o.n.b.r.MetricsReportingBoltConnection] The client is unauthorized due to authentication failure.

After this error we see the following exception

2020-01-08 16:01:33.126+0000 ERROR [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection: [id: 0x484aafd4, L:/169.47.231.181:7687 - R:/115.186.56.78:54538] readAddress(..) failed: Connection timed out
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection timed out
at io.netty.channel.unix.Errors.newIOException(Errors.java:122)
at io.netty.channel.unix.Errors.ioResult(Errors.java:146)
at io.netty.channel.unix.FileDescriptor.readAddress(FileDescriptor.java:178)
at io.netty.channel.epoll.AbstractEpollChannel.doReadBytes(AbstractEpollChannel.java:343)
at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:785)
at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:433)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:330)
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:909)
at java.lang.Thread.run(Thread.java:748)

When these errors occur, after running simple queries the server will stop responding the CPU will go to 100% on one CPU and then will go to 100% on all 4-CPU's. The server will then become responsive again. When re-logging in with browser we will see the following issues again

2020-01-08 15:48:14.388+0000 WARN [o.n.b.r.MetricsReportingBoltConnection] The client is unauthorized due to authentication failure.

and the behavior will repeat. We had stopped the JVM with no success then we rebooted the Linux server and the login issue persists but the database is now responsive.

Below is the output of uname -a

3.10.0-957.27.2.el7.x86_64 #1 SMP Mon Jul 29 17:46:05 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

We are running 12 Gig of heap for initial size and max size and our server has 32 gig of memory.

The database was initially created using the Community Edition and was migrated to the Enterprise Edition. I did read this post

but would rather stay on the enterprise edition and not use the suggested solution "Restoring to an identical version resolves issue"

Any suggestions other then regular reboots?

2

After continued testing we determined that Neo4j was hanging when the heap size hit the max heap size. We used JConsole to monitor the heap size. The max heap size (dbms.memory.heap.max_size) is set in the neo4j.conf file.

We also decreased our insert statement size which seemed to slow the heap growth rate.