Upon starting the neo4j instance on our server we run into a boatload of these kinds of “failed to load” exceptions.
In the end, the database starts, but it takes a really long time.
I tried looking up these kinds of errors, but I have not been able to find anything useful.
The reason I'm looking into this is that our Neo4j database crashes every so often and looking at the generated log files, I can't pinpoint what the problem is.
I have three parts of the debug.log file that indicate failure.
What would be a good way of trying to find out what the issue is if these three snippets are not a good indication?
- Neo4j version: 4.4.2 (enterprise)
- Driver: Neo4jClient (Dotnet)
This is the file information for the apoc .jar file which can't be loaded:
-rw-r--r-- 1 neo4j adm 18503746 Nov 18 2020 apoc-4.2.0.0-all.jar
2023-05-12 12:34:44.140+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.neo4j.driver.internal.shaded.io.netty.handler.ssl.JettyNpnSslEngine$2` from plugin jar `/var/lib/neo4j/plugins/apoc-4.2.0.0-all.jar`: org/eclipse/jetty/npn/NextProtoNego$ClientProvider
2023-05-12 12:34:44.141+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.neo4j.driver.internal.shaded.io.netty.handler.ssl.ReferenceCountedOpenSslContext` from plugin jar `/var/lib/neo4j/plugins/apoc-4.2.0.0-all.jar`: org/neo4j/driver/internal/shaded/io/netty/internal/tcnative/SSLPrivateKeyMethod
2023-05-12 12:34:44.143+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.neo4j.driver.internal.shaded.io.netty.handler.ssl.JettyAlpnSslEngine$ServerEngine` from plugin jar `/var/lib/neo4j/plugins/apoc-4.2.0.0-all.jar`: org/eclipse/jetty/alpn/ALPN$Provider
2023-05-12 12:34:44.146+0000 WARN [o.n.k.a.p.GlobalProcedures] Failed to load `org.neo4j.driver.internal.shaded.io.netty.handler.ssl.ConscryptAlpnSslEngine` from plugin jar `/var/lib/neo4j/plugins/apoc-4.2.0.0-all.jar`: org/conscrypt/BufferAllocator
2023-05-12 12:24:23.527+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=1214, gcTime=0, gcCount=0}
2023-05-12 12:24:23.835+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=382, gcTime=0, gcCount=0}
2023-05-12 12:24:24.785+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=750, gcTime=0, gcCount=0}
2023-05-12 12:14:59.670+0000 ERROR [o.n.b.r.s.i.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.General.UnknownError]: null, reference 4d08446f-901b-4cd5-a090-d8a6ec96648e.
2023-05-12 12:14:59.671+0000 ERROR [o.n.b.r.s.i.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.General.UnknownError]: null, reference 4d08446f-901b-4cd5-a090-d8a6ec96648e.
java.lang.NullPointerException: null
at java.util.concurrent.ConcurrentHashMap.get(ConcurrentHashMap.java:936) ~[?:?]
at org.neo4j.bolt.transaction.StatementProcessorTxManager.retrieveTx(StatementProcessorTxManager.java:200) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.transaction.StatementProcessorTxManager.commit(StatementProcessorTxManager.java:66) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.v4.runtime.InTransactionState.processCommitMessage(InTransactionState.java:107) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.v4.runtime.InTransactionState.processUnsafe(InTransactionState.java:56) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.v3.runtime.FailSafeBoltStateMachineState.process(FailSafeBoltStateMachineState.java:48) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.nextState(AbstractBoltStateMachine.java:154) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.process(AbstractBoltStateMachine.java:102) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.messaging.BoltRequestMessageReader.lambda$doRead$1(BoltRequestMessageReader.java:93) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.runtime.DefaultBoltConnection.lambda$enqueue$0(DefaultBoltConnection.java:156) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatchInternal(DefaultBoltConnection.java:252) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:187) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:177) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.executeBatch(ExecutorBoltScheduler.java:257) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.lambda$scheduleBatchOrHandleError$3(ExecutorBoltScheduler.java:240) ~[neo4j-bolt-4.4.2.jar:4.4.2]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-common-4.1.69.Final.jar:4.1.69.Final]
at java.lang.Thread.run(Thread.java:829) [?:?]
