cancel
Showing results for 
Search instead for 
Did you mean: 

Upgrade to v4.4.6 has broken my ingestor

tms
Graph Buddy

A few days ago, I updated neo4j-enterprise to v4.4.6. The upgrade appears to have broken Neo4J so that my ingestor code -- that has been running smoothly for nearly 2 years -- hangs while doing things that have worked for a long time.

I've spent the last two days trying to diagnose the issue with little success.

I see some unusual entries in debug.log that I don't remember seeing before. I'm trying to pin down precisely what provokes them -- I think it is a call to neo4j-admin to load a database from a snapshot.

These entries seem to occur in either pairs or even quadruples. Here is an example:

2022-05-17 16:50:26.493+0000 ERROR [o.n.b.r.s.i.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.General.UnknownError]: Lexical error at line 1, column 18.  Encountered: <EOF> after : "", reference 0886e4cc-e4df-409a-81b4-2dbd94d6e6bb.
2022-05-17 16:50:26.493+0000 ERROR [o.n.b.r.s.i.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.General.UnknownError]: Lexical error at line 1, column 18.  Encountered: <EOF> after : "", reference 0886e4cc-e4df-409a-81b4-2dbd94d6e6bb.
org.neo4j.cypher.internal.parser.javacc.TokenMgrException: Lexical error at line 1, column 18.  Encountered: <EOF> after : ""
	at org.neo4j.cypher.internal.parser.javacc.CypherTokenManager.getNextToken(CypherTokenManager.java:7616) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.jj_ntk_f(Cypher.java:23635) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.SymbolicDatabaseNameStringOrParameter(Cypher.java:18004) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.CreateDatabase(Cypher.java:17135) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.CreateCommand(Cypher.java:7263) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.SingleQueryOrCommand(Cypher.java:288) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.Statement(Cypher.java:262) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.Statements(Cypher.java:188) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.ast.factory.neo4j.JavaCCParser$.parse(JavaCCParser.scala:54) ~[neo4j-cypher-ast-factory-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.compiler.helpers.Neo4jJavaCCParserWithFallback$.parse(Neo4jJavaCCParserWithFallback.scala:48) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.process(JavaccParsing.scala:40) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.process(JavaccParsing.scala:36) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.Phase.$anonfun$transform$1(Phase.scala:36) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.helpers.package$.$anonfun$closing$1(package.scala:24) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.helpers.package$.using(package.scala:32) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.helpers.package$.closing(package.scala:24) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.Phase.transform(Phase.scala:35) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.Phase.transform$(Phase.scala:33) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.transform(JavaccParsing.scala:36) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.pipeline.FabricFrontEnd$Pipeline$parseAndPrepare$.process(FabricFrontEnd.scala:139) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.$anonfun$computePlan$1(FabricPlanner.scala:87) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.trace(FabricPlanner.scala:127) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.computePlan(FabricPlanner.scala:86) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.$anonfun$plan$1(FabricPlanner.scala:79) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.cache.FabricQueryCache.computeIfAbsent(FabricQueryCache.scala:52) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.plan$lzycompute(FabricPlanner.scala:80) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.plan(FabricPlanner.scala:76) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.executor.FabricExecutor.run(FabricExecutor.java:123) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.bolt.BoltFabricDatabaseService$BoltTransactionImpl.executeQuery(BoltFabricDatabaseService.java:186) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.AbstractTransactionStateMachineSPI$AbstractBoltResultHandle.start(AbstractTransactionStateMachineSPI.java:128) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine$State.startExecution(TransactionStateMachine.java:530) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine$State$2.run(TransactionStateMachine.java:326) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine.run(TransactionStateMachine.java:89) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine.run(TransactionStateMachine.java:82) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.transaction.StatementProcessorTxManager.runQuery(StatementProcessorTxManager.java:92) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.v4.runtime.InTransactionState.processRunMessage(InTransactionState.java:95) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.v4.runtime.InTransactionState.processUnsafe(InTransactionState.java:52) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.v3.runtime.FailSafeBoltStateMachineState.process(FailSafeBoltStateMachineState.java:48) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.nextState(AbstractBoltStateMachine.java:154) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.process(AbstractBoltStateMachine.java:102) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.messaging.BoltRequestMessageReader.lambda$doRead$1(BoltRequestMessageReader.java:93) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.DefaultBoltConnection.lambda$enqueue$0(DefaultBoltConnection.java:156) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatchInternal(DefaultBoltConnection.java:252) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:187) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:177) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.executeBatch(ExecutorBoltScheduler.java:257) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.lambda$scheduleBatchOrHandleError$3(ExecutorBoltScheduler.java:240) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [apoc-4.4.0.3-core.jar:4.4.0.3]
	at java.lang.Thread.run(Thread.java:829) [?:?]
2022-05-17 16:50:27.650+0000 ERROR [o.n.b.r.s.i.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.General.UnknownError]: Lexical error at line 1, column 25.  Encountered: <EOF> after : "", reference 326d903b-31ee-413f-8ba0-6943bd994f4d.
2022-05-17 16:50:27.650+0000 ERROR [o.n.b.r.s.i.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.General.UnknownError]: Lexical error at line 1, column 25.  Encountered: <EOF> after : "", reference 326d903b-31ee-413f-8ba0-6943bd994f4d.
org.neo4j.cypher.internal.parser.javacc.TokenMgrException: Lexical error at line 1, column 25.  Encountered: <EOF> after : ""
	at org.neo4j.cypher.internal.parser.javacc.CypherTokenManager.getNextToken(CypherTokenManager.java:7616) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.jj_ntk_f(Cypher.java:23635) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.SymbolicDatabaseNameStringOrParameter(Cypher.java:18004) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.CreateDatabase(Cypher.java:17135) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.CreateCommand(Cypher.java:7263) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.SingleQueryOrCommand(Cypher.java:288) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.Statement(Cypher.java:262) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.parser.javacc.Cypher.Statements(Cypher.java:188) ~[neo4j-cypher-javacc-parser-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.ast.factory.neo4j.JavaCCParser$.parse(JavaCCParser.scala:54) ~[neo4j-cypher-ast-factory-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.compiler.helpers.Neo4jJavaCCParserWithFallback$.parse(Neo4jJavaCCParserWithFallback.scala:48) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.process(JavaccParsing.scala:40) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.process(JavaccParsing.scala:36) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.Phase.$anonfun$transform$1(Phase.scala:36) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.helpers.package$.$anonfun$closing$1(package.scala:24) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.helpers.package$.using(package.scala:32) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.helpers.package$.closing(package.scala:24) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.Phase.transform(Phase.scala:35) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.Phase.transform$(Phase.scala:33) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.compiler.phases.JavaccParsing$.transform(JavaccParsing.scala:36) ~[neo4j-cypher-planner-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.cypher.internal.frontend.phases.PipeLine.transform(Transformer.scala:78) ~[neo4j-front-end-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.pipeline.FabricFrontEnd$Pipeline$parseAndPrepare$.process(FabricFrontEnd.scala:139) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.$anonfun$computePlan$1(FabricPlanner.scala:87) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.trace(FabricPlanner.scala:127) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.computePlan(FabricPlanner.scala:86) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.$anonfun$plan$1(FabricPlanner.scala:79) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.cache.FabricQueryCache.computeIfAbsent(FabricQueryCache.scala:52) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.plan$lzycompute(FabricPlanner.scala:80) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.planning.FabricPlanner$PlannerInstance.plan(FabricPlanner.scala:76) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.executor.FabricExecutor.run(FabricExecutor.java:123) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.fabric.bolt.BoltFabricDatabaseService$BoltTransactionImpl.executeQuery(BoltFabricDatabaseService.java:186) ~[neo4j-fabric-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.AbstractTransactionStateMachineSPI$AbstractBoltResultHandle.start(AbstractTransactionStateMachineSPI.java:128) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine$State.startExecution(TransactionStateMachine.java:530) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine$State$2.run(TransactionStateMachine.java:326) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine.run(TransactionStateMachine.java:89) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine.run(TransactionStateMachine.java:82) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.transaction.StatementProcessorTxManager.runQuery(StatementProcessorTxManager.java:92) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.v4.runtime.InTransactionState.processRunMessage(InTransactionState.java:95) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.v4.runtime.InTransactionState.processUnsafe(InTransactionState.java:52) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.v3.runtime.FailSafeBoltStateMachineState.process(FailSafeBoltStateMachineState.java:48) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.nextState(AbstractBoltStateMachine.java:154) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.process(AbstractBoltStateMachine.java:102) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.messaging.BoltRequestMessageReader.lambda$doRead$1(BoltRequestMessageReader.java:93) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.DefaultBoltConnection.lambda$enqueue$0(DefaultBoltConnection.java:156) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatchInternal(DefaultBoltConnection.java:252) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:187) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:177) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.executeBatch(ExecutorBoltScheduler.java:257) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.lambda$scheduleBatchOrHandleError$3(ExecutorBoltScheduler.java:240) ~[neo4j-bolt-4.4.6.jar:4.4.6]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [apoc-4.4.0.3-core.jar:4.4.0.3]
	at java.lang.Thread.run(Thread.java:829) [?:?]

I don't know enough about neo4j to know what this is trying to tell me.

I also notice that the most recent labs plugin is apoc-4.4.0.3-core.jar, while neo4j-enterprise is at version 4.4.6. My impression was that this apoc plugin was expected to track the running version.

I mention this because I see the following complaints in debug.log after each startup:

2022-05-16 20:49:14.173+0000 INFO  [c.n.c.n.Server] [backup-server] bound to '127.0.0.1:6362' with transport 'EpollServerSocketChannel'
2022-05-16 20:49:15.296+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.google.cloud.GCStorageURLConnection` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/google/cloud/storage/Storage
2022-05-16 20:49:15.383+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.hdfs.HDFSUtils` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataOutputStream
2022-05-16 20:49:15.384+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.hdfs.HDFSUtils$1` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/hadoop/fs/FSDataInputStream
2022-05-16 20:49:15.434+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.s3.S3Aws` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/amazonaws/auth/AWSCredentials
2022-05-16 20:49:15.435+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.s3.S3Aws$1` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/amazonaws/services/s3/model/S3ObjectInputStream
2022-05-16 20:49:15.437+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.s3.S3URLConnection` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/amazonaws/ClientConfiguration
2022-05-16 20:49:15.438+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `apoc.util.s3.S3OutputStream$S3UploadManager` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: Lcom/amazonaws/services/s3/AmazonS3;
2022-05-16 20:49:15.848+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.gui.TreeViewer$VariableExtentProvide` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/abego/treelayout/NodeExtentProvider
2022-05-16 20:49:15.849+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.gui.TreePostScriptGenerator` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/abego/treelayout/Configuration
2022-05-16 20:49:15.989+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.gui.TreeViewer` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/abego/treelayout/Configuration
2022-05-16 20:49:16.017+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.gui.TreePostScriptGenerator$VariableExtentProvide` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/abego/treelayout/NodeExtentProvider
2022-05-16 20:49:16.026+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.unicode.UnicodeDataTemplateController` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/ibm/icu/text/UnicodeSet
2022-05-16 20:49:16.118+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.v4.gui.TreeLayoutAdaptor` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/abego/treelayout/TreeForTreeLayout
2022-05-16 20:49:17.285+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `com.jayway.jsonpath.spi.json.GsonJsonProvider` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/google/gson/JsonElement
2022-05-16 20:49:17.314+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `com.jayway.jsonpath.spi.json.JsonOrgJsonProvider` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/json/JSONException
2022-05-16 20:49:17.397+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `com.jayway.jsonpath.spi.mapper.GsonMappingProvider$1` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/google/gson/Gson
2022-05-16 20:49:17.398+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `com.jayway.jsonpath.spi.mapper.GsonMappingProvider$2` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: com/google/gson/Gson
2022-05-16 20:49:17.571+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationNodePointer` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodePointer
2022-05-16 20:49:17.572+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationNodeIteratorBase` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodeIterator
2022-05-16 20:49:17.573+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationNodeIteratorAttribute` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodeIterator
2022-05-16 20:49:17.573+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationNodeIteratorChildren` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodeIterator
2022-05-16 20:49:17.574+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationNodePointerFactory` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodePointerFactory
2022-05-16 20:49:17.575+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.XPathExpressionEngine` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodePointerFactory
2022-05-16 20:49:17.575+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.XPathContextFactory` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/JXPathContext
2022-05-16 20:49:17.576+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.tree.xpath.ConfigurationAttributePointer` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jxpath/ri/model/NodePointer
2022-05-16 20:49:17.839+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.io.VFSFileSystem` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/vfs2/FileSystemException
2022-05-16 20:49:17.905+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.interpol.ExprLookup` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/jexl2/JexlContext
2022-05-16 20:49:17.911+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.reloading.VFSFileHandlerReloadingDetector` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/commons/vfs2/FileSystemException
2022-05-16 20:49:17.937+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.resolver.CatalogResolver` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/xml/resolver/CatalogManager
2022-05-16 20:49:17.938+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.resolver.CatalogResolver$Catalog` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/xml/resolver/Catalog
2022-05-16 20:49:17.938+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.resolver.CatalogResolver$CatalogManager` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/xml/resolver/CatalogManager
2022-05-16 20:49:17.939+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.spring.ConfigurationPropertySource` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/springframework/core/env/EnumerablePropertySource
2022-05-16 20:49:17.940+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.configuration2.spring.ConfigurationPropertiesFactoryBean` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/springframework/beans/factory/InitializingBean
2022-05-16 20:49:18.308+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `module-info` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: module-info is not a class because access_flag ACC_MODULE is set
2022-05-16 20:49:20.634+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.xerces.util.XMLCatalogResolver` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/xml/resolver/readers/CatalogReader
2022-05-16 20:49:20.838+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.antlr.runtime.tree.DOTTreeGenerator` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/antlr/stringtemplate/StringTemplate
2022-05-16 20:49:21.039+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.logging.impl.AvalonLogger` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/avalon/framework/logger/Logger
2022-05-16 20:49:21.041+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.logging.impl.Log4JLogger` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/log4j/Priority
2022-05-16 20:49:21.043+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `org.apache.commons.logging.impl.LogKitLogger` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/log/Logger
2022-05-16 20:49:22.344+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.netty.util.internal.logging.Log4JLogger` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/log4j/Priority
2022-05-16 20:49:22.360+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.netty.util.internal.Hidden$NettyBlockHoundIntegration$1` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: reactor/blockhound/integration/BlockHoundIntegration
2022-05-16 20:49:22.392+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.netty.util.internal.Hidden$NettyBlockHoundIntegration` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: reactor/blockhound/integration/BlockHoundIntegration
2022-05-16 20:49:22.399+0000 WARN  [o.n.k.a.p.GlobalProcedures] Failed to load `io.netty.util.internal.logging.Log4J2Logger` from plugin jar `/var/lib/neo4j/plugins/apoc-4.4.0.3-core.jar`: java.lang.NoClassDefFoundError: org/apache/logging/log4j/spi/ExtendedLoggerWrapper
2022-05-16 20:49:22.535+0000 INFO  [c.n.m.g.GlobalMetricsExtension] Sending metrics to CSV file at /var/lib/neo4j/metrics
...
2022-05-17 16:50:13.772+0000 WARN  [c.n.m.g.GlobalMetricsExtension] Unable to remove database specific metric files because metric namespaces are not enabled. In order to remove this warning you have to set 'metrics.namespaces.enabled=true' in your configuration file.

So far as I know, I don't have, don't want, and don't use a backup server (regarding the first complaint). I also see no mention of metrics.namespaces.enabled anywhere in neo4.conf or the documentation (regarding the last complaint)

My users expect this site to be updated every day. I've been unable to do so since 11-May (almost a week now).

I welcome any guidance the Neo4J community might offer.

1 REPLY 1

tms
Graph Buddy

My "ingestor" is Python3 code (v3.6.10) that uses the standard neo4j driver to store things in the db each day. None of that code has changed.

I've configured Neo4J following the recommendations of memrec, as follows:

dbms.memory.heap.initial_size=3500m
dbms.memory.heap.max_size=3500m
dbms.memory.pagecache.size=1900m
dbms.jvm.additional=-XX:+ExitOnOutOfMemoryError

The system is an AWS EC2 t2-large instance running CentOS 7. It has 2 vCPUs and 8G ram.

2022-05-18 Update: I've upgraded the instance type to t3.xlargeand the volumes from gp2 to gp3. This removes any throttling that AWS may have been applying.

Neo4J, sadly, continues to silently hang. I'm investigating the cause/source of the failures reported in debug.log.