I'm running neo4j 4.0.12 with JDK 1.11.0 on Ubuntu. One day this sort of errors started to happen after a restart:
[o.n.b.r.s.i.ErrorReporter] Client triggered an unexpected error [Neo.DatabaseError.General.UnknownError]: NOT PART OF CHAIN! RelationshipTraversalCursor[id=16066, open state with: denseNode=false, next=16066, mode=regular, underlying record=Relationship[16066,used=true,source=10853,target=10868,type=1,sPrev=16079,sNext=16061,tPrev=16065,tNext=-1,prop=-1,!sFirst,!tFirst]], reference 54de8ace-dbf3-4ee5-96b3-3d7b743dfc3c. NOT PART OF CHAIN! RelationshipTraversalCursor[id=16066, open state with: denseNode=false, next=16066, mode=regular, underlying record=Relationship[16066,used=true,source=10853,target=10868,type=1,sPrev=16079,sNext=16061,tPrev=16065,tNext=-1,prop=-1,!sFirst,!tFirst]] java.lang.IllegalStateException: NOT PART OF CHAIN! RelationshipTraversalCursor[id=16066, open state with: denseNode=false, next=16066, mode=regular, underlying record=Relationship[16066,used=true,source=10853,target=10868,type=1,sPrev=16079,sNext=16061,tPrev=16065,tNext=-1,prop=-1,!sFirst,!tFirst]] at org.neo4j.internal.recordstorage.RecordRelationshipTraversalCursor.computeNext(RecordRelationshipTraversalCursor.java:287) at org.neo4j.internal.recordstorage.RecordRelationshipTraversalCursor.next(RecordRelationshipTraversalCursor.java:176) at org.neo4j.kernel.impl.newapi.DefaultRelationshipTraversalCursor.next(DefaultRelationshipTraversalCursor.java:189) at org.neo4j.internal.kernel.api.helpers.RelationshipSparseSelection.fetchNext(RelationshipSparseSelection.java:135) at org.neo4j.internal.kernel.api.helpers.RelationshipSparseSelectionCursor.next(RelationshipSparseSelectionCursor.java:33) at org.neo4j.cypher.internal.runtime.interpreted.TransactionBoundQueryContext$$anon$1.fetchNext(TransactionBoundQueryContext.scala:166) at org.neo4j.cypher.internal.runtime.interpreted.TransactionBoundQueryContext$$anon$1.fetchNext(TransactionBoundQueryContext.scala:162) at org.neo4j.cypher.internal.runtime.interpreted.TransactionBoundQueryContext$CursorIterator.(TransactionBoundQueryContext.scala:900) at org.neo4j.cypher.internal.runtime.interpreted.TransactionBoundQueryContext$$anon$1.(TransactionBoundQueryContext.scala:162) at org.neo4j.cypher.internal.runtime.interpreted.TransactionBoundQueryContext.getRelationshipsForIds(TransactionBoundQueryContext.scala:162) at org.neo4j.cypher.internal.planning.ExceptionTranslatingQueryContext.$anonfun$getRelationshipsForIds$1(ExceptionTranslatingQueryContext.scala:221) at org.neo4j.cypher.internal.planning.ExceptionTranslationSupport.translateException(ExceptionTranslationSupport.scala:33) at org.neo4j.cypher.internal.planning.ExceptionTranslationSupport.translateException$(ExceptionTranslationSupport.scala:32) at org.neo4j.cypher.internal.planning.ExceptionTranslatingQueryContext.translateException(ExceptionTranslatingQueryContext.scala:40) at org.neo4j.cypher.internal.planning.ExceptionTranslatingQueryContext.getRelationshipsForIds(ExceptionTranslatingQueryContext.scala:221) at org.neo4j.cypher.internal.runtime.interpreted.DelegatingQueryContext.getRelationshipsForIds(DelegatingQueryContext.scala:88) at org.neo4j.cypher.internal.runtime.interpreted.pipes.ExpandAllPipe.$anonfun$internalCreateResults$1(ExpandAllPipe.scala:41) at scala.collection.Iterator$$anon$11.nextCur(Iterator.scala:480) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:486) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:507) at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:485) at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:507) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:454) at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:454) at scala.collection.Iterator.foreach(Iterator.scala:937) at scala.collection.Iterator.foreach$(Iterator.scala:937) at scala.collection.AbstractIterator.foreach(Iterator.scala:1425) at scala.collection.generic.Growable.$plus$plus$eq(Growable.scala:58) at scala.collection.generic.Growable.$plus$plus$eq$(Growable.scala:49) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:103) at scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:47) at scala.collection.TraversableOnce.to(TraversableOnce.scala:309) at scala.collection.TraversableOnce.to$(TraversableOnce.scala:307) at scala.collection.AbstractIterator.to(Iterator.scala:1425) at scala.collection.TraversableOnce.toBuffer(TraversableOnce.scala:301) at scala.collection.TraversableOnce.toBuffer$(TraversableOnce.scala:301) at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1425) at scala.collection.TraversableOnce.toArray(TraversableOnce.scala:288) at scala.collection.TraversableOnce.toArray$(TraversableOnce.scala:282) at scala.collection.AbstractIterator.toArray(Iterator.scala:1425) at org.neo4j.cypher.internal.runtime.interpreted.pipes.SortPipe.internalCreateResults(SortPipe.scala:32) at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:70) at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:66) at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:66) at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:66) at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:66) at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:66) at org.neo4j.cypher.internal.runtime.interpreted.pipes.PipeWithSource.createResults(Pipe.scala:66) at org.neo4j.cypher.internal.runtime.interpreted.PipeExecutionResult.request(PipeExecutionResult.scala:60) at org.neo4j.cypher.internal.result.StandardInternalExecutionResult.request(StandardInternalExecutionResult.scala:88) at org.neo4j.cypher.internal.result.ClosingExecutionResult.request(ClosingExecutionResult.scala:135) at org.neo4j.bolt.runtime.AbstractCypherAdapterStream.handleRecords(AbstractCypherAdapterStream.java:105) at org.neo4j.bolt.v3.messaging.ResultHandler.onPullRecords(ResultHandler.java:41) at org.neo4j.bolt.v4.messaging.PullResultConsumer.consume(PullResultConsumer.java:42) at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine$State.consumeResult(TransactionStateMachine.java:511) at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine$State$2.streamResult(TransactionStateMachine.java:355) at org.neo4j.bolt.runtime.statemachine.impl.TransactionStateMachine.streamResult(TransactionStateMachine.java:92) at org.neo4j.bolt.v4.runtime.InTransactionState.processStreamResultMessage(InTransactionState.java:73) at org.neo4j.bolt.v4.runtime.AbstractStreamingState.processUnsafe(AbstractStreamingState.java:49) at org.neo4j.bolt.v4.runtime.InTransactionState.processUnsafe(InTransactionState.java:60) at org.neo4j.bolt.v3.runtime.FailSafeBoltStateMachineState.process(FailSafeBoltStateMachineState.java:48) at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.nextState(AbstractBoltStateMachine.java:143) at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.process(AbstractBoltStateMachine.java:91) at org.neo4j.bolt.messaging.BoltRequestMessageReader.lambda$doRead$1(BoltRequestMessageReader.java:90) at org.neo4j.bolt.runtime.DefaultBoltConnection.lambda$enqueue$0(DefaultBoltConnection.java:151) at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatchInternal(DefaultBoltConnection.java:240) at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:175) at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:165) at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.executeBatch(ExecutorBoltScheduler.java:212) at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.lambda$scheduleBatchOrHandleError$2(ExecutorBoltScheduler.java:195) at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:829)
I did a check-consistency command on it and found that the database seems corrupted.
PROD1:~/CREDO/3rdparty/install/neo4j-community-4.0.12/bin$ ./neo4j-admin check-consistency --database=credo.db
2022-12-12 02:06:08.370+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Selected RecordFormat:StandardV4_0[SF4.0.0] record format from store /home/ubuntu/CREDO/3rdparty/install/neo4j-community-4.0.12/data/databases/credo.db
2022-12-12 02:06:08.372+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] Format not configured for store /home/ubuntu/CREDO/3rdparty/install/neo4j-community-4.0.12/data/databases/credo.db. Selected format from the store files: RecordFormat:StandardV4_0[SF4.0.0]
Index structure consistency check
.........2022-12-12 02:06:16.381+0000 ERROR [o.n.c.ConsistencyCheckService] Index inconsistency: Tree node has page id larger than registered last id, lastId=5, pageId=469762048.
Index will be excluded from further consistency checks. Index file: /home/ubuntu/CREDO/3rdparty/install/neo4j-community-4.0.12/data/databases/credo.db/neostore.nodestore.db.id.
...Consistency checking failed.Full consistency check did not complete
I tried copying the database to another one, and the new database has the same errors.
Any idea how to fix the database would be appreciated. Thanks.