RotationTimeoutException: Failed to rotate logs. Expected version: '', actual version: '', wait timeout (ms): 60006

Neo4j Version : 3.5.14 Community Edition


I have my transaction logs stored at /data folder and graph.db is at /data/databases/

dbms.tx_log.rotation.retention_policy=100M
I have the above configuration in my neo4j.conf file

2020-10-12 13:18:06.660+0000 ERROR Failed to start transaction. The database has encountered a critical error, and needs to be restarted. Please see database logs for more details.
org.neo4j.graphdb.TransactionFailureException: The database has encountered a critical error, and needs to be restarted. Please see database logs for more details.
	at org.neo4j.graphdb.facade.spi.ClassicCoreSPI.beginTransaction(ClassicCoreSPI.java:193)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransactionInternal(GraphDatabaseFacade.java:691)
	at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:375)
	at org.neo4j.server.rest.transactional.TransitionalTxManagementKernelTransaction.startTransaction(TransitionalTxManagementKernelTransaction.java:128)
	at org.neo4j.server.rest.transactional.TransitionalTxManagementKernelTransaction.<init>(TransitionalTxManagementKernelTransaction.java:52)
	at org.neo4j.server.rest.transactional.TransitionalPeriodTransactionMessContainer.newTransaction(TransitionalPeriodTransactionMessContainer.java:55)
	at org.neo4j.server.rest.transactional.TransactionHandle.ensureActiveTransaction(TransactionHandle.java:213)
	at org.neo4j.server.rest.transactional.TransactionHandle.commit(TransactionHandle.java:156)
	at org.neo4j.server.rest.web.TransactionalService.lambda$executeStatementsAndCommit$1(TransactionalService.java:204)
	at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:71)
	at com.sun.jersey.core.impl.provider.entity.StreamingOutputProvider.writeTo(StreamingOutputProvider.java:57)
	at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)
	at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1510)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1419)
	at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1409)
	at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:409)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:558)
	at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:733)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:873)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1623)
	at org.neo4j.server.rest.dbms.AuthorizationEnabledFilter.doFilter(AuthorizationEnabledFilter.java:132)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.neo4j.server.rest.web.CorsFilter.doFilter(CorsFilter.java:115)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.neo4j.server.rest.web.CollectUserAgentFilter.doFilter(CollectUserAgentFilter.java:69)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1610)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:540)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1700)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1345)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:480)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1667)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1247)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerList.handle(HandlerList.java:61)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:505)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:370)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:267)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:305)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:103)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:117)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:786)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:743)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.neo4j.internal.kernel.api.exceptions.TransactionFailureException: The database has encountered a critical error, and needs to be restarted. Please see database logs for more details.
	at sun.reflect.GeneratedConstructorAccessor38.newInstance(Unknown Source)
	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
	at org.neo4j.kernel.internal.DatabaseHealth.assertHealthy(DatabaseHealth.java:61)
	at org.neo4j.kernel.impl.api.KernelImpl.beginTransaction(KernelImpl.java:88)
	at org.neo4j.graphdb.facade.spi.ClassicCoreSPI.beginTransaction(ClassicCoreSPI.java:186)
	... 53 more
Caused by: org.neo4j.kernel.impl.store.UnderlyingStorageException: Error performing check point
	at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.constructCombinedFailure(CheckPointScheduler.java:101)
	at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:82)
	at org.neo4j.kernel.impl.scheduler.ScheduledJobHandle.lambda$new$0(ScheduledJobHandle.java:83)
	at org.neo4j.kernel.impl.scheduler.ThreadPool.lambda$submit$0(ThreadPool.java:57)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	... 1 more
	Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1281340, actual version: 1137217, wait timeout (ms): 600051
		at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
		at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
		at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
		at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
		at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
		at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
		at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
		at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
		at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
		at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
		... 7 more
	Suppressed: org.neo4j.kernel.impl.store.kvstore.RotationTimeoutException: Failed to rotate logs. Expected version: 1281340, actual version: 1137217, wait timeout (ms): 600018
		at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:78)
		at org.neo4j.kernel.impl.store.kvstore.RotationState$Rotation.rotate(RotationState.java:51)
		at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:317)
		at org.neo4j.kernel.impl.store.kvstore.AbstractKeyValueStore$RotationTask.rotate(AbstractKeyValueStore.java:294)
		at org.neo4j.kernel.impl.store.counts.CountsTracker.rotate(CountsTracker.java:126)
		at org.neo4j.kernel.impl.store.NeoStores.flush(NeoStores.java:251)
		at org.neo4j.kernel.impl.storageengine.impl.recordstorage.RecordStorageEngine.flushAndForce(RecordStorageEngine.java:421)
		at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.doCheckPoint(CheckPointerImpl.java:179)
		at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointerImpl.checkPointIfNeeded(CheckPointerImpl.java:153)
		at org.neo4j.kernel.impl.transaction.log.checkpoint.CheckPointScheduler$1.run(CheckPointScheduler.java:65)
		... 7 more

and suddenly neo4j went into above error..Any quick help would be much appreciated..Thanks

Good day,

I've seen similar behavior once db doesn't have enough space for transaction log. Could you please verify if your system has enough disk space?

Thanks,
Alexander