I am running Neo4j 4.1 on Azure. It has several databases. In testing on one database, I had a long running query which I terminated. However, the server was then not accessible. I restarted the server via Putty and then at Azure. The server restarted and I can see the databases in Neo4j Browser, but only neo4j (default) and system are online. I've tried START DATABASE XXX and then SHOW DATABASE XXX shows it is offline but requestedStatus=online. I've waited over an hour and it's still offline. I can also see the database using putty and it files looks normal. Furthermore my other databases are in a similar state. Thus, it's not just the database that had the terminated query.
I had similar difficulties with Neo4j on Azure in the past; but it involved not being able to even see the database in the Neoj browser. The only solution was to start over and rebuild the database. The tech team reviewed the log files and saw a difficulty which was put in the pipeline for a fix.
Is there any other way to get the database online?
While I can recreate the databases, I'd like to understand what caused this. In production this would be a disaster because every database was affected when I was working with only one.
The log files cannot be uploaded; too big. Can I send them by another mechanism for review?
Here's the last portion of debug.log
2020-10-26 18:15:08.881+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] [genealogy] Unable to read format for store /var/lib/neo4j/data/databases/genealogy. /var/lib/neo4j/data/databases/genealogy/neostore
2020-10-26 18:15:08.882+0000 INFO [o.n.k.i.s.f.RecordFormatSelector] [genealogy] Selected format 'RecordFormat:StandardV4_0[SF4.0.0]' for the new store /var/lib/neo4j/data/databases/genealogy
2020-10-26 18:15:08.891+0000 WARN [o.n.k.d.Database] [genealogy] Exception occurred while starting the database. Trying to stop already started components. /var/lib/neo4j/data/databases/genealogy/neostore
java.nio.file.AccessDeniedException: /var/lib/neo4j/data/databases/genealogy/neostore
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:71)
at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:53)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.createStoreChannel(SingleFilePageSwapper.java:143)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.(SingleFilePageSwapper.java:116)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapperFactory.createPageSwapper(SingleFilePageSwapperFactory.java:57)
at org.neo4j.io.pagecache.impl.muninn.MuninnPagedFile.(MuninnPagedFile.java:144)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.map(MuninnPageCache.java:393)
at org.neo4j.dbms.database.DatabasePageCache.map(DatabasePageCache.java:66)
at org.neo4j.kernel.impl.store.MetaDataStore.getRecord(MetaDataStore.java:305)
at org.neo4j.kernel.impl.store.MetaDataStore.getStoreId(MetaDataStore.java:371)
at org.neo4j.internal.recordstorage.RecordStorageEngineFactory.storeId(RecordStorageEngineFactory.java:190)
at org.neo4j.kernel.database.Database.checkStoreId(Database.java:513)
at org.neo4j.kernel.database.Database.start(Database.java:386)
at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187)
at com.neo4j.dbms.database.MultiDatabaseManager.forSingleDatabase(MultiDatabaseManager.java:134)
at com.neo4j.dbms.database.MultiDatabaseManager.startDatabase(MultiDatabaseManager.java:119)
at com.neo4j.dbms.Transition$Prepared.doTransitionAction(Transition.java:101)
at com.neo4j.dbms.Transition$Prepared.doTransition(Transition.java:88)
at com.neo4j.dbms.DbmsReconciler.doTransitionStep(DbmsReconciler.java:346)
at com.neo4j.dbms.DbmsReconciler.lambda$doTransitions$11(DbmsReconciler.java:315)
at com.neo4j.dbms.DbmsReconciler.namedJob(DbmsReconciler.java:326)
at com.neo4j.dbms.DbmsReconciler.doTransitions(DbmsReconciler.java:316)
at com.neo4j.dbms.DbmsReconciler.lambda$doTransitions$9(DbmsReconciler.java:307)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-10-26 18:15:08.892+0000 ERROR [c.n.d.DbmsReconciler] Encountered error when attempting to reconcile database genealogy from state 'EnterpriseDatabaseState{databaseId=DatabaseId{385cbe51[genealogy]}, operatorState=STOPPED, failed=false}' to state 'online' An error occurred! Unable to start database with name genealogy.
org.neo4j.dbms.api.DatabaseManagementException: An error occurred! Unable to start database with name genealogy.
at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:191)
at com.neo4j.dbms.database.MultiDatabaseManager.forSingleDatabase(MultiDatabaseManager.java:134)
at com.neo4j.dbms.database.MultiDatabaseManager.startDatabase(MultiDatabaseManager.java:119)
at com.neo4j.dbms.Transition$Prepared.doTransitionAction(Transition.java:101)
at com.neo4j.dbms.Transition$Prepared.doTransition(Transition.java:88)
at com.neo4j.dbms.DbmsReconciler.doTransitionStep(DbmsReconciler.java:346)
at com.neo4j.dbms.DbmsReconciler.lambda$doTransitions$11(DbmsReconciler.java:315)
at com.neo4j.dbms.DbmsReconciler.namedJob(DbmsReconciler.java:326)
at com.neo4j.dbms.DbmsReconciler.doTransitions(DbmsReconciler.java:316)
at com.neo4j.dbms.DbmsReconciler.lambda$doTransitions$9(DbmsReconciler.java:307)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.RuntimeException: java.nio.file.AccessDeniedException: /var/lib/neo4j/data/databases/genealogy/neostore
at org.neo4j.kernel.database.Database.start(Database.java:497)
at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187)
... 15 more
Caused by: java.nio.file.AccessDeniedException: /var/lib/neo4j/data/databases/genealogy/neostore
at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:90)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
at java.base/sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:182)
at java.base/java.nio.channels.FileChannel.open(FileChannel.java:292)
at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:71)
at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:53)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.createStoreChannel(SingleFilePageSwapper.java:143)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.(SingleFilePageSwapper.java:116)
at org.neo4j.io.pagecache.impl.SingleFilePageSwapperFactory.createPageSwapper(SingleFilePageSwapperFactory.java:57)
at org.neo4j.io.pagecache.impl.muninn.MuninnPagedFile.(MuninnPagedFile.java:144)
at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.map(MuninnPageCache.java:393)
at org.neo4j.dbms.database.DatabasePageCache.map(DatabasePageCache.java:66)
at org.neo4j.kernel.impl.store.MetaDataStore.getRecord(MetaDataStore.java:305)
at org.neo4j.kernel.impl.store.MetaDataStore.getStoreId(MetaDataStore.java:371)
at org.neo4j.internal.recordstorage.RecordStorageEngineFactory.storeId(RecordStorageEngineFactory.java:190)
at org.neo4j.kernel.database.Database.checkStoreId(Database.java:513)
at org.neo4j.kernel.database.Database.start(Database.java:386)
... 16 more
2020-10-26 18:15:16.647+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] [system] Checkpoint triggered by "Scheduled checkpoint for every 1 transactions threshold" @ txId: 115 checkpoint started...
2020-10-26 18:15:16.780+0000 INFO [o.n.k.i.t.l.c.CheckPointerImpl] [system] Checkpoint triggered by "Scheduled checkpoint for every 1 transactions threshold" @ txId: 115 checkpoint completed in 132ms
2020-10-26 18:15:16.813+0000 INFO [o.n.k.i.t.l.p.LogPruningImpl] [system] No log version pruned. The strategy used was '1 entries'. Last checkpoint was made in log version 0.
2020-10-26 20:22:47.942+0000 WARN [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection, remote peer unexpectedly closed connection: [id: 0x858c7c1d, L:/10.0.0.4:7687 - R:/66.140.176.47:55246]
2020-10-26 20:23:07.636+0000 WARN [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection, remote peer unexpectedly closed connection: [id: 0xabe58858, L:/10.0.0.4:7687 - R:/66.140.176.47:55251]
2020-10-26 20:23:07.861+0000 WARN [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection, remote peer unexpectedly closed connection: [id: 0x1b2b5bb9, L:/10.0.0.4:7687 - R:/66.140.176.47:55252]
2020-10-26 20:23:08.884+0000 WARN [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection, remote peer unexpectedly closed connection: [id: 0x4bfce685, L:/10.0.0.4:7687 - R:/66.140.176.47:55253]
2020-10-26 20:23:09.182+0000 WARN [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection, remote peer unexpectedly closed connection: [id: 0x7412dccf, L:/10.0.0.4:7687 - R:/66.140.176.47:55254]
2020-10-26 20:23:47.633+0000 WARN [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection, remote peer unexpectedly closed connection: [id: 0xbbfc2dcd, L:/10.0.0.4:7687 - R:/66.140.176.47:55259]
2020-10-26 20:23:48.164+0000 WARN [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection, remote peer unexpectedly closed connection: [id: 0x0648f136, L:/10.0.0.4:7687 - R:/66.140.176.47:55261]
2020-10-26 20:23:48.423+0000 WARN [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection, remote peer unexpectedly closed connection: [id: 0xb5876abc, L:/10.0.0.4:7687 - R:/66.140.176.47:55262]
2020-10-26 20:24:08.029+0000 WARN [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection, remote peer unexpectedly closed connection: [id: 0x682d0b62, L:/10.0.0.4:7687 - R:/66.140.176.47:55265]
2020-10-26 20:24:08.284+0000 WARN [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection, remote peer unexpectedly closed connection: [id: 0x3fb6dc10, L:/10.0.0.4:7687 - R:/66.140.176.47:55266]
2020-10-26 20:27:48.322+0000 ERROR [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection: [id: 0xbd5370aa, L:/10.0.0.4:7687 - R:/66.140.176.47:55243] readAddress(..) failed: Connection timed out
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection timed out
2020-10-26 20:27:48.322+0000 ERROR [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection: [id: 0x606e514e, L:/10.0.0.4:7687 - R:/66.140.176.47:55250] readAddress(..) failed: Connection timed out
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection timed out
2020-10-27 00:17:10.882+0000 ERROR [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection: [id: 0x51688cf0, L:/10.0.0.4:7687 - R:/66.140.176.47:57408] readAddress(..) failed: Connection timed out
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection timed out
2020-10-27 00:17:10.882+0000 ERROR [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection: [id: 0xd30f4c2a, L:/10.0.0.4:7687 - R:/66.140.176.47:57412] readAddress(..) failed: Connection timed out
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection timed out
2020-10-27 00:17:10.882+0000 ERROR [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection: [id: 0x643a6f27, L:/10.0.0.4:7687 - R:/66.140.176.47:57405] readAddress(..) failed: Connection timed out
io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection timed out
2020-10-27 03:23:45.659+0000 ERROR [o.n.b.t.p.ProtocolHandshaker] Fatal error occurred during protocol handshaking: [id: 0x5609eefb, L:/10.0.0.4:7687 - R:/66.140.176.47:62696]
io.netty.handler.timeout.ReadTimeoutException
2020-10-26 17:41:49.665+0000 ERROR Unexpected error detected in bolt session 'bolt-9763'. The transaction has been terminated. Retry your operation in a new transaction, and you should see a successful result. Explicitly terminated by the user.
org.neo4j.graphdb.TransactionTerminatedException: The transaction has been terminated. Retry your operation in a new transaction, and you should see a successful result. Explicitly terminated by the user.
at org.neo4j.bolt.runtime.statemachine.MutableConnectionState.ensureNoPendingTerminationNotice(MutableConnectionState.java:231)
at org.neo4j.bolt.runtime.statemachine.MutableConnectionState.getStatementProcessor(MutableConnectionState.java:173)
at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.statementProcessor(AbstractBoltStateMachine.java:316)
at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.validateTransaction(AbstractBoltStateMachine.java:187)
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatchInternal(DefaultBoltConnection.java:226)
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:172)
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:162)
at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.executeBatch(ExecutorBoltScheduler.java:246)
at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.lambda$scheduleBatchOrHandleError$3(ExecutorBoltScheduler.java:229)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-10-26 17:41:56.800+0000 ERROR Unexpected error detected in bolt session 'bolt-9764'. The transaction has been terminated. Retry your operation in a new transaction, and you should see a successful result. Explicitly terminated by the user.
org.neo4j.graphdb.TransactionTerminatedException: The transaction has been terminated. Retry your operation in a new transaction, and you should see a successful result. Explicitly terminated by the user.
at org.neo4j.bolt.runtime.statemachine.MutableConnectionState.ensureNoPendingTerminationNotice(MutableConnectionState.java:231)
at org.neo4j.bolt.runtime.statemachine.MutableConnectionState.getStatementProcessor(MutableConnectionState.java:173)
at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.statementProcessor(AbstractBoltStateMachine.java:316)
at org.neo4j.bolt.runtime.statemachine.impl.AbstractBoltStateMachine.validateTransaction(AbstractBoltStateMachine.java:187)
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatchInternal(DefaultBoltConnection.java:226)
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:172)
at org.neo4j.bolt.runtime.DefaultBoltConnection.processNextBatch(DefaultBoltConnection.java:162)
at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.executeBatch(ExecutorBoltScheduler.java:246)
at org.neo4j.bolt.runtime.scheduling.ExecutorBoltScheduler.lambda$scheduleBatchOrHandleError$3(ExecutorBoltScheduler.java:229)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
nohup: ignoring input
APOC couln't set a URLStreamHandlerFactory since some other tool already did this (e.g. tomcat). This means you cannot use s3:// or hdfs:// style URLs in APOC. This is caused by a limitation of the JVM which we cannot fix.
2020-10-26 17:47:28.312+0000 WARN The 'dbms.backup.listen_address' setting is overridden. Setting value changed from 'localhost:6362' to '0.0.0.0'.
2020-10-26 17:47:28.313+0000 WARN The 'dbms.default_listen_address' setting is overridden. Setting value changed from '10.0.0.4' to '0.0.0.0'.
2020-10-26 17:47:28.314+0000 WARN The 'dbms.security.procedures.unrestricted' setting is overridden. Setting value changed from 'apoc.' to 'apoc.,gds.* '.
2020-10-26 17:47:28.314+0000 INFO Starting...
2020-10-26 17:47:30.995+0000 INFO ======== Neo4j 4.1.0 ========
2020-10-26 17:47:36.752+0000 ERROR Failed to start Neo4j on dbms.connector.http.listen_address, a socket address. If missing port or hostname it is acquired from dbms.default_listen_address. Error starting Neo4j database server at /var/lib/neo4j/data/databases
java.lang.RuntimeException: Error starting Neo4j database server at /var/lib/neo4j/data/databases
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:158)
at com.neo4j.server.enterprise.EnterpriseManagementServiceFactory.createManagementService(EnterpriseManagementServiceFactory.java:38)
at com.neo4j.server.enterprise.EnterpriseBootstrapper.createNeo(EnterpriseBootstrapper.java:20)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:117)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:87)
at com.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:25)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.internal.locker.LockerLifecycleAdapter@3e27aa33' was successfully initialized, but failed to start. Please see the attached cause exception "Lock file has been locked by another process: /var/lib/neo4j/data/databases/store_lock. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:463)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:189)
... 6 more
Caused by: org.neo4j.kernel.internal.locker.FileLockException: Lock file has been locked by another process: /var/lib/neo4j/data/databases/store_lock. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)
at org.neo4j.kernel.internal.locker.Locker.storeLockException(Locker.java:176)
at org.neo4j.kernel.internal.locker.Locker.checkLock(Locker.java:96)
at org.neo4j.kernel.internal.locker.GlobalFileLocker.checkLock(GlobalFileLocker.java:58)
at org.neo4j.kernel.internal.locker.GlobalLocker.checkLock(GlobalLocker.java:28)
at org.neo4j.kernel.internal.locker.LockerLifecycleAdapter.start(LockerLifecycleAdapter.java:36)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442)
... 8 more
2020-10-26 17:47:36.752+0000 INFO Neo4j Server shutdown initiated by request
nohup: ignoring input
APOC couln't set a URLStreamHandlerFactory since some other tool already did this (e.g. tomcat). This means you cannot use s3:// or hdfs:// style URLs in APOC. This is caused by a limitation of the JVM which we cannot fix.
2020-10-27 04:39:04.772+0000 WARN The 'dbms.backup.listen_address' setting is overridden. Setting value changed from 'localhost:6362' to '0.0.0.0'.
2020-10-27 04:39:04.774+0000 WARN The 'dbms.default_listen_address' setting is overridden. Setting value changed from '10.0.0.4' to '0.0.0.0'.
2020-10-27 04:39:04.774+0000 WARN The 'dbms.security.procedures.unrestricted' setting is overridden. Setting value changed from 'apoc.' to 'apoc.,gds.* '.
2020-10-27 04:39:04.775+0000 INFO Starting...
2020-10-27 04:39:07.565+0000 INFO ======== Neo4j 4.1.0 ========
2020-10-27 04:39:13.405+0000 ERROR Failed to start Neo4j on dbms.connector.http.listen_address, a socket address. If missing port or hostname it is acquired from dbms.default_listen_address. Error starting Neo4j database server at /var/lib/neo4j/data/databases
java.lang.RuntimeException: Error starting Neo4j database server at /var/lib/neo4j/data/databases
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:158)
at com.neo4j.server.enterprise.EnterpriseManagementServiceFactory.createManagementService(EnterpriseManagementServiceFactory.java:38)
at com.neo4j.server.enterprise.EnterpriseBootstrapper.createNeo(EnterpriseBootstrapper.java:20)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:117)
at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:87)
at com.neo4j.server.enterprise.EnterpriseEntryPoint.main(EnterpriseEntryPoint.java:25)
Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.internal.locker.LockerLifecycleAdapter@3e27aa33' was successfully initialized, but failed to start. Please see the attached cause exception "Lock file has been locked by another process: /var/lib/neo4j/data/databases/store_lock. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)".
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:463)
at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110)
at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:189)
... 6 more
Caused by: org.neo4j.kernel.internal.locker.FileLockException: Lock file has been locked by another process: /var/lib/neo4j/data/databases/store_lock. Please ensure no other process is using this database, and that the directory is writable (required even for read-only access)
at org.neo4j.kernel.internal.locker.Locker.storeLockException(Locker.java:176)
at org.neo4j.kernel.internal.locker.Locker.checkLock(Locker.java:96)
at org.neo4j.kernel.internal.locker.GlobalFileLocker.checkLock(GlobalFileLocker.java:58)
at org.neo4j.kernel.internal.locker.GlobalLocker.checkLock(GlobalLocker.java:28)
at org.neo4j.kernel.internal.locker.LockerLifecycleAdapter.start(LockerLifecycleAdapter.java:36)
at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442)
... 8 more
2020-10-27 04:39:13.405+0000 INFO Neo4j Server shutdown initiated by request
This suggests it's a file permission/ownership issue. You need to make sure all of the files under data/ are owned by the user running neo4j (usually it's the neo4j user). If you did a recent import, restore, or load, you may want to follow-up with checking (and if necessary resetting) the ownership to avoid errors like this, since the user you use to run the import/restore/load will usually make that the owner of the files.