Container can't run if I use another neo4j version

General
1

Hi All,

An EC2 instance running two neo4j docker containers, as follow:
First one:
sudo docker run --name application -p 3001:7474 -p 3002:7687 -d -v /home/ubuntu/neo4j-application:/data -v /home/ubuntu/application/logs:/logs -v /home/ubuntu/application/import:/var/lib/neo4j/import -v /home/ubuntu/application/plugins:/plugins --env NEO4J_AUTH=neo4j/'password' --env NEO4JLABS_PLUGINS='["graph-data-science"]' neo4j:4.3.0

Second one:
sudo docker run --name data -p 3050:7474 -p 3060:7687 -d -v /home/ubuntu/neo4j-data:/data -v /home/ubuntu/data/logs:/logs -v /home/ubuntu/data/import:/var/lib/neo4j/import -v /home/ubuntu/data/plugins:/plugins --env NEO4J_AUTH=neo4j/'password' --env NEO4JLABS_PLUGINS='["graph-data-science"]' neo4j:4.2.7-community

You notice here the difference in version between them, the second one works well without any problem, but the first container has a lot of problems.

The first container runs fine but after a period of time the problem appears, it couldn't access the database.

I thought the issue was the version of the first container, I tried to run the container with neo4j:4.2.7-community but it failed.

Why I can't run the container with another version? and why the container can't access the database?

2

@obada.qafisheh

do you have more details/examples of the problems? Are their error messages? How are the problems observed?

3

Ok, let me explain

I'm not an expert with graph databases, I'm a cloud architect but working with a customer on the project.

I ran two docker containers on the same EC2 instance, the only difference in configuration between them is the version as mentioned in the question. The customer is using Graphileon with Neo4, suddenly, he was not able to connect to the application store - which is the first neo4j container with 4.3.0 version - and I checked it from the neo4j browser, the problem was that I can't log in to the store using neo4j:// scheme, but I can using bolt:// scheme, and I realized that the store can't access the database. When this problem occurs, the docker container is still running without any error logs.

I tried to restart the container but it didn't solve the problem, I solved it by backing to an old well-running backup. But after a period of time, the same problem occurred again.

I noticed that the docker container with 4.2.7-community is working well all the time even after restarting or moving the database volume, so, I tried to run another docker container like the first broken one but with 4.2.7-community instead of 4.3.0 on the same data volume of the old broken container, but it failed and I got this error while I start the container:

*ERROR Failed to start Neo4j on dbms.connector.http.listen_address, a socket address. If missing port or hostname it is acquired from dbms.defaul
*t_listen_address.
*java.lang.RuntimeException: Error starting Neo4j database server at /data/databases
*at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:207) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:163) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:36) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:134) [neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:90) [neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35) [neo4j-4.2.7.jar:4.2.7]
*Suppressed: java.lang.RuntimeException: Shutdown failed
*at org.neo4j.dbms.database.DatabaseManagementServiceImpl.shutdown(DatabaseManagementServiceImpl.java:137) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:216) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:163) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:36) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:134) [neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:90) [neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35) [neo4j-4.2.7.jar:4.2.7]
*Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.kernel.impl.pagecache.PageCacheLifecycle@7ace0b5d' failed to transition from stopped t
*o shutting_down. Please see the attached cause exception "Cannot close the PageCache while files are still mapped:
*/data/databases/system/neostore.id (1 mapping)
*/data/databases/system/neostore (1 mapping)".
*at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.shutdown(LifeSupport.java:516) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport.shutdown(LifeSupport.java:184) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.DatabaseManagementServiceImpl.shutdown(DatabaseManagementServiceImpl.java:131) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*... 6 more
*Caused by: java.lang.IllegalStateException: Cannot close the PageCache while files are still mapped:
*/data/databases/system/neostore.id (1 mapping)
*/data/databases/system/neostore (1 mapping)
*at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.close(MuninnPageCache.java:678) ~[neo4j-io-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.pagecache.PageCacheLifecycle.shutdown(PageCacheLifecycle.java:42) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.shutdown(LifeSupport.java:508) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport.shutdown(LifeSupport.java:184) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.DatabaseManagementServiceImpl.shutdown(DatabaseManagementServiceImpl.java:131) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*... 6 more
*Caused by: java.lang.IllegalStateException: Cannot close the PageCache while files are still mapped:
*/data/databases/system/neostore.id (1 mapping)
*/data/databases/system/neostore (1 mapping)
*at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.close(MuninnPageCache.java:678) ~[neo4j-io-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.pagecache.PageCacheLifecycle.shutdown(PageCacheLifecycle.java:42) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.shutdown(LifeSupport.java:508) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport.shutdown(LifeSupport.java:184) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.DatabaseManagementServiceImpl.shutdown(DatabaseManagementServiceImpl.java:131) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*... 6 more
*Caused by: org.neo4j.kernel.lifecycle.LifecycleException: Component 'org.neo4j.dbms.database.DefaultSystemGraphInitializer@fd413fb' was successfully initialized, but failed
*to start. Please see the attached cause exception "Unknown store version 'AF4.3.0'".
*at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:463) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198) ~[neo4j-4.2.7.jar:4.2.7]
*... 5 more
*Caused by: java.lang.IllegalStateException: Failed to initialize system graph component: This database is shutdown.
*at org.neo4j.dbms.database.SystemGraphComponents.initializeSystemGraph(SystemGraphComponents.java:102) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.java:41) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.SystemGraphInitializer.start(SystemGraphInitializer.java:29) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198) ~[neo4j-4.2.7.jar:4.2.7]
*... 5 more
*Caused by: org.neo4j.graphdb.DatabaseShutdownException: This database is shutdown.
*at org.neo4j.kernel.availability.DatabaseAvailabilityGuard.assertDatabaseAvailable(DatabaseAvailabilityGuard.java:172) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginKernelTransaction(GraphDatabaseFacade.java:197) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransactionInternal(GraphDatabaseFacade.java:176) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:122) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:116) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:104) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTx(GraphDatabaseFacade.java:99) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractSystemGraphComponent.detect(AbstractSystemGraphComponent.java:118) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractSystemGraphComponent.initializeSystemGraph(AbstractSystemGraphComponent.java:93) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.SystemGraphComponents.initializeSystemGraph(SystemGraphComponents.java:92) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.ja
*at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198) ~[neo4j-4.2.7.jar:4.2.7]
*... 5 more
*Suppressed: org.neo4j.graphdb.DatabaseShutdownException: This database is shutdown.
*at org.neo4j.kernel.availability.DatabaseAvailabilityGuard.assertDatabaseAvailable(DatabaseAvailabilityGuard.java:172) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginKernelTransaction(GraphDatabaseFacade.java:197) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransactionInternal(GraphDatabaseFacade.java:176) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:122) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:116) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTransaction(GraphDatabaseFacade.java:104) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.factory.GraphDatabaseFacade.beginTx(GraphDatabaseFacade.java:99) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractSystemGraphComponent.detect(AbstractSystemGraphComponent.java:118) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractSystemGraphComponent.initializeSystemGraph(AbstractSystemGraphComponent.java:93) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.SystemGraphComponents.initializeSystemGraph(SystemGraphComponents.java:92) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.DefaultSystemGraphInitializer.initializeSystemGraph(DefaultSystemGraphInitializer.java:41) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.SystemGraphInitializer.start(SystemGraphInitializer.java:29) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:163) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:36) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:134) [neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:90) [neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:35) [neo4j-4.2.7.jar:4.2.7]
*Caused by: java.lang.IllegalArgumentException: Unknown store version 'AF4.3.0'
*at org.neo4j.kernel.impl.store.format.RecordFormatSelector.selectForVersion(RecordFormatSelector.java:108) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.store.NeoStores.verifyRecordFormat(NeoStores.java:205) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.store.NeoStores.(NeoStores.java:117) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.store.StoreFactory.openNeoStores(StoreFactory.java:135) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.store.StoreFactory.openAllNeoStores(StoreFactory.java:99) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.internal.recordstorage.RecordStorageEngine.(RecordStorageEngine.java:165) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.internal.recordstorage.RecordStorageEngineFactory.instantiate(RecordStorageEngineFactory.java:138) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7
*]
*at org.neo4j.kernel.database.Database.start(Database.java:431) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:152) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:36) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractDatabaseManager.forEachDatabase(AbstractDatabaseManager.java:165) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractDatabaseManager.startAllDatabases(AbstractDatabaseManager.java:112) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractDatabaseManager.start(AbstractDatabaseManager.java:106) ~[neo4j-4.2.7.jar:4.2.7]
*... 8 more
*Caused by: java.lang.IllegalArgumentException: Unknown store version 'AF4.3.0'
*at org.neo4j.kernel.impl.store.format.RecordFormatSelector.selectForVersion(RecordFormatSelector.java:108) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.store.NeoStores.verifyRecordFormat(NeoStores.java:205) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.store.NeoStores.(NeoStores.java:117) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.store.StoreFactory.openNeoStores(StoreFactory.java:135) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.impl.store.StoreFactory.openAllNeoStores(StoreFactory.java:99) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.internal.recordstorage.RecordStorageEngine.(RecordStorageEngine.java:165) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.internal.recordstorage.RecordStorageEngineFactory.instantiate(RecordStorageEngineFactory.java:138) ~[neo4j-record-storage-engine-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.database.Database.start(Database.java:431) ~[neo4j-kernel-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractDatabaseManager.startDatabase(AbstractDatabaseManager.java:187) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:152) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.DefaultDatabaseManager.startDatabase(DefaultDatabaseManager.java:36) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractDatabaseManager.forEachDatabase(AbstractDatabaseManager.java:165) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractDatabaseManager.startAllDatabases(AbstractDatabaseManager.java:112) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.dbms.database.AbstractDatabaseManager.start(AbstractDatabaseManager.java:106) ~[neo4j-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:442) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:110) ~[neo4j-common-4.2.7.jar:4.2.7]
*at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:198) ~[neo4j-4.2.7.jar:4.2.7]
*... 5 more
*2021-10-21 09:48:43.370+0000 INFO Neo4j Server shutdown initiated by request
*2021-10-21 09:48:43.370+0000 INFO Stopped.