Database availability issue when migrating Neo4j Aura DB to EC2 Neo4j instance

Neo4j Graph Platform
1

Question:

I am migrating a Neo4j Aura DB to a Neo4j instance running on an AWS EC2 instance, and I am encountering an issue with database availability. The situation is as follows:

  • The Neo4j Aura DB is using version 5.19-aura.
  • The Neo4j DB on the AWS EC2 instance is using version 5.3.0.
  • When I create a dump file from the EC2 Neo4j DB using bin/neo4j-admin database dump and import it into the Aura DB, the import is successful.
  • However, when I create a dump file from a snapshot of the Aura DB and try to import it into the EC2 Neo4j DB using bin/neo4j-admin database load, the import executes, but when I access the browser, it displays "Database 'neo4j' is unavailable. Run :sysinfo for more info."

I need to perform this Aura DB to EC2 migration regularly, so any insights into resolving the database availability issue when importing from Aura to EC2 would be appreciated.

The actual steps I followed are:

  1. sudo systemctl stop neo4j
  2. cd /usr/share/neo4j/
  3. bin/neo4j-admin server unbind
  4. mkdir dump
  5. aws s3 cp s3://xxxx/neo4j.dump /root/dump/
  6. cd /usr/share/neo4j/
  7. bin/neo4j-admin database load --from-path=/root/dump/ neo4j --overwrite-destination=true --verbose
  8. sudo systemctl restart neo4j
  9. Access the specified IP address:7474/browser in the browser, and it's OK, but it displays "Database 'neo4j' is unavailable. Run :sysinfo for more info."
2

@yuuka.mizu87

why 2 different versions? i.e. 5.19 vs 5.3?

on the EC2 instance and after running neo4j-admin database load .... does logs\neo4j.log or logs\debug.log report any issues?

Is this an issue whereby read/write and owndership of data/ is not correct?

3

@dana_canzano

After aligning neo4j DB on EC2 instance to version 5.19 and running the neo4j-admin database load ... command, the issue persisted. Let's investigate the logs to see if we can find any clues.

4

When I checked /var/log/neo4j/debug.log, the following error appeared.
”””
2024-04-25 01:32:49.278+0000 ERROR [o.n.d.d.DatabaseLifecycles] Failed to start DatabaseId{ca992062[neo4j]}

java.lang.RuntimeException: java.lang.IllegalStateException: Failed to read current store version.

at org.neo4j.kernel.database.AbstractDatabase.handleStartupFailure(AbstractDatabase.java:215) ~[neo4j-kernel-5.19.0.jar:5.19.0]

at org.neo4j.kernel.database.AbstractDatabase.start(AbstractDatabase.java:168) ~[neo4j-kernel-5.19.0.jar:5.19.0]

at org.neo4j.dbms.database.DatabaseLifecycles.startDatabase(DatabaseLifecycles.java:123) [neo4j-5.19.0.jar:5.19.0]

at org.neo4j.dbms.database.DatabaseLifecycles.initialiseDefaultDatabase(DatabaseLifecycles.java:89) [neo4j-5.19.0.jar:5.19.0]

at org.neo4j.dbms.database.DatabaseLifecycles$DefaultDatabaseStarter.start(DatabaseLifecycles.java:185) [neo4j-5.19.0.jar:5.19.0]

at org.neo4j.kernel.lifecycle.LifeSupport$LifecycleInstance.start(LifeSupport.java:347) [neo4j-common-5.19.0.jar:5.19.0]

at org.neo4j.kernel.lifecycle.LifeSupport.start(LifeSupport.java:91) [neo4j-common-5.19.0.jar:5.19.0]

at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:269) [neo4j-5.19.0.jar:5.19.0]

at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:213) [neo4j-5.19.0.jar:5.19.0]

at org.neo4j.server.CommunityBootstrapper.createNeo(CommunityBootstrapper.java:38) [neo4j-5.19.0.jar:5.19.0]

at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:184) [neo4j-5.19.0.jar:5.19.0]

at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:99) [neo4j-5.19.0.jar:5.19.0]

at org.neo4j.server.CommunityEntryPoint.main(CommunityEntryPoint.java:30) [neo4j-5.19.0.jar:5.19.0]

Caused by: java.lang.IllegalStateException: Failed to read current store version.

at org.neo4j.kernel.impl.storemigration.StoreMigrator.doUpgradeCheck(StoreMigrator.java:392) ~[neo4j-kernel-5.19.0.jar:5.19.0]

at org.neo4j.kernel.impl.storemigration.StoreMigrator.upgradeIfNeeded(StoreMigrator.java:248) ~[neo4j-kernel-5.19.0.jar:5.19.0]

at org.neo4j.kernel.database.Database.upgradeStore(Database.java:784) ~[neo4j-kernel-5.19.0.jar:5.19.0]

at org.neo4j.kernel.database.Database.specificStart(Database.java:415) ~[neo4j-kernel-5.19.0.jar:5.19.0]

at org.neo4j.kernel.database.AbstractDatabase.start(AbstractDatabase.java:160) ~[neo4j-kernel-5.19.0.jar:5.19.0]

... 11 more

Caused by: java.nio.file.AccessDeniedException: /var/lib/neo4j/data/databases/neo4j/neostore

at sun.nio.fs.UnixException.translateToIOException(UnixException.java:90) ~[?:?]

at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106) ~[?:?]

at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) ~[?:?]

at sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:181) ~[?:?]

at java.nio.channels.FileChannel.open(FileChannel.java:298) ~[?:?]

at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:71) ~[neo4j-io-5.19.0.jar:5.19.0]

at org.neo4j.io.fs.DefaultFileSystemAbstraction.open(DefaultFileSystemAbstraction.java:57) ~[neo4j-io-5.19.0.jar:5.19.0]

at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.createStoreChannel(SingleFilePageSwapper.java:143) ~[neo4j-io-5.19.0.jar:5.19.0]

at org.neo4j.io.pagecache.impl.SingleFilePageSwapper.(SingleFilePageSwapper.java:119) ~[neo4j-io-5.19.0.jar:5.19.0]

at org.neo4j.io.pagecache.impl.SingleFilePageSwapperFactory.createPageSwapper(SingleFilePageSwapperFactory.java:80) ~[neo4j-io-5.19.0.jar:5.19.0]

at org.neo4j.io.pagecache.impl.muninn.MuninnPagedFile.(MuninnPagedFile.java:218) ~[neo4j-io-5.19.0.jar:5.19.0]

at org.neo4j.io.pagecache.impl.muninn.MuninnPageCache.map(MuninnPageCache.java:683) ~[neo4j-io-5.19.0.jar:5.19.0]

at org.neo4j.dbms.database.DatabasePageCache.map(DatabasePageCache.java:96) ~[neo4j-kernel-5.19.0.jar:5.19.0]

at org.neo4j.io.pagecache.PageCache.map(PageCache.java:115) ~[neo4j-io-5.19.0.jar:5.19.0]

at org.neo4j.kernel.impl.store.MetaDataStore$FieldAccess.readValue(MetaDataStore.java:575) ~[neo4j-record-storage-engine-5.19.0.jar:5.19.0]

at org.neo4j.kernel.impl.store.MetaDataStore$FieldAccess.isLegacyFieldValid(MetaDataStore.java:558) ~[neo4j-record-storage-engine-5.19.0.jar:5.19.0]

at org.neo4j.kernel.impl.storemigration.RecordStoreVersionCheck.readVersion(RecordStoreVersionCheck.java:70) ~[neo4j-record-storage-engine-5.19.0.jar:5.19.0]

at org.neo4j.kernel.impl.storemigration.RecordStoreVersionCheck.getAndCheckUpgradeTargetVersion(RecordStoreVersionCheck.java:143) ~[neo4j-record-storage-engine-5.19.0.jar:5.19.0]

at org.neo4j.kernel.impl.storemigration.StoreMigrator.doUpgradeCheck(StoreMigrator.java:384) ~[neo4j-kernel-5.19.0.jar:5.19.0]

at org.neo4j.kernel.impl.storemigration.StoreMigrator.upgradeIfNeeded(StoreMigrator.java:248) ~[neo4j-kernel-5.19.0.jar:5.19.0]

at org.neo4j.kernel.database.Database.upgradeStore(Database.java:784) ~[neo4j-kernel-5.19.0.jar:5.19.0]

at org.neo4j.kernel.database.Database.specificStart(Database.java:415) ~[neo4j-kernel-5.19.0.jar:5.19.0]

at org.neo4j.kernel.database.AbstractDatabase.start(AbstractDatabase.java:160) ~[neo4j-kernel-5.19.0.jar:5.19.0]

... 11 more
”””

5

Neo4j database built on EC2 is not an enterprise version.

6

I verified the owner as follows:
ls -l val/lib/noe4j/data
total 20
drwxr-xr-x 4 neo4j adm 4096 Apr 25 05:17 databases
drwxr-xr-x 2 neo4j adm 4096 Apr 25 05:17 dbms
drwxr-xr-x 2 neo4j adm 4096 Apr 12 08:22 dumps
-rw-r--r-- 1 neo4j neo4j 17 Apr 25 05:17 server_id
drwxr-xr-x 4 neo4j adm 4096 Apr 25 05:17 transactions

7

When I executed the part marked with (*) below, I was able to successfully read and execute the dump file.

bin/neo4j-admin database load --from-path=/root/dump/ neo4j --overwrite-destination=true --verbose
(* cd var/lib/neo4j/data
(* sudo chown -R neo4j:neo4j databases/
(* sudo chown -R neo4j:neo4j transactions/
sudo systemctl restart neo4j
sudo systemctl status neo4j