Backup and Restore to same instance Error

Neo4j Graph Platform Operations
,
1

Neo4j V5.26.5 on AWS running in EC2

I'm trying to run backup and restore on a stand alone single server (not clustered) in AWS EC2.
The initial Neo4j server runs correctly, I add some test nodes, then take a backup. Once I restore the backup to the same instance it won't start.

To restore the backup I stop the service (sudo systemctl stop neo4j), then restore both the system and neo4j databases:

sudo neo4j-admin database restore --overwrite-destination=true --from-path=s3://myBucket/myDirectory/system.backup system
sudo neo4j-admin database restore --overwrite-destination=true --from-path=s3://myBucket/myDirectory/neo4j.backup neo4j

then I try to start neo4j again (sudo systemctl start neo4j)

/bin/neo4j status --verbose
neo4j 5.26.5
VM Name: OpenJDK 64-Bit Server VM
VM Vendor: Amazon.com Inc.
VM Version: 17.0.14+7-LTS
JIT compiler: HotSpot 64-Bit Tiered Compilers
VM Arguments: [-Xmx128m, -Dapp.name=neo4j, -Dapp.pid=4179, -Dapp.repo=/usr/share/neo4j/repo, -Dapp.home=/usr/share/neo4j, -Dbasedir=/usr/share/neo4j]
Configuration files used (ordered by priority):
/etc/neo4j/neo4j.conf
--------------------
org.neo4j.cli.CommandFailedException: Neo4j is not running.
        at org.neo4j.server.startup.Bootloader$Dbms.status(Bootloader.java:528)
        at org.neo4j.server.startup.StatusCommand.execute(StatusCommand.java:37)
        at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:92)
        at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:37)
        at picocli.CommandLine.executeUserObject(CommandLine.java:2045)
        at picocli.CommandLine.access$1500(CommandLine.java:148)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2421)
        at picocli.CommandLine.execute(CommandLine.java:2174)
        at org.neo4j.server.startup.Neo4jAdminCommand.call(Neo4jAdminCommand.java:123)
        at org.neo4j.server.startup.Neo4jAdminCommand.call(Neo4jAdminCommand.java:42)
        at picocli.CommandLine.executeUserObject(CommandLine.java:2045)
        at picocli.CommandLine.access$1500(CommandLine.java:148)
        at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2465)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2457)
        at picocli.CommandLine$RunLast.handle(CommandLine.java:2419)
        at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2277)
        at picocli.CommandLine$RunLast.execute(CommandLine.java:2421)
        at picocli.CommandLine.execute(CommandLine.java:2174)
        at org.neo4j.server.startup.Neo4jCommand.main(Neo4jCommand.java:41)
        at org.neo4j.server.startup.Neo4jBoot.main(Neo4jBoot.java:30)

The error in the neo4j.log file is:

2025-04-16 13:02:39.889+0000 INFO  Starting...
2025-04-16 13:02:46.686+0000 INFO  ======== Neo4j 5.26.5 ========
2025-04-16 13:02:46.714+0000 INFO  This instance is ServerId{4488b52a} (4488b52a-46f3-4918-b891-52ea97490881)
2025-04-16 13:02:47.286+0000 INFO  Resolved endpoints with LIST{endpoints:'[10.0.130.209:7476]'} to '[10.0.130.209:7476]'
2025-04-16 13:02:49.144+0000 INFO  Resolved endpoints with LIST{endpoints:'[10.0.130.209:7476]'} to '[10.0.130.209:7476]'
2025-04-16 13:03:01.854+0000 ERROR Failed to start Neo4j on 0.0.0.0:7474.
java.lang.RuntimeException: Error starting Neo4j database server at /var/lib/neo4j/data/databases
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.startDatabaseServer(DatabaseManagementServiceFactory.java:288) ~[neo4j-5.26.5.jar:5.26.5]
        at org.neo4j.graphdb.facade.DatabaseManagementServiceFactory.build(DatabaseManagementServiceFactory.java:223) ~[neo4j-5.26.5.jar:5.26.5]
        at com.neo4j.server.enterprise.EnterpriseDBMSProvider.createManagementService(EnterpriseDBMSProvider.java:39) ~[neo4j-enterprise-5.26.5.jar:5.26.5]
        at com.neo4j.server.enterprise.EnterpriseDBMSProvider.<init>(EnterpriseDBMSProvider.java:29) ~[neo4j-enterprise-5.26.5.jar:5.26.5]
        at com.neo4j.server.enterprise.EnterpriseDBMSFactory.createManagementService(EnterpriseDBMSFactory.java:19) ~[neo4j-enterprise-5.26.5.jar:5.26.5]
        at com.neo4j.server.enterprise.EnterpriseBootstrapper.createNeo(EnterpriseBootstrapper.java:25) ~[neo4j-enterprise-5.26.5.jar:5.26.5]
        at org.neo4j.server.NeoBootstrapper.start(NeoBootstrapper.java:185) [neo4j-5.26.5.jar:5.26.5]

This is a backup from the same instance of the server and database so it should be perfect.

Any help would be welcome.

2

@RichardBonnett

any errors reported in logs\debug.log?

3

Hello Dana,

The only error in the debug.log is as follows:

025-04-22 09:32:41.091+0000 INFO  [c.n.d.r.DbmsReconciler] Database 'neo4j' is requested to transition from DIRTY{db=neo4j/eb844f78} to NOT_HERE{db=neo4j/044dafe0} by SystemGraph:StoreReplaced:174 (requestCount:1)2025-04-22 09:32:41.100+0000 INFO  [c.n.c.i.EnrichmentModeWatcher] Notify listeners for DB 'neo4j' with new enrichment mode: OFF2025-04-22 09:32:41.092+0000 ERROR [c.n.d.r.DbmsReconciler] Encountered error when attempting to reconcile database neo4j (request by SystemGraph:StoreReplaced:174 (requestCount:1)) to state 'not allocated here', database remains in state 'dirty'java.util.NoSuchElementException: No value present        at java.util.Optional.orElseThrow(Optional.java:377) ~[?:?]
        at com.neo4j.dbms.reconciler.ReconcilerTransitions.forceDrop(ReconcilerTransitions.java:213) ~[neo4j-dbms-enterprise-5.26.5.jar:5.26.5]        at com.neo4j.dbms.reconciler.Transition$Prepared.doTransitionAction(Transition.java:107) ~[neo4j-dbms-enterprise-5.26.5.jar:5.26.5]        at com.neo4j.dbms.reconciler.Transition$Prepared.doTransition(Transition.java:94) ~[neo4j-dbms-enterprise-5.26.5.jar:5.26.5]        at com.neo4j.dbms.reconciler.DbmsReconciler.doTransitionStep(DbmsReconciler.java:291) ~[neo4j-dbms-enterprise-5.26.5.jar:5.26.5]        at com.neo4j.dbms.reconciler.DbmsReconciler.executeJob(DbmsReconciler.java:248) ~[neo4j-dbms-enterprise-5.26.5.jar:5.26.5]        at com.neo4j.dbms.reconciler.DbmsReconciler.lambda$scheduleReconciliationJob$2(DbmsReconciler.java:225) ~[neo4j-dbms-enterprise-5.26.5.jar:5.26.5]
        at com.neo4j.dbms.reconciler.ReconcilerJob.executeJob(ReconcilerJob.java:50) ~[neo4j-dbms-enterprise-5.26.5.jar:5.26.5]        at com.neo4j.dbms.reconciler.ReconcilerJobManager$ReconciliationWorker.run(ReconcilerJobManager.java:192) ~[neo4j-dbms-enterprise-5.26.5.jar:5.26.5]
        at org.neo4j.kernel.impl.scheduler.ThreadPool.lambda$asCallable$1(ThreadPool.java:136) ~[neo4j-kernel-5.26.5.jar:5.26.5]        at org.neo4j.kernel.impl.scheduler.ThreadPool.lambda$submit$0(ThreadPool.java:108) ~[neo4j-kernel-5.26.5.jar:5.26.5]
        at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) [?:?]        at java.lang.Thread.run(Thread.java:840) [?:?]2025-04-22 09:32:41.104+0000 INFO  [c.n.d.r.DbmsReconciler] Database 'neo4j' transition is complete from DIRTY{db=neo4j/eb844f78} to DIRTY{db=neo4j/eb844f78} (request by SystemGraph:StoreReplaced:174 (requestCount:1))

This might be of interest, if I do 'show databases' then it looks like I have got 2 system databases:

name type aliases access address role writer requestedStatus currentStatus statusMessage default home constituents
1 "neo4j" "standard" [] "read-write" *null* "primary" false "online" "unknown" "Server is unavailable" true true []
2 "system" "system" [] "read-write" *null* *null* false "online" "unknown" "Server is unavailable" false false []
3 "system" "system" [] "read-write" "10.0.148.8:7475" "primary" true "online" "online" "" false false []

It also shows I have two servers, despite this being a single stand alone DB (no clustering).

+-----------------------------------------------------------------------------------------------------+
| name                                   | address           | state     | health        | hosting    |
+-----------------------------------------------------------------------------------------------------+
| "4488b52a-46f3-4918-b891-52ea97490881" | NULL              | "Enabled" | "Unavailable" | []         |
| "6d286966-d9fa-46b8-a047-e9c3b3172129" | "10.0.148.8:7475" | "Free"    | "Available"   | ["system"] |
+-----------------------------------------------------------------------------------------------------+

The backup was taken from a different Neo4j server with exactly the same EC2 spec. I am trying to restore this onto a new Neo4j EC2 instance. Could it be that there is a ghost of the original server and the server from the restored backup?

I restored both the system and the neo4j databases when using the neo4j-admin database restore command.