Neo4j can't recover after running out of space

Neo4j Graph Platform
,
1

Hi all,

The Issue
I am running neo4j 4.4 enterprise on an Ubuntu ARM machine. I recently had a very large insertion job which caused my server to run out of disk space due to massive amounts of transaction logs. The error I get is Database does not exist. Database name: 'neo4j'.. I also checked the logs and found that by neo4j db has been quarantined.

SHOW DATABASES gives this error: Automatic quarantine because of panic: Failed to apply transaction: Transaction #1627569 at log position LogPosition{logVersion=6624, byteOffset=140746438} {started 2023-04-24 06:34:42.129+0000, committed 2023-04-24 06:34:42.468+0000, with 15870 commands in this transaction, lease -1, latest committed transaction id when started was 1627568, additional header bytes: }".

At this point, I don't care about recovering any transactions. Any way to manually purge them and recover from quarantine?

What I have Tried

  1. I know I am not supposed to touch transaction logs, but in an effort to recover the DB I moved the earliest ones to a different disk, and restarted neo4j after dozens of GB were freed. Still no progress so I moved the TX logs back to the proper directory. FYI, I do not have enough space on the other disk to move all transactions or the database itself.
  2. I set the retention policy to keep_none and used a checkpoint of 30s but no transactions have been purged by the DBMS.
2

Hello @alex3 :slight_smile:

You should never touch the transactions files.

The only solution I see is to increase the disk space, let the database do the recovery then purge the transactions with the retention policy.

Best regards,
Cobra

3

Thanks! I moved my transactions to a new folder on a bigger EBS drive on my AWS EC2 instance. I was constrained before since the tx disk was a capped NVME drive. Then I restarted after changing directory in the neo4j.conf and all good.

I did tell the system db to unquarantine my neo4j db but not sure if that was necessary. Cheers!

4

I'm happy to hear that :slight_smile:

When you start the DBMS, if my memory is good, I believe it tries to do the recover the database which need it.

Best regards,
Cobra