Database didn't start anymore critical error

Hi,

My database give the following error;

The database has encountered a critical error, and needs to be restarted. Please see database logs for more details.

After restart Neo4j and the full server Neo4j didn't start anymore.

Log:

2020-02-28 09:14:15.601+0000 ERROR [o.n.k.i.DatabaseHealth] Database panic: The database has encountered a critical error, and needs to be restarted. Please see database logs for more details. Failed to apply transaction: Transaction #347431025 at log position LogPosition{logVersion=5943, byteOffset=18387457} {started 2020-02-28 09:06:46.518+0000, committed 2020-02-28 09:06:46.519+0000, with 1 commands in this transaction, authored by -1, with master id -1, lock session -1, latest committed transaction id when started was 347431022, additional header bytes: 
E4
59
DE
71
BD
6D
C3
99}
org.neo4j.kernel.api.exceptions.TransactionApplyKernelException: Failed to apply transaction: Transaction #347431025 at log position LogPosition{logVersion=5943, byteOffset=18387457} {started 2020-02-28 09:06:46.518+0000, committed 2020-02-28 09:06:46.519+0000, with 1 commands in this transaction, authored by -1, with master id -1, lock session -1, latest committed transaction id when started was 347431022, additional header bytes: 

What can i do to fix my database?

I have fix it by deleting

these files.

In general we do not recommend deleting transaction logs, that can have an unexpected impact on the system. That said in this case we may need to do a manual pruning of a log file.

The important bit is the logVersion and byteOffset in this:

Failed to apply transaction: Transaction #347431025 at log position LogPosition{logVersion=5943, byteOffset=18387457}

The logVersion number signifies that the problem is in neostore.transaction.db.5943. We have the byte offset, which we can use to prune this log file, if needed, though it's supposed to be that versions after 3.4.x would attempt to recover automatically (this is after an out of disk space scenario, which you should check for and resolve first before attempting recovery).

For the following instructions, we are using the logVersion and byteOffset values from the exception message...when troubleshooting your own similar issues, use those values from your own exception message when performing the following:

To truncate, we recommend the following (as mentioned, after resolving any issue of disk space):

  1. Rename the transaction file in question, so we keep it around in case this recovery attempt fails or further analysis is needed:
mv neostore.transaction.db.5943 copy.neostore.transaction.db.5943
  1. Perform the truncation using the byteOffset (in this case 18387457), naming the truncated file so it takes the place of the original transaction file:
head -c 18387457 copy.neostore.transaction.db.5943 > neostore.transaction.db.5943
  1. Attempt startup

We also recommend reviewing your transaction log rotation and retention settings if needed. If less transaction logs are desired please correct that via these settings and restarting, since directly deleting transaction logs isn't recommended and may end up inducing corruption.

1 Like

Thank for the answer.

I have check my data and some data are now corrupt.

How can i delete the records that are corrupt? records.txt (1.3 KB)