Could not start the server, once the disk was full

Neo4j Graph Platform Operations
1

1). Neo4j Version: 3.4.7 (CE)

  1. Uname - Linux ip-10-1-3-221 4.4.0-176-generic #206-Ubuntu SMP Fri Feb 28 05:02:04 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

3). Server start was failing, we increased the disk size and extended the filesystem with the following message

Starting...
2020-04-01 22:59:17.091+0000 ERROR Failed to start Neo4j: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@1d3ac898' was successfully initialized, but failed to start. Please see the attached cause exception "null. At position LogPosition{logVersion=959, byteOffset=95379345} and entry version V3_0_10". Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@1d3ac898' was successfully initialized, but failed to start. Please see the attached cause exception "null. At position LogPosition{logVersion=959, byteOffset=95379345} and entry version V3_0_10".
org.neo4j.server.ServerStartupException: Starting Neo4j failed: Component 'org.neo4j.server.database.LifecycleManagingDatabase@1d3ac898' was successfully initialized, but failed to start. Please see the attached cause exception "null. At position LogPosition{logVersion=959, byteOffset=95379345} and entry version V3_0_10".

  • at org.neo4j.server.exception.ServerStartupErrors.translateToServerStartupError(ServerStartupErrors.java:68)*

4). So we deleted all neostore.transaction.db.* we were able to start the database but it has broken relationships in the graph making it unusable

We have relaunched a week-old backup on the database

We have a backup of the current version (albeit its a corrupted one) If anyone has thought on how to fix it please let me know. Is there a way to repair the corrupted? otherwise we have work cut out for us to recompute few things.

appreciate help here.

2

You should be able to set this flag in your neo4j.conf file to attempt recovery/cleanup in this situation:

unsupported.dbms.tx_log.fail_on_corrupted_log_files=false

If that doesn't work for you, then you may need to manually truncate the transaction log in question. Given this fragment from your logs:

At position LogPosition{logVersion=959, byteOffset=95379345}

We can tell this is referring to log file neostore.transaction.db.959, and the given byte offset is where we need to truncate.

We would recommend first copying this transaction log, just in case:

mv neostore.transaction.db.959 copy.neostore.transaction.db.959

Then perform the truncation from the given byte offset noted in the log message:

head -c 95379345 copy.neostore.transaction.db.959 > neostore.transaction.db.959

Aside from these narrow kinds of recovery operations, we do not recommend deleting transaction logs, as we don't know if Neo4j has yet flushed the contents of those tx logs to the store files, you may be wiping out previously committed transactions.

1 Like
3

thanks for your help