Hey guys,
I have neo4j 5.16.0 Community installed in a docker container fashion. While making the container live, I mounted the data and import folders to a physical drive outside of the container as follows:
docker run -d --publish=8082:7474 --publish=8084:7687 --env NEO4J_AUTH=none --volume=/data01/ds:/data --volume=/data01/ds/import:/import neo4j
I did a pilot by placing files in the import folder, and importing it via the LOAD CSV command in the neo4j browser interface.
Once the pilot concluded, I DELETE DETACHed the graph, copied the import folder to an archive location, and created a new import folder.
Placed the csv files for a new graph, however the same was not accessible via the LOAD CSV command on the neo4j browser. Hence, I used the python driver to populate the nodes.
However, after 900k nodes were added the following error showed up:
DatabaseError: {code: Neo.DatabaseError.Transaction.TransactionLogError} {message: Could not append transaction: Transaction #904780 {started 2024-02-06 14:37:18.893+0000, committed 2024-02-06 14:37:18.894+0000, with 5 commands in this transaction, lease -1, latest committed transaction id when started was 904779, consensusIndex: -1} to log.}
When I tried deleting any node, the following error showed up on the neo4j browser:
Could not append transaction: Transaction #904782 {started 2024-02-06 19:35:04.950+0000, committed 2024-02-06 19:35:06.197+0000, with 2706008 commands in this transaction, lease -1, latest committed transaction id when started was 904779, consensusIndex: -1} to log.
Given the error message, it seemed to be a transaction log specific error. Given that the transaction log is created in the data folder (that mounted on my physical drive), I checked if there was space available. I found that only 1% of it was consumed. Next I checked the config file, which had the following parameters set for the relevant settings:
- db.tx_log.buffer.size: "4194304"
- db.tx_log.rotation.retention_policy: "2 days 2G"
- db.tx_log.rotation.size: "256.00MiB"
Even though the rotation size was set to 256 MBs, it created a single transaction log file of 565 MBs in data/transactions/neo4j.
Next I stopped the container and restarted it. When I tried displaying 100 nodes, the following error showed up:
The count value for key 'CountsKey[type:1, first:-1, second:0]' is invalid. This is a serious error which is typically caused by a store corruption
How to fix these issues?