Using neo4j v5
I have a simple cron job to run a backup script every night. It runs neo4j-admin database backup --to-path="$BACKUP_DIR" "$DATABASE_NAME" (variables are set earlier in the .sh file), then uploads the new file to an azure blob.
Sometimes, at some point during this, the neo4j service stops. The log files are completely empty regarding this. There's no exceptions, no shut down message, it just stops. When I manually restart it, the start messages are just like if I stopped & started it manually, there's no recovery or anything.
Is there anything people are aware of that could cause neo4j to be stopped when attempting a backup? We're using backup rather than dump specifically because we don't want the downtime...
@henry.bruce , @dan.godbold
Are you both working together or is this 2 distinct experiences?
2 concerns.
a. sounds like maybe the there was a linux oom killer that detected starvation of available memory and decided to start killing processes. At the time Neoj stopped is there anything meaningful in /var/log/messages?
When you run backup are you providing any memory configurations?
b. the GH reference
I believe it's backing up the whole database from scratch again. But I don't see anything in our usage of the db that could cause it - we're not deleting/rebuilding vast numbers of nodes or anything, it's mostly some standard 'crud' use. And, importantly, the backup completes - we get a file at the end.
whether a backup is full or incremental is predicated on whether or not you have sufficient transaction logs to allow for a increment. If for example you perform a backup on Monday and it includes all data up until 9am Monday and on Wednesday at 9am you attempt another backup, if you current graph has transactions from Monday 9am forward then an incremental is possible. If not then a full backup is forced
Dana,
Many thanks for your inputs, Dan & I are working together.
I don't believe we're setting any memory configuration, so can try this if not. Are you able to shed any light on where/how we might find the backup logs?
Many thanks, Henry.
@henry.bruce
thanks for the detail that you and @dan.godbold are effectively on the same environement
Per Back up an online database - Operations Manual
this does describe how to configure memory for neo4j-admin database backup... if not mistaken if you run backup on the same machine where Neo4j is running then it will use the same values for min/max heap as defined for the running database itself. if for example you have 32G of RAM and you have configured conf/neo4j.conf with min/max heap of 12G, then when you start backup, the backup process is going to use 12G for heap as well.
Are you able to shed any light on where/how we might find the backup logs?
you can run neo4j-admin database backup ..... with a --verbose option. This too is described in the documentation
Some of this comes down to, if your current RAM on the server is X and you have configured the running database to use Y for min/max heap and Z for pagecache, then that leaves X-Y-Z as remaining RAM to be used by backup. Now you probably dont want to use 100% of what remains after X-Y-Z but...
Thanks again Dana, much appreciated. Will let you know how we get on.