Failure to dump database, integer overflow

Neo4j Graph Platform
1

I am attempting to create backups for our production neo4j database and am running into an issue I cannot seem to find elsewhere. Taking the database offline and running the neo4j-admin dump process fails with an integer overflow error.

The neo4j-admin copy command is working without problem and is our current backup solution, but is far less convenient since it doesn't retain the indexes. My speculation is that it may have to do with the size of the database and trying to retain the indexes. It is roughly 500GB without indexes, 1.6TB with indexes built. There should be enough hardware to handle this, though we have had to increase the java heapsize to 50GB. No run-time problems. This unusual heapsize is another concern I have with regards to this neo4j-admin dump failure, but I'm not sure if it's related. I know very little about Java.

neo4j 5.4.0
ubuntu 22.04
4TB ram
256 cores

The command used:

sudo neo4j-admin database dump dummydbname --to-path=/backup/

The output:

2023-12-13 04:00:13.530+0000 ERROR [c.n.c.d.EnterpriseDumpCommand] Failed to dump database 'dummydbname': integer overflow
2023-12-13 04:00:13.532+0000 ERROR [c.n.c.d.EnterpriseDumpCommand] Dump failed for databases: 'dummydbname'
org.neo4j.cli.CommandFailedException: Dump failed for databases: 'dummydbname'
		at org.neo4j.commandline.dbms.DumpCommand.execute(DumpCommand.java:199)
		at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:92)
		at org.neo4j.cli.AbstractCommand.call(AbstractCommand.java:37)
		at picocli.CommandLine.executeUserObject(CommandLine.java:2041)
		at picocli.CommandLine.access$1500(CommandLine.java:148)
		at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2461)
		at picocli.CommandLine$RunLast.handle(CommandLine.java:2453)
		at picocli.CommandLine$RunLast.handle(CommandLine.java:2415)
		at picocli.CommandLine$AbstractParseResultHandler.execute(CommandLine.java:2273)
		at picocli.CommandLine$RunLast.execute(CommandLine.java:2417)
		at picocli.CommandLine.execute(CommandLine.java:2170)
		at org.neo4j.cli.AdminTool.execute(AdminTool.java:94)
		at org.neo4j.cli.AdminTool.main(AdminTool.java:82)
Caused by: java.lang.ArithmeticException: integer overflow
		at java.base/java.lang.Math.addExact(Math.java:883)
		at org.apache.commons.compress.utils.ExactMath.add(ExactMath.java:43)
		at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.closeArchiveEntry(TarArchiveOutputStream.java:439)
		at org.neo4j.dbms.archive.Dumper$ArchiveOperation.addToArchive(Dumper.java:199)
		at org.neo4j.dbms.archive.Dumper.dump(Dumper.java:121)
		at org.neo4j.commandline.dbms.DumpCommand.dump(DumpCommand.java:268)
		at org.neo4j.commandline.dbms.DumpCommand.execute(DumpCommand.java:170)
		... 12 more
		Suppressed: java.io.IOException: This archive contains unclosed entries.
				at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.finish(TarArchiveOutputStream.java:297)
				at org.apache.commons.compress.archivers.tar.TarArchiveOutputStream.close(TarArchiveOutputStream.java:315)
				at org.neo4j.dbms.archive.Dumper.dump(Dumper.java:118)
				... 14 more

It may or may not be relevant, but the current running database is from a restored old file-system copy (I am aware this is not supported). We had managed to use neo4j-admin copy command on the old file-system copy to restore the data successfully and have no other issues since.

2

@nicholas.guy.shaffer

Community edition or Enterprise?

Has neo4j-admin database dump ever succeeded?

A search of This archive contains unclosed entries. through google seems to suggest errors related to file sizes etc

3

Enterprise version

neo4j-admin database dump hasn't ever succeeded as far as I'm aware for the dataset in question

4

@nicholas.guy.shaffer

Given you are running Neo4j 5.4.0 and to which we believe this is addressed by Neo4j 5.7.0 and later, and of which are latest is 5.15.o could you please upgrade to 5.7.0 or later and let us know if this is resolved

5

Thanks for the reply. Neo4j has been upgraded to 5.15.0 and that seemed to have solved the issue. The consistency checker was also ran to make sure it ran right and it did so without error.

The checker did leave thousands of these "referenced record is not in use" warnings, below is a just the first few lines, but I am assuming this is an issue unrelated to the original.

2024-01-04 01:38:38.461+0000 WARN  The schema rule has a reference to another schema rule that is not online.
	SchemaRecord[5,used=true,created=false,nextProp=6490979572,constraint=false,secondaryUnitId-1,fixedReferences=false] (Index( id=5, name='tx_id_constraint', type='RANGE', schema=(:Transaction {tx_id}), indexProvider='range-1.0' ))
	Inconsistent with: Index( id=5, name='tx_id_constraint', type='RANGE', schema=(:Label[1] {PropertyKey[5]}), indexProvider='range-1.0' )
2024-01-04 01:48:48.437+0000 WARN  The referenced record is not in use but the id is not up for reuse in the id file.
	Node[2820000062,used=false,created=false,rel=0,prop=0,labels=Inline(0x0:[]),light,fixedRefs=false]
6

i suspect 2 different issues.
If you drop and recreate the constraint named tx_id_constraint does the WARN cease?