Neo4j-admin check-consistency potentially broken

Hey there everybody!

Hoping someone might have some insight on this...

I've been running into issues with the consistency checking tool in the form of using

neo4j-admin backup

and

neo4j check-consistency

What happens is when neo4j-admin backup is run with the consistency check, the consistency check makes it to about 90% and then presents the following error message:

Consistency checking failed. Full consistency check did not complete

This is occurring on a causal cluster.

Now, what I've done is run the backup command without the consistency check, and I've taken that backup data and loaded it into a local enterprise edition neo4j server.

The data loads just fine when running the 'restore' tool from neo4j-admin, but things get a little weird when attempting a backup with the local version.

The local version takes a very long time, and outputs only a series of .'s, never making it to the point where it shows a completion percentage, e.g. ".................. 10%" etc.

It produces the same error message:

Consistency checking failed. Full consistency check did not complete

Now, when running the consistency-check tool from neo4j-admin, what I have found is that the tool will make it through consistency checking if I set check-indexes=false. So there seems to be a problem with the index check.

Note, if I detach and delete all nodes from the local db (can't do this to the prod db), and then run the backup tool, it runs just fine.

Final note, I have tried running jstack and it shows many threads 'parked', which, I think is normal. I've also tried adjusting pagecache and heap size settings to no avail. (Increased them to very large sizes, that were appropriate for the local machine).

So, I'm not sure if this is a bug, but because there is no specific error message output, and because no inconsistency report is generated, it is a bit odd...

Really hoping to get this one solved, as a backup isn't really useful unless its consistency can be fully validated.

Any insight would be greatly appreciated, I know this was a long read!