Transactions log errors

Neo4j Graph Platform Server
1

Hi,

I'm running Neo4j v5.17.0 enterprise via Docker and I have CDC enabled.

When checking the debug.logs, I find lots of errors like these:

2024-07-24 08:47:48.527+0000 WARN  [o.n.k.i.t.l.f.LogFileChannelNativeAccessor] [db/e570a0d6] Unable to advise sequential access for transaction log version: 0. Error: ErrorCode=-1, errorMessage='Incorrect file descriptor.'
2024-07-24 08:47:48.528+0000 WARN  [o.n.k.i.t.l.f.LogFileChannelNativeAccessor] [db/e570a0d6] Unable to advise preserve data in cache for transaction log version: 0. Error: ErrorCode=-1, errorMessage='Incorrect file descriptor.'
2024-07-24 08:47:48.534+0000 WARN  [o.n.k.i.t.l.f.c.CheckpointFileChannelNativeAccessor] [db/e570a0d6] Unable to advise sequential access for transaction log version: 0. Error: ErrorCode=-1, errorMessage='Incorrect file descriptor.'
2024-07-24 08:47:48.540+0000 WARN  [o.n.k.i.t.l.f.c.CheckpointFileChannelNativeAccessor] [db/e570a0d6] Unable to evict transaction log from cache with version: 0. Error: ErrorCode=-1, errorMessage='Incorrect file descriptor.'

I found this article from Neo4j for this one of the errors:
https://support.neo4j.com/s/article/24596542981523-explanation-of-logged-error-Unable-to-advise-sequential-access-for-transaction-log-version-0-Error-ErrorCode-1-errorMessage-Incorrect-file-descriptor

I added the JVM arguments to neo4j.conf:

server.jvm.additional=--add-opens=java.base/java.nio=ALL-UNNAMED
server.jvm.additional=--add-opens=java.base/java.io=ALL-UNNAMED 
server.jvm.additional=--add-opens=java.base/sun.nio.ch=ALL-UNNAMED

After a restart I can confirm that the arguments were added VM Arguments. However, the errors above still occur in the debug logs.

2

@chrszrkl

Not for nothing but the debug.log lines you reference are WARN and not ERROR.
Is there something else causing you to investigate debug.log lines?

3

Hi @dana_canzano

I noticed these are just warnings, but I had some errors for CDC in the debug.log and I think the warnings are somehow related to CDC failing with this error:

2024-07-24 00:19:01.643+0000 ERROR [c.n.c.CDCService] [db/0a214b04] Unable to create the log scanner for CDC
4

@chrszrkl

oh. different issue.
Are you able to attach the entire logs\debug.log?

5

@chrszrkl

The error appears to suggest some IOException when reading the txn logs? Are the txn logs present and with expected read/write permissions etc.
Anything unusual about the files in data/transactions/db/

But also it appears the database name in question is db and but 2 characters. But yet

@neo4j> create database db;
Could not create database 'db'. The provided database name must have a length between 3 and 63 characters.
@neo4j>

or maybe im completely reading this wrong

6

Hi @dana_canzano

I have shortened the database name in the logs for the sake of brevity - that's why it is only two characters long. The actual database name is fashionbeauty19qnngn

Here's the full log:

The txn files are readable-writable by user 7474 which should be the Neo4j user from Docker:

root@server:/neo4j/data/transactions/fashionbeauty19qnngn# ls -al
total 2280512
drwxr-xr-x  2 7474 7474      4096 Jul 24 13:58 .
drwxr-xr-x 19 7474 7474      4096 Jul 23 13:23 ..
-rw-r--r--  1 7474 7474      3840 Jul 25 00:16 checkpoint.0
-rw-r--r--  1 7474 7474 268734247 Jul 23 15:59 neostore.transaction.db.10
-rw-r--r--  1 7474 7474 268475789 Jul 23 16:07 neostore.transaction.db.11
-rw-r--r--  1 7474 7474 269113296 Jul 23 16:16 neostore.transaction.db.12
-rw-r--r--  1 7474 7474 268646770 Jul 23 16:23 neostore.transaction.db.13
-rw-r--r--  1 7474 7474 185238660 Jul 25 00:14 neostore.transaction.db.14
-rw-r--r--  1 7474 7474 268586955 Jul 23 15:26 neostore.transaction.db.6
-rw-r--r--  1 7474 7474 268608373 Jul 23 15:34 neostore.transaction.db.7
-rw-r--r--  1 7474 7474 268820285 Jul 23 15:42 neostore.transaction.db.8
-rw-r--r--  1 7474 7474 268955812 Jul 23 15:50 neostore.transaction.db.9
7

@chrszrkl

ah... thanks for this detail that that database name of db was not its real name. I regrettably went down a path trying to see if i could get a 2 character database name and maybe somehow this triggered the failure.

Was anything else changed?

Has CDC ever workded on db/ fashionbeauty19qnngn?

Is this db / fashionbeauty19qnngn the results of a backup/restore or dump/load ?

Have you manipulated any files under data/?

8

Sorry to confuse you with that database name :-D

Yes, it usually works without a problem.

My assumption is the following:
I executed a big transactions with lots of changes that took some time to execute. I think the checkpoint to prune the txn log files every 15 minutes somehow interfered with the running transaction, maybe even corrupting the files.

Meanwhile, I added the JVM parameters and increased the allowed log size and duration. It is currently running without issues, so maybe it was related to either the txn log file size or these arguments.