2021-03-12 02:47:50.956+0530 ERROR [o.n.k.i.DatabaseHealth] Database panic: The database has encountered a critical error, and needs to be restarted.. No space left on device java.io.IOException: No space left on device

Hi Everyone,

I am getting below error in debug.log.

2021-03-12 02:47:50.956+0530 ERROR [o.n.k.i.DatabaseHealth] Database panic: The database has encountered a critical error, and needs to be restarted. Please see database logs for more details. No space left on device
java.io.IOException: No space left on device

Please let me know why this error is coming and what can be done to prevent this error in coming future?

Regards
Akshat

Hi @akshat.mittal ,

Could you share some details about the DBMS installation? Are you running in Docker, using Desktop, directly using the server distribution?

And of course, which platform: Windows, Linux, MacOS?

Running out of disk space could happen because either the database itself has gotten too big, or because log files need to be rotated more often.

-ABK

Hi Andreas ,

Greetings !!!

I hope you are doing good !!!

Neo4j version 3.5.4 installed on Linux Server using tarball method.

Regards
Akshat

Hi Andreas ,

Do let me know if you need logs ?

Regards
Akshat

Hello again @akshat.mittal ,

The logs will show us the problem of what wants more disk space, but not what has taken up all the disk space.

Could you take a look at disk space generally, then also specifically within the Neo4j DBMS folder?

This is a useful general reference for diagnosing disk space utilization:

Within the Neo4j DBMS directory, try running:

du -sh *

Best,
ABK

Hi Andreas ,

Thanks for sharing the link and details.

I have gone through the link and below are stats -

[sineoj@server1 DataLoad]$ du -sh /datasan01/neo4japp/neo4j-enterprise-3.5.3/data/databases/graph.db/
215G /datasan01/neo4japp/neo4j-enterprise-3.5.3/data/databases/graph.db/

[sineoj@server DataLoad]$ ls -lrth /datasan01/neo4japp/neo4j-enterprise-3.5.3/data/databases/graph.db/neostore.transaction.db.* | wc -l
576

There are log of transaction files getting accumulated in graph.db directory.

Regards
Akshat

OK, that looks normal. It is worth setting up system monitoring to look for trends that may run into problems.

You could start with a tool like Neo4j Labs' Halin: Halin - Neo4j Monitoring Tool - Neo4j Labs

For production use, you could review the recommendations here:

Best,
ABK

Hi Andreas ,

I am aware about this Halin tool and it's very good.
But setting dbms.tx_log.rotation.retention_policy=100M size is also not deleting the file.

Regards
Akshat

The file won't be deleted. The retention policy describes how much of the file to keep.

There's a good article here: Checkpointing and Log Pruning interactions - Knowledge Base

Best,
ABK

Hi Andreas ,

Thanks for sharing the link.

So is there any parameter need to set for automatically transaction files deletion ?

Regards
AKshat