Neo4j crashing regularly because of "OutOfMemoryError"

Hello,

I'm a beginner, not even a professional programmer: I script tools for CG software, we need a db to store and track production data like assets and shots and Neo4j looked perfect for this. Yet the learning curve is a bit steep for me, for I need to handle many new problematic at the same time, and here's one more:

Our server, a DigitalOcean droplet with 2GB memory, has been crashing roughly every two days for a month. Always for the same reason: OutOfMemoryError.

Oct 08 08:50:37 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.VmPauseMonitor-1"
Oct 08 08:52:31 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.Scheduler-1"
Oct 08 08:52:38 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1979980170-10360"
Oct 08 08:52:41 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1979980170-10599"
Oct 08 08:54:15 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.BoltWorker-5 [bolt]"
Oct 08 08:54:48 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Session-HouseKeeper-6c9bf3b5"
Oct 08 08:55:17 Database neo4j[1286]: 2020-10-08 08:55:17.222+0000 ERROR Unexpected error detected in bolt session 'bolt-3552'. Java heap space
Oct 08 08:55:17 Database neo4j[1286]: java.lang.OutOfMemoryError: Java heap space
Oct 08 08:55:18 Database neo4j[1286]: Exception in thread "neo4j.BoltNetworkIO-1" java.lang.OutOfMemoryError: Java heap space
Oct 08 08:58:50 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.BoltNetworkIO-5"
Oct 08 08:58:53 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.BoltWorker-3 [bolt]"

I understand that 2GB of memory for a server that is queried by 40 users all day long is not much, but for now, I have to stick to this configuration, so I have several questions:

  • How can I troubleshoot this ? Maybe it's one specific query that causes that ?
  • Are there some settings in the Neo4j configuration to avoid this ?
  • Is there a cleaner way than rebooting the machine every time it happens to "fix" the server ?

I found this to help me understand, but I don't really know what to do with that :confused:

Thanks a lot!

As the error says the heap memory is not enough for all the queries to run. Most likely reason is a query is using up too much memory. Enable query logs and see how much memory is being consumed by each query. You might have to tune or adjust the queries using too much memory.

Thanks !

Regarding enabling query logging. Do I just add dbms.logs.query.enabled=VERBOSE in my neo4j.conf?
I'm confused because the doc says that VERBOSE is already the default value, but I see no query.log in /var/log/neo4j/.

I actually just saw that this page was labelled as "Enterprise Edition". Does that means that there is no query logging for the basic edition ?

Currently, that's correct. That may change in the future.

You should look into the queries that you're running. Use EXPLAIN and PROFILE to review the query plans, and if you realize that certain queries are contributing to the error, ask for help for tuning them.

If you're using 4.1, there are some config properties that may help. Check the following config properties for those that begin with dbms.memory.transaction:

As for what happens when the heap goes out of memory, unfortunately you must restart.

Thanks a lot!

I did not know about EXPLAIN and PROFILE, so I will explore this :slight_smile:

Hi,

I think that a server with more memory may be in order.
You can change the memory settings (even in the basic community edition)


The three settings are (I have guess at something that might work with only 2GB of RAM), but this is tiny.
dbms.memory.heap.initial_size=256m
dbms.memory.heap.max_size=512m
dbms.memory.pagecache.size=512m
Or...
Did you know that you can download Neo4j Desktop for free? It has a no cost license for development on your laptop or other Windows/Mac or Linux device. This device may have much more memory than the 2GB droplet.