Neo4j crashing regularly because of "OutOfMemoryError"

General Newbie Questions
1

Hello,

I'm a beginner, not even a professional programmer: I script tools for CG software, we need a db to store and track production data like assets and shots and Neo4j looked perfect for this. Yet the learning curve is a bit steep for me, for I need to handle many new problematic at the same time, and here's one more:

Our server, a DigitalOcean droplet with 2GB memory, has been crashing roughly every two days for a month. Always for the same reason: OutOfMemoryError.

Oct 08 08:50:37 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.VmPauseMonitor-1"
Oct 08 08:52:31 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.Scheduler-1"
Oct 08 08:52:38 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1979980170-10360"
Oct 08 08:52:41 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "qtp1979980170-10599"
Oct 08 08:54:15 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.BoltWorker-5 [bolt]"
Oct 08 08:54:48 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Session-HouseKeeper-6c9bf3b5"
Oct 08 08:55:17 Database neo4j[1286]: 2020-10-08 08:55:17.222+0000 ERROR Unexpected error detected in bolt session 'bolt-3552'. Java heap space
Oct 08 08:55:17 Database neo4j[1286]: java.lang.OutOfMemoryError: Java heap space
Oct 08 08:55:18 Database neo4j[1286]: Exception in thread "neo4j.BoltNetworkIO-1" java.lang.OutOfMemoryError: Java heap space
Oct 08 08:58:50 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.BoltNetworkIO-5"
Oct 08 08:58:53 Database neo4j[1286]: Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "neo4j.BoltWorker-3 [bolt]"

I understand that 2GB of memory for a server that is queried by 40 users all day long is not much, but for now, I have to stick to this configuration, so I have several questions:

  • How can I troubleshoot this ? Maybe it's one specific query that causes that ?
  • Are there some settings in the Neo4j configuration to avoid this ?
  • Is there a cleaner way than rebooting the machine every time it happens to "fix" the server ?

I found this to help me understand, but I don't really know what to do with that :confused:

Thanks a lot!

2

As the error says the heap memory is not enough for all the queries to run. Most likely reason is a query is using up too much memory. Enable query logs and see how much memory is being consumed by each query. You might have to tune or adjust the queries using too much memory.

3

Thanks !

Regarding enabling query logging. Do I just add dbms.logs.query.enabled=VERBOSE in my neo4j.conf?
I'm confused because the doc says that VERBOSE is already the default value, but I see no query.log in /var/log/neo4j/.

4

I actually just saw that this page was labelled as "Enterprise Edition". Does that means that there is no query logging for the basic edition ?

5

Currently, that's correct. That may change in the future.

You should look into the queries that you're running. Use EXPLAIN and PROFILE to review the query plans, and if you realize that certain queries are contributing to the error, ask for help for tuning them.

If you're using 4.1, there are some config properties that may help. Check the following config properties for those that begin with dbms.memory.transaction:

As for what happens when the heap goes out of memory, unfortunately you must restart.

6

Thanks a lot!

I did not know about EXPLAIN and PROFILE, so I will explore this :slight_smile:

7

Hi,

I think that a server with more memory may be in order.
You can change the memory settings (even in the basic community edition)

The three settings are (I have guess at something that might work with only 2GB of RAM), but this is tiny.
dbms.memory.heap.initial_size=256m
dbms.memory.heap.max_size=512m
dbms.memory.pagecache.size=512m
Or...
Did you know that you can download Neo4j Desktop for free? It has a no cost license for development on your laptop or other Windows/Mac or Linux device. This device may have much more memory than the 2GB droplet.

8

Please run memrec tool and then just set appropriate memory parameters.

9

Usually, this error is thrown when the Java Virtual Machine cannot allocate an object because it is out of memory, and no more memory could be made available by the garbage collector.

Therefore you pretty much have two options:

  • Increase the default memory your program is allowed to use using the -Xmx option (for instance for 1024 MB: -Xmx1024m)

  • Modify your program so that it needs less memory, using less big data structures and getting rid of objects that are not any more used at some point in your program

Increasing the heap size is a bad solution, 100% temporary. It will crash again in somewhere else. To avoid these issues, write high performance code.

  • Use local variables wherever possible.
  • Make sure you select the correct object (EX: Selection between String, StringBuffer and StringBuilder)
  • Use a good code system for your program(EX: Using static variables VS non static variables)
  • Other stuff which could work on your code.
  • Try to move with Multy Threading