cancel
Showing results for 
Search instead for 
Did you mean: 

Experiencing GC pause and high CPU

Hello,

I am in a big trouble for the last couple of weeks.

Here is the setup:

  • neo4j 4.4.6 community on a dedicated machine
  • database is small ~2-3GB
  • dotnet client 4.1.26
  • every query is parameterized
  • I have apoc plugins install but they are not in use during the query or any other system touching the database
  • this is the startup log of neo4j

I am processing ~500 messages per second which are coming from an external system. Every message generates exactly two read-only queries:

Q1:



3X_4_6_46d22fb3c97ba712aebed32ac92c8ec0aa28ac83.png

Q2:



3X_6_2_626951df06841b71ff6c1a5a59ecca8bd2bffeae.png

Observed behavior:

High CPU - despite having indices for everything I am querying for, the CPU is at 60-70% and slowly going up to 100%. I am fine with this because neo4j is working at this point.

After processing couple of million messages I see error in the logs describer here:

Detected VM stop-the-world pause:

Temp fix (for couple of hours):
The only way to resurrect neo4j is to destroy the container and create a new one. After I do this everything is back to normal.

What I have tried:

  • every possible combination tuning the heap and pagecache size. As I am writing this the configuration is --env NEO4J_dbms_memory_heap_initial__size=10G --env NEO4J_dbms_memory_heap_max__size=10G --env NEO4J_dbms_memory_pagecache_size=17G
  • machine with 8, 16, 32 GB RAM
  • machine with 2, 4, 8 CPU
  • commented dbms.jvm.additional=-XX:+DisableExplicitGC
  • Any possible combination of the 4 above.
  • The less memory given to neo4j faster it goes to GC pause

I really have no idea what I am doing wrong. Any suggestion which I could try is warmly welcome.

Thank you in advance!
mynkow

2 REPLIES 2

bennu_neo
Neo4j
Neo4j

Hi @nikolai.mynkow !

Remember that cypher will calculate trails by default, so it's expanding with the 'only' condition Don't repeat relationships. Is your graph heavily connected? Even with small databases you can eat the whole heap if your queries request a full expansion of your db several times. Based on your query (2), I feel like you can do it with

Install APOC if you haven't done it yet, then try something like:

MATCH(u:User {userId : $UserId})
with u
CALL apoc.path.spanningTree(n, {
   relationshipFilter : 'SPONSORED'
}) yield path 
with last(nodes(path)) as end,  length(path) as l
with {userId: end.userId, depth: l} as res
return res

Lemme know how it goes

Bennu

Oh, y’all wanted a twist, ey?

Hey, thank you for the suggestion.

The graph is actually a tree. There is no circles between the nodes.
I am trying to use the suggested spanningTree function but for now it creates a lot of memory pressure on the client side. Investigating why.

HERE IS A DUMP FROM FLIGHT RECORDER:
https://drive.google.com/drive/folders/1nmROQTiKm3xLk01StuQ0rcrITpgHRYNm?usp=sharing

Nodes 2022
Nodes
NODES 2022, Neo4j Online Education Summit

On November 16 and 17 for 24 hours across all timezones, you’ll learn about best practices for beginners and experts alike.