Experiencing GC pause and high CPU

Neo4j Graph Platform Cypher
1

Hello,

I am in a big trouble for the last couple of weeks.

Here is the setup:

  • neo4j 4.4.6 community on a dedicated machine
  • database is small ~2-3GB
  • dotnet client 4.1.26
  • every query is parameterized
  • I have apoc plugins install but they are not in use during the query or any other system touching the database
  • this is the startup log of neo4j

I am processing ~500 messages per second which are coming from an external system. Every message generates exactly two read-only queries:

Q1:

image
image934×408 19 KB

image
image923×506 23 KB

image

Q2:

image
image1195×669 42.1 KB

image
image1110×908 48.2 KB

image

Observed behavior:

High CPU - despite having indices for everything I am querying for, the CPU is at 60-70% and slowly going up to 100%. I am fine with this because neo4j is working at this point.

After processing couple of million messages I see error in the logs describer here:

Detected VM stop-the-world pause:

Temp fix (for couple of hours):
The only way to resurrect neo4j is to destroy the container and create a new one. After I do this everything is back to normal.

What I have tried:

  • every possible combination tuning the heap and pagecache size. As I am writing this the configuration is --env NEO4J_dbms_memory_heap_initial__size=10G --env NEO4J_dbms_memory_heap_max__size=10G --env NEO4J_dbms_memory_pagecache_size=17G
  • machine with 8, 16, 32 GB RAM
  • machine with 2, 4, 8 CPU
  • commented dbms.jvm.additional=-XX:+DisableExplicitGC
  • Any possible combination of the 4 above.
  • The less memory given to neo4j faster it goes to GC pause

I really have no idea what I am doing wrong. Any suggestion which I could try is warmly welcome.

Thank you in advance!
mynkow

2

Hi @nikolai.mynkow !

Remember that cypher will calculate trails by default, so it's expanding with the 'only' condition Don't repeat relationships. Is your graph heavily connected? Even with small databases you can eat the whole heap if your queries request a full expansion of your db several times. Based on your query (2), I feel like you can do it with

Install APOC if you haven't done it yet, then try something like:

MATCH(u:User {userId : $UserId})
with u
CALL apoc.path.spanningTree(n, {
   relationshipFilter : 'SPONSORED'
}) yield path 
with last(nodes(path)) as end,  length(path) as l
with {userId: end.userId, depth: l} as res
return res

Lemme know how it goes

Bennu

3

Hey, thank you for the suggestion.

The graph is actually a tree. There is no circles between the nodes.
I am trying to use the suggested spanningTree function but for now it creates a lot of memory pressure on the client side. Investigating why.

HERE IS A DUMP FROM FLIGHT RECORDER:
https://drive.google.com/drive/folders/1nmROQTiKm3xLk01StuQ0rcrITpgHRYNm?usp=sharing