What is the ideal heap memory size for GDS in Neo4j

Neo4j Graph Platform Graph Data Science / Graph Analytics
,
1

I am using Neo4j community version with 95 GB of single RAM. I am running job to create top clusters and I have ~320 million nodes.

Here is my heap configuration looks like -
server.memory.heap.initial_size=75G
server.memory.heap.max_size=75G

Even after the above config I get heap memory related errors like below. What should be the ideal memory configuration so that we can run the top cluster calculation ?

Error -

org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 0.0 failed 4 times, most recent failure: Lost task 0.3 in stage 0.0 (TID 3) (brdn7979.target.com executor 1): java.io.IOException: org.neo4j.driver.exceptions.ClientException: Failed to invoke procedure `gds.graph.project`: Caused by: java.lang.IllegalStateException: Procedure was blocked since minimum estimated memory (23 GiB) exceeds current free memory (23 GiB).

Even though I have set the memory as 75 GB, its taking max heap as 23 GB.

2

Welcome to the community. I moved this question to our clustering category so it may be more visible. Is this problem still persisting?

3

Thanks!! Yes, I am still seeing this issue .

4

baiscally my instance has 95gb ram and neo4j conf has server.memory.heap.initial_size=28g
server.memory.heap.max_size=28g
server.memory.pagecache.size=45g memory settings which are reflecting in neo4j as well, when i querry through cypher.SHOW SETTINGS

                 YIELD name, value

                 WHERE name IN \[

                   'server.memory.heap.initial_size',

                   'server.memory.heap.max_size',

                   'dbms.memory.heap.initial_size',

                   'dbms.memory.heap.max_size',

                   'server.memory.pagecache.size'

                 \]

                 RETURN name, value;

+------------------------------------------------+

| name | value |

+------------------------------------------------+

| "server.memory.heap.initial_size" | "28.00GiB" |

| "server.memory.heap.max_size" | "28.00GiB" |

| "server.memory.pagecache.size" | "45.00GiB" |

+------------------------------------------------+. where as if i run GDS querry its failing as heap required 25gb instead of 23 gb only availabel which is 25% total memory. is any restriction at neo4j community for jvm should not keep beyond 25% of total RAM?

CALL gds.systemMonitor() YIELD jvmHeapStatus RETURN jvmHeapStatus;

+----------------------------------------------------------------+

| jvmHeapStatus |

+----------------------------------------------------------------+

| {totalHeap: "928 MiB", freeHeap: "549 MiB", maxHeap: "23 GiB"} |

+----------------------------------------------------------------+

any thing i am missing here to get min and max memory limits to alter in neo4j community edition

5

NVM, we figured out and overriden JVM parameters and it worked, Thanks

6

Awesome! Glad to hear it. @shivangi.jaiswal is this potentially a clue to your issue as well?

7

Yes, this solved my issue. Thanks @chinnidw @Ari_Neo4j

8

Awesome! Welcome both of you to the community!

9

basically i have unset explicity jvm opts and in my run-neo4j.sh before strting neo4j i just added export command as below it started working

export JAVA_OPTS="-Xms28g -Xmx28g $JAVA_OPTS"

unset JAVA_TOOL_OPTIONS

unset _JAVA_OPTIONS

echo " [run-neo4j.sh -- neo4j] in neo4j-run.sh, starting neo4j service"

/usr/lib/neo4j/bin/neo4j console &