Unable to do certain writes on bigger graphs (not huge)

Neo4j Graph Platform
1

Hi everyone,

i'm currently working on neo4j 3.5.5 in docker on a server with 128gb ram and 24 threads. My graph has 11M nodes and 180M relationships. First of all my problem arised when i tried to weight my graph. Weighting my graph simply means that i write a cost as property on each relationship. This went out of memory (or i stopped it manually) after more than 12h.
I tried several things like writing on a single relationship type. This works for every relationship type singularly but not on the most frequent one which has a 140M count.
I tried to write on that type with cypher: MATCH (n)-[r:my_type]->() SET r.myproperty=some_number and i tried it with the java framework (traversing all relationships and writing on those i'm interested in).
I tried also to delete every relationship with that given type but i obtained the same results.

I tried also different memory configurations like 20gb page cache and 80gb java heap or 50gb page cache, 50gb heap, 50gb transaction max memory. Memrec is advising me to use 80gb page cache and 30gb heap which i don't understand since my graph seems to be only 16gb.

I noticed with htop that initially the cpu is working hard (with some seconds where only 1 thread is on full load while the others idle) and at some point everything goes idle beside 1 single thread. When only 1 thread is working, the database is totally stuck and i can't connect anymore (with neo4j browser or docker exec cypher-shell).

Is there anything i'm missing? Some configurations for bigger writes? It seems that I have troubles only if i exceed some threshold during my writes. I really dont want to use apoc with periodic commits (which i didn't try yet)

2

APOC and usage of periodic commit us the way so as to batch the updates into smaller sizes. Otherwise if for example you try to update millions of nodes/rels in a single transaction you will more than likely run out of heap

3

Ok thanks. Is it possible to use several "CALL apoc.periodic.commit(...)" inside a user-defined procedure? Or is a procedure seen as a whole transaction? I already tried to do several manual transactions inside a procedure but it always felt like a procedure is seen as one unique transaction.

Thank you in advance :)

4

I tried also different memory configurations like 20gb page cache and 80gb java heap or 50gb page cache, 50gb heap, 50gb transaction max memory. Memrec is advising me to use 80gb page cache and 30gb heap which i don't understand since my graph seems to be only 16gb.

how did you determien that your graph was only 16GB in size?

Further we generally do not recommend setting min/max heap to be larger than 31GB

5

Cattura

I see that in neo4j desktop/browser. Correct me if i'm wrong.