Importing Relationships / Nodes very slow

Neo4j Graph Platform Import / Export
, ,
1

Hi,

Before posting this I searched for a similar topic and the closest is this one but unfortunately there is not reply since more than 1 year.

here is my setup :
Neo4j 3.5.14
Ubuntu 18.04.3
Intel(R) Xeon(R) W-2195 18 cores 36 threads
250 GB RAM

informations about my Neo4J DB :
total size : 298 GB
number of nodes: 399 027 743
number of relationship: 818 190 432
number of properties: 1 951 751 856

and the conf file about the RAM :
dbms.jvm.additional=-XX:+UseG1GC -Xms32g -Xmx160g

I let memory.heap conf with the default parameter (dynamically calculated based on available system resources).

I need to import more relationship and data every weeks but i'm facing performance issues. This is not happening everytime, i could import data very fast before (15k/s nodes or relationship insertion) but now it can no longer go beyond 1k nodes/s and 50-100 relationship/s.

This is my cypher command i use to import data :

PROFILE USING PERIODIC COMMIT 100
LOAD CSV WITH HEADERS FROM 'file:///exports/rel/test.csv' AS row
FIELDTERMINATOR '|'
WITH row WHERE row.END_ID IS NOT NULL AND row.START_ID IS NOT NULL
MATCH (vir:Virement_SEPA {vir_sepa_id: row.START_ID})
MATCH (rib:RIB { rib_id: row.END_ID })
MERGE (vir)-[:A_RECU]->(rib)

This is the PROFILE with only 100 rows :

plan (1)
plan (1)1125×1376 87.7 KB

Did i do something wrong ?
Thanks for helping

2

I'm trying to merge my relationships with my entire file (30 millions rows) and this happened :
At the beggining (20-30 minutes ago) i was stuck at 300-500 new relationship per second and now i can reach 15-20k new relationship per second. I don't understand what makes this difference but at least it's now importing at normal speed.

3

Had you refresh the browser before second iteration?

4

Hi Vivek, i don't remember but i think i didn't refreshed my browser before.

ps: it took about 1.5 hours to import 30 millions new relationship