Import query runtime grows exponentially

Neo4j Graph Platform Cypher
, ,
1

I am trying to bulk-import relationships from CSV files. The relationships are batched in groups of ~25,000 relationships per CSV file. A Cypher load query is used to read relationships from each CSV file. The query works as expected, though the runtime of the load query increases exponentially. Please see the following runtimes.

batch 1: 0.4 secs
batch 2: 0.9 secs
batch 3: 0.8 secs
...
batch 120: 3.9 secs
...
batch 200: 45.6 secs

The load query that is executed for each CSV file.

LOAD CSV WITH HEADERS FROM 'file:relationships_batch_1.csv' AS row
MATCH (x:Person {eid:row.xEid}), (y:Person {eid:row.yEid}) 
MERGE (x)-[r:`row.Type` {Price:toFloat(row.Price), Num:toInteger(row.Num)}]->(y)

where row.Type can have three possible values, hence not sure creating an index on that could improve performance.

For performance reasons, I define the following index on Person.

CREATE INDEX EidIndex
IF NOT EXISTS
FOR (p:Person) ON (p.eid)
2

What piece of the relationship pattern would you like to identify uniqueness of the relationship? Currently the combination of row.Type, row.Price and row.Num are used to identify uniqueness (because they are all specified in the MERGE statement and therefore every relationship needs to be checked and the relationship property values compared to the values in row for each row, which helps explain why you're seeing the slowdown.

If instead only the relationship type identifies uniqueness of the relationship you could try moving the relationship properties out of the MERGE and into a SET. Here's a version of that using apoc.merge.relationship:

LOAD CSV WITH HEADERS FROM 'file:relationships_batch_1.csv' AS row
MATCH (x:Person {eid:row.xEid})
MATCH (y:Person {eid:row.yEid}) 
CALL apoc.merge.relationship(x, row.Type,
  {},
  {price: toFloat(row.Price), num: toInteger(row.Num)},
  y,
  {}
)
YIELD rel

I used apoc.merge.relationship here because otherwise I don't think the value of row.Type is actually being picked up in your MERGE statement and rather your relationship type is always being set as the string value "row.Type"

3

Thanks for the suggestion. I tried this, but unfortunately, it did not help. In fact, I reached ~10 sec per batch at batch 40, while previously I was at ~4 secs at batch 120.

Additionally, I need to use Type, Price and Num from my first example to identify the uniqueness of the relationship, which is not covered in your suggested solution.