Java heap space error when loading a huge csv file

[andrew_bowman]

Hello,

The FOREACH in your query is causing the planner to plan an Eager operation (see the EXPLAIN of the plan and note the dark-blue Eager operator) which prevents the query from being able to periodically commit, and this ends up blowing up your heap.

For more info about what the Eager operator is, and why it causes this, see this article.

Cypher Sleuthing: the eager operator

[image] Why is it that some query syntax seems to run faster than another when there is very little difference between the statements? Sometimes, queries will invoke what is called the eager operator in order to maintain consistent operations and avoid conflicting data changes. It still leaves the question, though, of what does this eager operation do differently and why would it be important enough to specifically avoid in certain situations? What are those situations where we would want to c…

To avoid this in your query, you'll have to ditch the FOREACH. If the query you posted is the query in full, then we can just substitute with a WHERE clause for the filter, and keep the MERGE of the box and the relationship the same:

USING PERIODIC COMMIT LOAD CSV WITH HEADERS FROM 'file:///mydata.csv' AS row 
MERGE (f:Frame {id: row.id })
WITH f, row 
WHERE row.x0 IS NOT NULL
MERGE (b:Box { code:(row.x0 + ',' + row.y0 + ',' + row.x1 + ',' + row.y1) })
MERGE (f)-[r:HAS_BOX]->(b);