Size limit of a singular csv file in LOAD CSV method

Neo4j Graph Platform Import / Export
, ,
1

Hey community folks,

I wanted to know that is there a limit to which the load csv can open a file.

The question is when I do a LOAD CSV does it opens the whole csv at one go or does it fetches data in chunks that can be worked upon in the configured memory ?

TIA,
Aman

2

@aman.negi

With transactions, even with a 4 GB Java heap, 100 million cases can be loaded.
If you don't write a transaction, you will get a memory error trying to process all the cases.

Subqueries in transactions

apoc.periodic.iterate

1 Like
3

So If I wrote this :

LOAD CSV WITH HEADERS FROM 'file:///biblio.csv' AS row
CALL {
	WITH row
	CREATE (p:PATENT {property : row[0]})
}
IN TRANSACTIONS OF 10000

then I won't get a memory error even if the total file size of this csv exceeds my configured memory limit.
Right ?

4

@aman.negi

In apoc.periodic.iterate, I write code like this.
This one requires less memory.

CALL apoc.periodic.iterate(
  "LOAD CSV WITH HEADERS FROM 'file:///biblio.csv' AS row RETURN row",
  "CREATE (:PATENT {property : row[0]})",
  {batchSize:10000, parallel:false});
5

hey @koji ,

thanks for the reply. I will try this and get back to you.