Load CSV with Error Handling

Neo4j Graph Platform
, ,
1

Hi,

When i import 5 mil. rows it stops with an error that "there is a character i can't read...".
The problem is that the process stops and i want it to write the row to log and continue working.
I found an example but it doesn't work (it's complaining about the "totalOperations", so i deleted it, and also that "row.row" is type mismatch, so i changed it to "row".)
Anyway, it started working and inserted 2 mil. rows but then failed on the "charcter" problem...
Why is it stopping all the time? i put "failOnError:false" and still not working.

CALL apoc.periodic.iterate(
** "LOAD CSV WITH HEADERS FROM 'file:///path/to/your_file.csv' AS row",**
** "CREATE (p:Person {name: row.name, age: toInteger(row.age)})",**
** {batchSize:1000, parallel:false, iterateList:true,**
** failOnError:false, retries:3}**
) YIELD batch, operations, timeTaken, failedOperations, failedBatches, totalOperations, errorMessages
WITH failedOperations AS failedRows
UNWIND failedRows AS row
RETURN row.row, row.error;

2

You can try something like this to see if the error gets triggered. You should be able to see the line that was processed immediately before the error.

LOAD CSV WITH HEADERS FROM 'file:///file.csv' AS row 
return  linenumber() as line_number, row.name as name,  toInteger(row.age) as age
3

Hi thanks,

When it fails, i see the error, he can't read some row...
OK, but i want it to continue processing and not stopping the process.
In the "batch" section there is a parameter "failOnError:false" that should allow it to continue, isn't it?

BTW, the problem is with malformed rows (some character sequence it can't read).

Here is a simple script that fails:

CALL apoc.periodic.iterate(
"LOAD CSV WITH HEADERS FROM 'file:///TEST/Persons.csv' AS row RETURN row",
"MATCH(p:Person{idNumber: row.id_Number})",
{batchSize:5000, parallel:false, iterateList:true, failOnError:false, retries:0} )
YIELD batch, operations, timeTaken, failedOperations, failedBatches, errorMessages WITH failedOperations AS failedRows UNWIND failedRows AS row RETURN row;

4

Would it be better to fix the file?

I don’t see a failOnError configuration parameter in the documentation.

Maybe it is a non-recoverable error, as it can parse the rest of the file.

5

Hi,

Strange, I found the solution in google, are there other ways?

Anyway, i fixed the file but i have 10 files from different tables in the sql and there are some problematic characters in several fields for each table.

The idea is to create a schedule task to export and import the data daily, and i don't want the daily tasks to fail.
Also' if a row fails i want to log it, so i can fix what needed without losing data.