Greetings,
I have the following load csv cypher wrapped inside the apoc.periodic.iterate.
I tried to increase the batch size and I am getting locking errors. It is loading at very slow pace. What is a way to increase the loading speed and not get dead lock errors.
I tried increasing the batchsize and setting parallel to false but it is still very slow. I need to load three files like this each with 1.5 million records. It ran for almost two hours and all I could see was 500K records processed. I have a box with 24 cores and 96Gig memory. CPU is barely used.
Is there a way to make this go faster. ? Thanks a bunch in advance.
CALL apoc.periodic.iterate('
LOAD CSV WITH HEADERS FROM
"file:///<path>/filename.csv" AS line return line ','
MATCH (c:Company {name: line.COM_CODE})
MATCH (frt:FreightReadingTariff {name: toInteger(line.READ_TARIFF)})
MERGE (fb:FreightBasis {name: line.FBASIS})
CREATE (r1:Renege {ID: line.SERID + "_" + line.XXXX + "_" + line.YYYY + "_" + line.ABC + "_" + line.MNO + "_" + line.NNN,
serid: toInteger(line.SERID), xxxx: toInteger(line.XXXX), yyyy: line.YYYY, abc: line.ABC, mno: line.MNO, nnn: toInteger(line.NNN),
a1a: line.a1a, a2b: line.a2b, a2c: line.a2c, a3c: line.a3c, a4c: line.a4c, mmm: toInteger(line.MMM),
n12: toInteger(line.N12), mn1: line.MN1, eff1: line.eff1, disc1: line.disc1, ftp: line.ftp, stp: line.stp,
dowtype: line.dow, pcat: line.pcat, dcat: line.dcat, unav: line.unav, txtbl: toInteger( line.txtbl),
ch1: toInteger( line.ch1)})
MERGE (c)-[:company_to_renege]->(r1)
MERGE (fb)-[:freightbasis_to_renege]-(r1)
MERGE (frt)-[:freightreadingtariff_to_renege]-(r1)
',{batchSize:1, iterateList:true, parallel:true})