Hi, Newbie here. For a story I'm trying to import 330.000 json documents on academic papers, with relationships between papers, authors, institutions, funding and patents. I'm using apoc load json for this, but it's terribly slow, it's starts out pretty slow already, but crawls almost to a halt. A…

Hi Dimitry Please check the reference from feb 2020 similar to the case you describe. [image] Json loading is slow after some time Import / Export Hello Everyone, I am trying to load some thousands JSON in neo4j with the help of python script and apoc.json.load. Init…

Thanks Refeaeli, I did read that and tried some of the suggestions. One of the cases is pretty similar to mine and I rewrote part of the query, but to no avail.

what indexes are on :Paper :Funding :Author :Institution

Hi Dana, I see that I have one duplicate index on author, but here's the rest. authorInstitutionNameIndex BTREE NONUNIQUE NODE [ "Author" ] [ "nameInstitution" ] ONLINE authors BTREE NONUNIQUE NODE [ "Author" ] [ "name", "nameInstitution" ] ONLINE constraint_1439150b BTREE UNIQUE NODE [ "Paper" ] …

@dimitri.tokmetzis unless mistaken I'm not seeing an index on :Funding(name) which would be of benefit for MERGE (f:Funding {name: COALESCE(fund.org, 'NULL')}) ON CREATE SET ... ...

Yes, that's true, thank you. I'm trying that now, but it still is very slow....

If you have all your indexes correct and that is not what is slowing down the merges, then I would think the problem is you are bringing the entire JSON into memory and committing it in one go. To that end I would recommend you use apoc.periodic.iterate to create batches. so something like: CALL …

Dimitri, I would try creating a temporary node for each document with all relevant JSON fields imported as properties of the node. Once the data is imported into neo4j, it's much faster to perform the parsing functions you have listed to create the specific nodes you need rather than performing the…

thanks for the suggestion. I discovered a small error in my index and decided to load the json through python and that worked a lot quicker. In the end, it still took 7 hours, but that is manageble for now. I will try your suggestion to shave off some extra hours though.

How to speed up apoc json load

Neo4j Graph Platform Procedures & APOC

reffaeli1 (Reffaeli) October 4, 2021, 6:34am 2

Hi Dimitry
Please check the reference from feb 2020 similar to the case you describe.

Regards

Topic		Replies	Views
Json loading is slow after some time Import / Export	19	2365	October 25, 2020
Need help to optimize json load performance Neo4j Graph Platform	11	372	February 8, 2024
How to batch json records using apoc library for better importing? Procedures & APOC apoc , performance , cypher , import , json , batching	4	1402	October 16, 2021
Slow merge and match when I use unwind Neo4j Graph Platform migrated	10	148	November 14, 2022
Confused about performance Neo4j Graph Platform	2	552	January 16, 2020

How to speed up apoc json load

Related topics