Creating nodes and relationships from a map with 500K items

[g.j.westfall (George J Westfall)]

Hello. I am trying to learn how to load a json file and create the nodes and relationships. My json file has ~500K messages with 5 lists each: frame, ip, upd, eth, and data. I have learned how to read in the json file and then create one (1) message in the DBMS with all the appropriate nodes and relationships (thanks to those who helped school me). Not I need to figure out how to go through the map and to that for all 500k messages. I tried a FOREACH, but it only creates the first message an nothing else. I don't know if I am using the FOREACH incorrectly or if I should be using something else. Here is the pertinent parts:

CALL apoc.load.json("file:///FileExample.json") YIELD value
UNWIND value._source.layers as bigData

//THIS OPENS THE FOREACH LOOP
FOREACH (data in bigData |
a bunch of merge and sets
)

How can I cycle through the 500K messages to get them in the DBMS?

[ameyasoft]

@g.j.westfall

UNWIND value._source.layers as bigData

Check these two links:

Loading JSON in Neo4j

Read nested json

Hi, there is an issue you should consider in your model. Currently, It is not possible to add nested property as a property (path values). So you may have to create separate nodes. UNWIND ['13171.json'] as filename CALL apoc.load.json(filename) YIELD value AS v WITH v.nets as nets UNWIND nets as net WITH net MERGE (Net:NetNode {name: net.name}) FOREACH (xyz in net.path | MERGE (Net)-[:has_xyz]->(:XYZ {x: xyz.x, y: xyz.y, z:xyz.z})) If you unpack your XYZ values into separate nodes you can …

[g.j.westfall (George J Westfall)]

Yes, I have downloaded and installed JSON.

[g.j.westfall (George J Westfall)]

Yes, I do have to be careful about that. But the main problem is not being able to go through the data and pull the next message. My code goes through one, creates the nodes and relationships and that's it. How do I get it to continue through the list creating all 500K nodes?

[g.j.westfall (George J Westfall)]

I found the solution and I am posting it here to help others and close this out. I found an example in the book "Hands-On Graph Analytics with Neo4j" (I have no affiliation). Here is what worked to go through a nested JSON file and add it to the DBMS. I do have to add the other parts, but that is just like the example.

//Load JSON file
CALL apoc.load.json("file:///FileExample.json") YIELD value AS data
//Create top level object
CREATE (l:Layers {name: "FileExample"})
//Create ETH
CREATE (e:ETH {addr_oui: data._source.layers.eth.eth.dst_tree.eh.addr.oui,
addr: data._source.layers.eth.eth.dst_tree.eh.addr
})
MERGE (l)-[:CONTAINS]-(e)
//Create UDP
CREATE (u:UDP {srcport: data._source.layers.udp.udp.srcport})
MERGE (l)-[:CONTAINS]-(u)
//Create Data
CREATE (d:Data {len: data._source.layers.data.data.len})
MERGE (l)-[:CONTAINS]-(d)
//Create Frame
CREATE (f:Frame {number: data._source.layers.frame.frame.number})
MERGE (l)-[:CONTAINS]-(f)

[ameyasoft]

@g.j.westfall

Hands-On Graph Analytics with Neo4j

Thanks for sharing this.