Hello people,
i have a springboot java app with the neo4j driver 5.18.0 installed and I use the local Neo4j DB on my laptop. My use case is that i keep getting thousands of incoming events (that i want to store as nodes). A real use case of my app would have for example million nodes together. The only one attribute of my event object that is unique for each event is the "timestamp". These events that i have to store in neo4j DB are related to each other, which is why i need to create reltionships between them.
In the below code, you can see that i have implemented this based on making a query for each event individually. But since i get million of nodes, the write process is very slow..the time for storing like this is growing exponentially..it gets slower and slower (e.g it needs hours just for 100K-200K )
public static void addEvent(Event event, .. workspace, .. leafNodeElement, ..leafNodeProperty) {
var eventProperties = Util.serializeEvent(event); => gives the part "{e.elementId:$elementId, e.timestamp:$timestamp...} and the param values
var params = new TreeMap<String, Object>();
params.put("wsId", workspace.id());
params.putAll(eventProperties.getValue().asMap());
//command for appending an event which is an Element
var doesLeafExist = "MATCH (ws:Workspace{wsId:$wsId})-[r:OP]->(leaf:Event) RETURN elementId(leaf)";
var foundLeafNode = runQuery(new Query(doesLeafExist, parameters("wsId", workspace.id())));
//run this if ws has a leaf node
if (foundLeafNode.hasNext()) {
var command = "MATCH (ws:Workspace{wsId:$wsId})-[r:OP]->(leaf:Event) WITH ws,leaf,r \n" +
"DELETE r WITH ws,leaf \n" +
"CREATE (ws)-[:OP]->(e:Event" + eventProperties.getKey() + ")-[:OP]->(leaf)";
if (leafNodeElement != null) {
command = "MATCH (ws:Workspace{wsId:$wsId})-[r:OP]->(leaf:Event) WITH ws,leaf,r \n" +
"DELETE r WITH ws,leaf \n" +
"CREATE (ws)-[:OP]->(e:Event" + eventProperties.getKey() + ")-[:OP]->(leaf) WITH e\n" +
"MATCH(lastElement:Event{timestamp:$lastElementTimestamp}) WITH lastElement, e\n" +
"CREATE (e)-[:ELEMENT_" + event.elementId() + "]->(lastElement)";
var elem = leafNodeElement.getData();
params.put("lastElementId", elem.getElementId().getId());
params.put("lastElementName", elem.getClass().getSimpleName());
params.put("lastElementTimestamp", elem.getTimestamp());
//need to execute another query if a lastNodeProperty also exist
if (leafNodeProperty != null) {
command = "MATCH (ws:Workspace{wsId:$wsId})-[r:OP]->(leaf:Event) WITH ws,leaf,r \n" +
"DELETE r WITH ws,leaf \n" +
"CREATE (ws)-[:OP]->(e:Event" + eventProperties.getKey() + ")-[:OP]->(leaf) WITH e\n" +
"MATCH(lastElement:Event{timestamp:$lastElementTimestamp}) WITH lastElement, e\n" +
"CREATE (e)-[:ELEMENT_" + event.elementId() + "]->(lastElement) WITH e\n" +
"MATCH(lastProperty:Event{timestamp:$lastPropertyTimestamp}) WITH lastProperty, e\n" +
"CREATE (e)-[:PROP_" + ((ElementUpdate) event).getName().replace("@", "_atsign_") + "]->(lastProperty)";
var prop = leafNodeProperty.getData(); //lastPropertyState.getLeafNode(!event.isConcluded()).getData();
params.put("lastPropertyId", prop.getElementId().getId());
params.put("lastPropertyName", prop.getClass().getSimpleName());
params.put("lastPropertyTimestamp", prop.getTimestamp());
}
}
runQuery(new Query(command, params));
} else {
var command = "MATCH (ws:Workspace{wsId:$wsId}) WITH ws \n" +
"CREATE (ws)-[:OP]->(e:Event" + eventProperties.getKey() + ")";
// "CREATE (lastElement:ElementPointer{name:'lastElement',elementId:$elementId})-[:LAST]->(e)";
runQuery(new Query(command, params));
}
}
Now for speeding up the write process, i want to store these events in batches. So I have to implement the above but for a list of events (that i need to attached to each other via :OP relationship). However, since its not just adding nodes but also the relationships, it is very complex for me to figure out how i can do this for a list of events based on batches. Because for each event node, i have to check if a leafElement for that event exists and if so then i have to add a relationship and if the event also has a leafProperty, i have to also add another extra relationship (if a leafProperty exists, then we also have a leafElement).
Can you help me with the queries or query for doing this?
public static void addEventTransaction(List<Operation> events.., workspace)
THe use case is that i get for example 1000 events in one transaction or even 3000 events.
I appreciate any help.
Thank you :)