We’re developing a tool that uses Neo4j (Community Edition) to analyze workflows represented as graphs. To analyse the data, we first import the data into the Neo4j. The process consists of:
Importing data from SQL databases into Neo4j (first ~10 queries)
Preparation steps on the data (setting properties, aggregations, helper nodes/relationships)
Each import starts with an empty Neo4j instance. The full preparation phase includes ~400 Cypher queries executed in a fixed order. All queries:
Mutate the graph (i.e., write/update)
Use CALL { ... } IN TRANSACTIONS OF 5000 ROWS
Some use IN CONCURRENT TRANSACTIONS to speed things up
This usually completes in ~4 minutes. However, we’re encountering an intermittent issue:
Sometimes, one query (usually after query 300+) hangs for 1–2 hours, then completes.
However, if we cancel and re-run it (with the same data and parameters), it finishes in seconds.
Some context:
The problem is not tied to dataset size; even small datasets can trigger it.
The same dataset may work fine one time, and hang the next.
The problem only occurs after the SQL import phase, once all data is already in Neo4j. Thus, possible connection problems are not the cause.
The query that hangs varies, and doesn’t necessarily involve complex logic.
Not all queries that hang are using CONCURRENT TRANSACTIONS. Thus, I think it is not related to a deadlock.
We’ve noticed that adding a small delay between phases reduces the frequency of these issues. Even though we could do this, it would be more like a workaround then a solution.
Has anyone experienced similar symptoms?
What could be the reason for the occasionally long running queries.
What are the best practices for concurrency and batching in Community Edition?
Are there tuning strategies (other than switching to Enterprise) that help mitigate this?
Currently we use Neo4j 5.24.2, but we saw the same problem also with older versions
Unfortunately, it does not always hang on the same query. But as an example this is the query that hanged the last time:
MATCH (:`Dimension` {id: <id>})<-[:`PART_OF_DIMENSION`]-(caseId)
WHERE exists((caseId)-[:`HAS_VIOLATIONS`]->())
WITH caseId
CALL {
WITH caseId
OPTIONAL MATCH (caseId)<-[:`PART_OF_CASE`]-()-[r1:`BEFORE`|`BEFORE_START` {suspended: true}]->()-[r2:`BEFORE` {reverse: true}]->()
SET r1.disarray = not r1.simplification,
r2.disarray = not r2.simplification
SET r1.violation = r1.disarray, r2.violation = r2.disarray
WITH caseId
OPTIONAL MATCH (caseId)<-[:`PART_OF_CASE`]-()-[r1:`BEFORE` {reverse: true}]->()-[r2:`BEFORE`|`BEFORE_END` {suspended: true}]->()
SET r1.disarray = not r1.simplification,
r2.disarray = not r2.simplification
SET r1.violation = r1.disarray, r2.violation = r2.disarray
}
IN TRANSACTIONS OF 5000 ROWS
Can you replace the two optional match with a union/union all to do the search first and apply the set ops after ?
I suppose in concurrent transactions, Neo4j has to put eager operators and you might encounter potential deadlocks. It depends on the degree node, Neo4j might need to lock the relations and the nodes before an update.
Or (maybe dumb, but i can't test it and it doesn't throw an error), remove the direction of the relationships, since it seems that's all those 2 statements are doing:
MATCH (:`Dimension` {id: <id>})<-[:`PART_OF_DIMENSION`]-(caseId)
WHERE exists((caseId)-[:`HAS_VIOLATIONS`]->())
WITH caseId
CALL {
WITH caseId
MATCH (caseId)<-[:`PART_OF_CASE`]-()-[r1:`BEFORE`|`BEFORE_START` {suspended: true}]-()-[r2:`BEFORE` {reverse: true}]-()
SET r1.disarray = not r1.simplification,
r2.disarray = not r2.simplification
SET r1.violation = r1.disarray, r2.violation = r2.disarray
}
IN TRANSACTIONS OF 5000 ROWS
