We have a file system modeled in Neo4j. The nodes can be:
Users
Groups - A user can be MEMBEROF a group
Resources - A resource can be OWNS by a user, CANREAD or CANWRITE by a group or a user, and also can be CONTAINS in another resource (capitals are labels of relationships)
We had a query which executed in a couple of milliseconds under Neo4j 3.5:
MATCH (user:User {_id:$userId})
WITH user
MATCH (resource:Resource)
WHERE resource.resourceType in $resourceTypeList AND (resource.isUserHome IS NULL OR resource.isUserHome <> true)
WITH user, resource
WHERE (user)-[:OWNS]->()-[:CONTAINS*0..]->(resource)
OR (user)-[:MEMBEROF*0..1]->()-[:CANREAD]->()-[:CONTAINS*0..]->(resource)
OR (user)-[:MEMBEROF*0..1]->()-[:CANWRITE]->()-[:CONTAINS*0..]->(resource)
RETURN count(resource)
This counts all the resources that the user can access by owning it, reading it or writing it directly, or through a group, again directly, or through a recursive folder structure.
When switching to Neo4j 5, this query suddenly runs around 3 minutes for our case (4000 users, 70000 resources)
While I was profiling the query, it seems that instead of pattern matching a cartesian product is created.
I am aware that an infinite length path can cause issues, but this was working fine previously for our data.
I am wondering if I could get back the same query execution plan as before, or how to approach the rewrite of this query.
Thanks for your input!
Can you share the profile. Btw, using a pattern as a predicate has been deprecated. The new approach is to use the 'exists' function, so wrap those patterns in your 'where' statement in 'exists' functions.
Your query gets all the eligible resource nodes and process each to filter out the ones that have relationships to the user. The efficiency of this would depend on how many resources are in the set that get filtered out, compared to how many get filtered in.
I think the below query produces the same results. It looks for the resource nodes directly and does not get a superset to filter. Maybe it executes faster. It is worth a try, assuming the results are the same. Note, the 'UNION' clause will remove duplicate resources, so the count will be unique Resource nodes.
MATCH (user:User {_id:$userId})
RETURN COUNT {
MATCH (user)-[:OWNS]->()-[:CONTAINS*0..]->(resource:Resource)
WHERE resource.resourceType in $resourceTypeList
AND (resource.isUserHome IS NULL OR resource.isUserHome <> true)
RETURN resource
UNION
MATCH (user)-[:MEMBEROF*0..1]->()-[:CANREAD|CANWRITE]->()-[:CONTAINS*0..]->(resource:Resource)
WHERE resource.resourceType in $resourceTypeList
AND (resource.isUserHome IS NULL OR resource.isUserHome <> true)
RETURN resource
} as noOfResources
Thank you for your reply, and sorry for my late response. We had some infra issues, but now I am back into fixing this problem.
Your query indeed returns the same result, this is great!
Instead of running for 3999ms as the original query, this runs now for 2146ms.
This IS an increase, but it is still too slow compared to how quick (below 500ms as I recall) the original query ran on Neo4j 3.5
For Reference, I am sharing the two profiles.
Original query:
I will try to work on this, but if you have any suggestions, I am happy to listen. Thanks for the great help!
Updated query:
BTW, the final result is 239,888, so reducing the branching to 600K+ would probably improve the performance
