cancel
Showing results for 
Search instead for 
Did you mean: 

Refactoring graph model to overcome super node problem

serpear
Node

Neo4j version: 4.2.3 enterprise
Driver: Python Neo4j driver (neo4j-driver 4.2.1)
Server: Single node (EC2 r5a.8xlarge) - 32vCPUs - 256 GB RAM - Amazon Linux 4.14.219-161.340.amzn2.x86_64
Heap: 31 GB
Pagecache: 203700m

Hello,

My company develops a web application and we're trying a Neo4j graph as the store. The model of the graph is something like what you see below. The data modelled is basically medical terms appeared in documents and those documents are associated to patients; the documents have different sections and a medical term appearing in one section or another is relevant. The queries we usually run are anchored in a medical concept node and traverse the graph in order to find out which documents have that concept appeared in specific sections and which patients are associated to those documents. Some volumetry of the graph is ~265M nodes and ~1.6B relationships.

3X_5_2_52444198852fd787526588f4c7ff198d84da8870.png

Usual query we run:

MATCH (parent_concept:Concept {id: "a_concept_id"})
MATCH (parent_concept)<-[:IS_A*0..]-(concept:Concept)
MATCH (concept)<-[:MENTIONS_IN_SECTION_0|...|:MENTIONS_IN_SECTION_n]-(doc:Document)
MATCH (patient:Patient)<-[:APPLIES_TO]-(doc)
<do_aggregations_on_documents_and_patients>
RETURN <results_of_aggregations>

Something I've realized is that we have different kinds of medical concepts from a performance point of view, some concepts could be called normal nodes, others maybe super nodes. Given the limitation for new users of adding more than one attachment to a post I'll try to share the profile plans of one normal concept and two super node concepts in a further comment. The above query for the normal concept takes ~300ms and for the super node concepts can take up to several seconds.

We've thought of refactoring the graph model to "break" the super nodes by basically adding new artificial concept nodes so no original or new concept node has more than 20k incoming MENTIONS_IN... relationships, something like what you see in a further comment. Unfortunately running the query above on the refactored graph has the same bad performance.

Unfortunately I can't share any sample data I import or logs.

Anyway I'd appreciate the opinion of Neo4j experts on how to improve the performance of the query for those super node concepts but any other comments about how to optimize the query, improve the model, etc are welcome.

Thanks,

6 REPLIES 6

serpear
Node

Profile plans following in next comments:

  • Normal concept

serpear
Node
  • Super node concept 1

serpear
Node
  • Super node concept 2

serpear
Node
  • Graph model refactor

tard_gabriel
Ninja
Ninja

As a first approach, I would install the APOC plugin in the database and use this procedure:

CALL apoc.meta.stats()

You will have a quick overview of your data scalability and will know how and where to optimise. Somethings wrong with the way you are querying the related relationships to your concepts. Is it possible to have the queries generating these plans, feel free to replace literal value with fake names if needed.

Note: APOC might be already installed if you are using a cloud service to host Neo4j.

anthony_gatlin
Node Clone

@serpear , I would be interested to know how you finally solved this problem or at least what techniques you used to minimize the performance impact of the super nodes.

Nodes 2022
Nodes
NODES 2022, Neo4j Online Education Summit

On November 16 and 17 for 24 hours across all timezones, you’ll learn about best practices for beginners and experts alike.