# Graph Data Modeling Question **URL:** \ **Category:** Modeling\ **Tags:** performance, neo4j-desktop, modeling, data-modeling\ **Created:** [December 17, 2020, 7:08am UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522 "2020-12-17T07:08:28Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![TonyOD27](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/tonyod27/32/5200_2.png) [@TonyOD27](https://community.neo4j.com/u/TonyOD27)\ **Post date:** [December 17, 2020, 7:08am UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/1 "2020-12-17T07:08:28Z")
neo4j specifications: dbms.memory.heap.initial\_size=24G dbms.memory.heap.max\_size=24G dbms.memory.pagecache.size=8G neo4j version: Community 4.2.0 desktop version: 1.3.11 Hello, How can I improve on my data model to reduce the number of relationships? Currently, my graph database contains a total of 300,000 nodes connected by about 550,000,000 relationships. After debating the graph data model with my colleagues for several weeks and performing numerous refactorings, I can't figure out a way to diminish the number of relationships between a subset of nodes on the part of my knowledge graph that is illustrated below. I'm hoping that by reducing the number of relationships I can speed up the performance of my cypher queries which are beginning to take more than hour to finish as they become more complex. Here's a proxy example of my current use case: ![image](https://us1.discourse-cdn.com/flex021/uploads/neo4jcommunity/original/2X/1/16fd53bdbcce1e948dc4c4a26ce98dde562c02d3.png) In this example, here are the counts for each of the nodes and relationships: **Nodes** Person: 100,000 nodes with 5 properties A\_Law: 5,000 nodes with 4 properties B\_Law: 3,000 nodes with 4 properties C\_Law: 2,000 nodes with 4 properties Business: 7,000 nodes with 3 properties **Relationships** A\_Access: 500,000,000 relationships (every person node is connected to every single unique A\_Law node. Therefore, we have 100,000 times 5,000 is 500,000,000) with 0 properties B\_Access: 2,000 relationships with 0 properties C\_Access: 3,000 relationships with 0 properties A\_Rel: 5,000 relationships with 2 properties B\_Rel: 3,000 relationships with 2 properties C\_Re;: 2,000 relationships with 2 properties As you can see above, the problem here is with the number of the A\_Access relationship that has exploded as a result of the product between all of the person nodes to all of the A\_Law nodes. Our business domain requires that each of the unique person nodes must have one relationship to each of the unique A\_Law nodes. This results in 500,000,000 relationships and this is causing our cypher queries to take more than an hour to finish. Another approach that I'll implement here is to add more indices to the properties of the nodes or relationships, but I would like to be careful when adding various indices. Here's a warning I read about adding too many indices: [Welcome to the Dark Side: Neo4j Worst Practices (& How to Avoid Them)](https://neo4j.com/blog/dark-side-neo4j-worst-practices/) I've tried profiling and explaining our queries and I see that there are billions of rows being accessed. I also read and watched the posts and videos listed below multiple times, but I still don't know how else to improve on the data model: > **[Graph Data Modeling for Neo4j - Graph Data Modeling for Neo4j](https://neo4j.com/graphacademy/training-gdm-40/00-graph-data-modeling-about/)** > **[Graph Databases for Beginners: Data Modeling Pitfalls to Avoid - Neo4j Graph...](https://neo4j.com/blog/data-modeling-pitfalls/)** > > Learn how to avoid these common data modeling design mistakes and mess-ups made often by beginners – especially when you're using a graph database. > **[Tips & Tricks: Data Modeling Best Practices \[Neo4j Ninjas Exclusive\]](https://neo4j.com/videos/tips-tricks-data-modeling-best-practices-neo4j-ninjas-exclusive/)** > > Advanced level webinar presented by Max De Marzi (Field Engineer & Graph Database Expert @ Neo4j) More info on Noe4j Ninja’s Program: https://community.neo4j.com/t5/ninjas-program/ct-p/ninjas Access exclusive advanced Neo4j webinars &... > **[Tips and Tricks for Graph Data Modeling](https://neo4j.com/videos/tips-and-tricks-for-graph-data-modeling-2/)** > > Mark Needham, Neo4j Engineer:This webinar will walk you through how to model your data as a graph. We will demonstrate how to avoid pitfalls early on and how to optimize your model for answering questions as Cypher queries. > **[8 Solid Tips for Succeeding with Neo4j - Neo4j Graph Data Platform](https://neo4j.com/blog/8-tips-succeeding-with-neo4j/)** > > Watch (or read) Neo4j's Executive Manager Chris Leishman present on eight tips to help you achieve optimal performance with Neo4j. I also read the entire O'Reilly book on Graph Databases, but I still don't know how to improve my data model. Thanks for any ideas or suggestions. -Tony ---
**Author:** ![senthilc78](https://avatars.discourse-cdn.com/v4/letter/s/3ab097/32.png) [@senthilc78](https://community.neo4j.com/u/senthilc78)\ **Post date:** [December 17, 2020, 2:27pm UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/2 "2020-12-17T14:27:20Z")
> [@TonyOD27](#): > > Our business domain requires that each of the unique person nodes must have one relationship to each of the unique A\_Law nodes Hi , though i may yet get complete understanding on this domain/context but few questions here pls 1. What is the query you are trying to get from here - Any Data Model - Best way to design based on the Buesiness Question you are trying to do 2. From your Business need, I understand you need to have unique relations for each node. This leads to me to think -unique ID for each person right? so better to have this as property in my the Node itself - instead of relations which actually reduces all these millions of paths Based on my experience , i wud suggest, 1. Validate when we can turn property to Node or Node to Property- Based on whats your Business question 2. When we have dense node- connecting millions of relations...best approach wud be having relations by Month or category . (Fan\_out technique) Examples like Orders -[:ORDER]-\> Product to Orders-[: ORDER\_JAN|ORDER\_FEB...]-\>(:Product) Hope it may get you some thoughts to redefine the model . With Smiles, Senthil Chidambaram ---
**Author:** ![TonyOD27](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/tonyod27/32/5200_2.png) [@TonyOD27](https://community.neo4j.com/u/TonyOD27)\ **Post date:** [December 19, 2020, 2:46am UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/3 "2020-12-19T02:46:38Z")
Hi Senthil, Thank you for your advice. You're right in that we have a densely connected node on A\_Law and Person, and quite frankly, I don't know what to do that about that. These articles I found seem promising in helping with these super node issues. I'll give these a shot: > **[Graph Modeling: All About Super Nodes](https://medium.com/neo4j/graph-modeling-all-about-super-nodes-d6ad7e11015b)** > > This article is the latest in a series on advanced graph modeling techniques; the other articles in the series deal with keys… > **[How to avoid costly traversals with join hints - Knowledge Base](https://neo4j.com/developer/kb/how-to-avoid-costly-traversals-with-join-hints/)** > > When matching a pattern using Cypher, the number of possible paths to evaluate often correlates with query execution time. When there is a supernode in the path (a node with… Still, we must have that A\_Access connection there with all 500,000,000 relationships as this is giving my team important insights as to how the Person nodes can access the A\_Law node. Otherwise, this Neo4j effort is going to fail because we can't have queries taking hours to finish. Here are my answers: > [@senthilc78](#): > > 1. What is the query you are trying to get from here - Any Data Model - Best way to design based on the Buesiness Question you are trying to do For example, let's say we want to count the number of A\_Law's being used by a person in the sales department. Then that query will be: ```auto MATCH(p:Person {dept: 'sales'})-[:A_Access]->(a:A_Law) RETURN COUNT(DISTINCT a.title) ``` This query alone takes about 20 minutes to finish. Additionally, this query forms part of a bigger query that we use to answer more complex business questions and this is causing the bigger queries to take more than an hour to finish. > [@senthilc78](#): > > 1. From your Business need, I understand you need to have unique relations for each node. This leads to me to think -unique ID for each person right? so better to have this as property in my the Node itself - instead of relations which actually reduces all these millions of paths Yes, there are unique ID's for each person. We loaded this graph using the neo4j-admin import tool which forced us to use unique id's on the nodes. > [@senthilc78](#): > > 1. Validate when we can turn property to Node or Node to Property- Based on whats your Business question Agreed. We discussed this internally at length to all agree on the Node entities and properties based on the business questions we wanted to answer. We iterated through this as well to arrive at a working data model. Querying all of the other relationships outside of the A\_Access relationship actually perform really well, within seconds actually. > [@senthilc78](#): > > When we have dense node- connecting millions of relations...best approach wud be having relations by Month or category . (Fan\_out technique) > Examples like > Orders -[:ORDER]-\> Product to Orders-[: ORDER\_JAN|ORDER\_FEB...]-\>(:Product) This is a good point. I guess I could shuffle some of the properties from A\_Rel into A\_Access to further segregate the relationships that we need, but I still think we're going to end up with 500,000,000. It would also be useful if I could place an index on these relationship properties for faster look up but I don't think that's supported yet as shown in the docs or this thread: > **[Indexes for search performance - Cypher Manual](https://neo4j.com/docs/cypher-manual/current/indexes-for-search-performance/)** > > This section explains how to manage indexes used for search performance. > [@How can i use index in relationship?](https://community.neo4j.com/t/how-can-i-use-index-in-relationship/1627/6): > > No, not currently. We would advise refactoring your model such that nodes have the relevant numerical or date properties, and add an index on them instead. Or do I need more memory on my Neo4j instance? I thought having 24G in heap would be more than plenty for a graph database this size, no? Thanks ---
**Author:** ![senthilc78](https://avatars.discourse-cdn.com/v4/letter/s/3ab097/32.png) [@senthilc78](https://community.neo4j.com/u/senthilc78)\ **Post date:** [December 19, 2020, 6:21pm UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/4 "2020-12-19T18:21:58Z")
Hi @TonyOD27 , Really i like your kind way to explain each and every point. Let me go through in detail and update but got the core 1.Query Execution time is your concern and wants to ensure number of relationship count to super node may be the issue. But 3 quick pointers to share [1]. Aggregated operations always have some performance issue (Graph vs RDBMS ). So when you try COUNT(DISTINCT(node)) obvious it will load all ur nodes into memory and need more computing since you are using 'DISTINCT' . [2]. since you mentioned on heap-size- 24GB. We faced similar issue (a blind mistake) since we ignored the memory config while setting up initial instance (AWScloud) so thought to confirm is that same on your side. Please re-check whats your server RAM config. Hope it shud be greater that 24GB. There is a recommendation and formulae Memory = Heap Size + Page Cache + 2-3 GB for OS We made HeapSize and page Cache but sum of these 2 config value was equal to RAM of Server itself ; so OS have no space to allocate for its I/O and process. so even simple MATCH query took too much time .Then we realized logic and importance of checking Heap Size + Page Cache+ OS memory (min 2-3 GB should be left for OS) Example like - Instance got 24 GB RAM, recommended configuration is like Initial Heap\_Size 8 GB and made it Heap\_Max to 12 GB = 50% of server Memory Page Cache : 8 GB So left 4 GB for OS I was trying to get the actual Memory estimation technique link but missed to book mark. But you pls check this below though it might be just 1% cause for the issue. > **[How to estimate initial memory configuration - Knowledge Base](https://neo4j.com/developer/kb/how-to-estimate-initial-memory-configuration/)** > > The initial and eventual memory configuration parameters can be a moving target, based on how your store changes in size and how your workload increases or changes over time. This… just one more point to share my learning since it wud help you other scenario .. [3]. Neo4J stored data with its relations as we knew . I remember some example- Library- Books sorted and placed in Racks and each rack labeled like 'Graph Books, DB Books, OS, Server, Programming like .. so when the user wants to take 'Neo4J' book, he/she may not need to look from RACK 1 to RACK 10 but directly go to 'Graph Books rack 6' and take out books related to Graph right....this example helped me to have multiple RELATIONSHIP Name - and having multiple RELATIONS actually helps to fetch the data so quick instead of having just ONE Relations....this is the principle behind FAN\_out.. Example: ( :Customer )-[:ORDER] -\> (:Product) for last 3 years wud be many but same, i had it like 2 ways based on my Business need 1 .(:Customer : GuestUser) -[:ORDER]-\> (Product) here instead of searching all customer base, i have a label to get a subset of customer so performance/query hit will be balanced 2. ( : Customer: MonthlyCustomer)-[: ORDER\_JAN\_1WEEK]-\>(:Product) basically subset of customers based on some property /category aligned and helped Business to get the context So this is what i tried and it worked well for Customer360 view and i got to learn- it is all OK to have multiple Relationships between 2 Nodes when we have Millions of connection of course each Business scenario is different and what we want to check /query is the key differentiator here. > [@TonyOD27](#): > > ```auto > MATCH(p:Person {dept: 'sales'})-[:A_Access]->(a:A_Law) > RETURN COUNT(DISTINCT a.title) > > ``` Instead of 'Dept' as property in 'Person' node, is that OK to have 'DEPARTMENT' as new Node (Sales, HR, FINANCE,OPERATIONS) - basically group by Segments may work for you i think if its aligned to your Business scope. Hope it helps! With Smiles, Senthil Chidambaram ---
**Author:** ![mp.chean](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/mp.chean/32/15362_2.png) [@mp.chean](https://community.neo4j.com/u/mp.chean)\ **Post date:** [December 27, 2020, 2:09am UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/5 "2020-12-27T02:09:30Z")
I am a rank beginner - but was wondering how long the query takes without the aggregate command? Also is there some way to get a table output - and pull the distinct values from there and use that in a subsequent query? ---
**Author:** ![leonard.panichi](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/leonard.panichi/32/1484_2.png) [@leonard.panichi](https://community.neo4j.com/u/leonard.panichi)\ **Post date:** [December 27, 2020, 9:50am UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/6 "2020-12-27T09:50:11Z")
Hi, in my experience, sometimes you can go faster by 1. adding more nodes and relationships 2. precomputing Let's take Person and A\_Law. You say you have 500 000 000 possible combinations. Idea 1) For each Person.department, create a single node, it means ```auto MATCH(p:PersonByDept{dept: 'sales'}) ``` matches one node. For each A\_Law.title, create a single node. It means ```auto COUNT (a:A_LawByName{name: 'aname'}) ``` matches one node. Use these nodes to query what you need. So : ```auto MATCH(p:PersonByDept{dept: 'sales'})-[:something]->(a) RETURN COUNT(a) ``` Idea 2: Precompute what you will query. Attach a node counter to whatever entity you need and keep a count of what you want to count in this node counter. For instance create a node `(x:Department)` and attach a node `(x:Department)<-[]-(s:Stats)` and keep a property `s.ALawTitlesCount = 6` Does it work ? ---
**Author:** ![clem](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/clem/32/4364_2.png) [@clem](https://community.neo4j.com/u/clem)\ **Post date:** [December 27, 2020, 5:41pm UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/7 "2020-12-27T17:41:46Z")
Decreasing the relationships by increasing the number of properties might make performance **worse**. It depends on your queries and the nature of your data. For example, suppose a Person has a property "JobTitle", and there are different Law types, and Business types. Now if a lot of your queries are: `MATCH(p:Person {JobTitle:"Paralegal"})-[:B_access]->(l:B_Law {Name:"PatentLaw"})-[:A_Rel]->(b:Business {Type:"HighTech"})` The more properties you have to access during the query, the slower it will get. I'm making a conjecture as to the type of query you're trying to make... but it would be better to have the types represented as different **Node Labels** : `MATCH(p:Paralegal) -[:REL]->(l:PatentLaw)->(b:BusinessHighTech)` This is because the number of `Paralegal` nodes is far smaller than the number of `Person` nodes, so Cypher has to scan a lot fewer nodes when scanning for a `Paralegal` than `Person`. Similarly, there are a lot fewer `PatentLaw` nodes than all `Law` nodes, and a lot fewer `BusinessHighTech` nodes than all `Business` nodes. I believe` MATCH`ing based a Property is slower. This is because all Nodes of a certain _Label_ are already collected in a Set (behind the scenes in Neo4J), so all the work of winnowing down which Nodes to look at is already done! If you try to `MATCH` Nodes by _Property_, and if the properties are indexed then, Neo4J has to hit a B-Tree to collect all the nodes into a temporary set. If the Properties are **not indexed** then Cypher has to do a **linear search** (expensive!) through all the Nodes of the more general Label type (e.g. `Person`). One advantage of Neo4J is a node can have multiple Labels (e.g. Types), so that a Business could have two labels: `HighTech` label and a `Retailer` label (e.g. Amazon), or `Partner` and `Lawyer` and `Litigator`, so you don't have to worry about painting yourself into a corner when making the Labels. That is in a regular relationship DB, the categories could be too fine grained resulting in a many-to-many relationship, which is a PITA. Similarly, if you can split up the \*\_Law nodes into a finer set of types, you'd also be better off. Since access must be done in memory, repeated queries are faster once the data (Set of Nodes of a Label type) is loaded in memory from a previous query, subsequent queries will be faster. I recommend taking a subset of your data and experiment with the performance on the subset so that you don't have to wait an hour to see if any tweaks you do makes an improvement. If you gave us some sample queries you are trying to make, it would help us understand the nature of the problem better. I hope that helps. ---
**Author:** ![mp.chean](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/mp.chean/32/15362_2.png) [@mp.chean](https://community.neo4j.com/u/mp.chean)\ **Post date:** [December 28, 2020, 12:18am UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/8 "2020-12-28T00:18:27Z")
This is a great reply with lots of information to think about. It would be interesting to mock this data and try what was suggested above. Tangentially, I wonder if other products like DGraph and TigerGraph might be faster, and whether the language that Neo4j is written in (Java, Haskell) might be slower than DGraph (Go) and TigerGraph (C++). ---
**Author:** ![clem](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/clem/32/4364_2.png) [@clem](https://community.neo4j.com/u/clem)\ **Post date:** [December 28, 2020, 12:43am UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/9 "2020-12-28T00:43:15Z")
It’s certainly worth trying to see which system performs better. BUT remember, the data is probably stored on disk, which typically means the bottleneck is going to be I/O not CPU (where C++ would have an advantage.). In addition, with HotSpot, Java might not perform significantly worse than C++. It maybe that more memory will have a greater impact on performance than anything else, as more memory means more data can be kept in memory (cached) than having to be read off of disk. So, I don’t think you can a priori claim one system will be better based solely on the language of implementation. There are too many factors that will effect performance, some of which won’t be readily visible to you. Neo4J had been around for a long time, so it might be better (and stabler) because of it being a more mature product. ---
**Author:** ![mp.chean](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/mp.chean/32/15362_2.png) [@mp.chean](https://community.neo4j.com/u/mp.chean)\ **Post date:** [December 28, 2020, 3:22am UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/10 "2020-12-28T03:22:56Z")
DGraph is claiming all kinds of performance capabilities here: [Neo4j vs Dgraph - The numbers speak for themselves - Dgraph Blog](https://dgraph.io/blog/post/benchmark-neo4j/) but the implementation does look far more basic than Neo4j. ---
**Author:** ![clem](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/clem/32/4364_2.png) [@clem](https://community.neo4j.com/u/clem)\ **Post date:** [December 28, 2020, 5:47am UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/11 "2020-12-28T05:47:47Z")
Note that the Blog is making a comparison with version 3 of Neo4J. It’s now on 4.2.1. You’ll need to benchmark it on your data. Benchmarks created by a company to show off their product may or may not be valid when applying to your situation. And then there is this URL. I’m not sure how accurate it is but it’s another data point: > **[Dgraph vs. Neo4j Comparison](https://db-engines.com/en/system/Dgraph%3BNeo4j)** > > Detailed side-by-side view of Dgraph and Neo4j I have been looking at Azure Cosmos/Gremlin but I’m not happy with it, as it’s been a bit painful to get up and running for me. I’m not well versed in the Graph DB competition, so I’m hesitant to say for sure which is the best Graph DB out there. It’s just from my long work experience, I’ve learned to take benchmarks with a grain of salt. ---
**Author:** ![mp.chean](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/mp.chean/32/15362_2.png) [@mp.chean](https://community.neo4j.com/u/mp.chean)\ **Post date:** [December 28, 2020, 11:11pm UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/12 "2020-12-28T23:11:36Z")
From the standpoint of tools - based on a cursory review - Neo4j looks the best so far. ---
**Author:** ![clem](https://sea1.discourse-cdn.com/flex021/user_avatar/community.neo4j.com/clem/32/4364_2.png) [@clem](https://community.neo4j.com/u/clem)\ **Post date:** [May 4, 2021, 5:38pm UTC](https://community.neo4j.com/t/graph-data-modeling-question/30522/13 "2021-05-04T17:38:15Z")
I've been exploring TigerGraph. It's been a bit rough... All the intuition that I've developed with Cyper and SQL has led me astray a number of times with TG. They claim better performance (which I believe is true), but TG definitely has a steeper learning curve than with Neo4J, which I got up and running pretty easily. I think the Cypher language is more intuitive too and the documentation is better. That is, I pretty much figured out most of what I needed with Cypher on my own (and I strongly believe that I could train semi-technical people to do their own Cypher queries), but I feel like a total noob with TG.