Need advice on performance tuning for Neo4j Cypher on a large dataset with relationships

Hi
I’m working with a fairly large Neo4j dataset containing about ~2.3 million OrderLines nodes and roughly ~.5 millionComplianceRecord nodes stored for a tenant. The below cypher takes approximately 12-14 seconds to return response. I would like to know if it is possible make this query return response faster without reducing the dataset? Any suggestions for improving the performance of this query would be greatly appreciated.

Cypher:

MATCH (orderline:`OrderLine` {
  tenantId: 'tenant1', type:'OrderLine'
}) USING INDEX orderline:`OrderLine`(tenantId, plannedShipDate)
WHERE (orderline.plannedShipDate < '2025-06-20T18:30:00.001Z' AND orderline.plannedShipDate >= '2022-12-31T18:30:00.000Z')
WITH orderline
OPTIONAL MATCH (orderline)-[:`ORDERLINE_PCFCOMPLIANCERECORD`]->(compliancerecord_pcfComplianceRecord:`ComplianceRecord`)
RETURN sum((compliancerecord_pcfComplianceRecord.score * orderline.quantity)) AS TotalEmissionsByAll

PROFILE Output:

{
	"results": [
		{
			"columns": [
				"TotalEmissionsByAll"
			],
			"data": [
				{
					"row": [
						5.1915847641664444E10
					],
					"meta": [
						null
					]
				}
			],
			"plan": {
				"root": {
					"operatorType": "ProduceResults@neo4j",
					"GlobalMemory": 168,
					"planner-impl": "IDP",
					"Memory": 0,
					"runtime": "INTERPRETED",
					"runtime-impl": "INTERPRETED",
					"version": "CYPHER 4.4",
					"DbHits": 0,
					"Details": "TotalEmissionsByAll",
					"planner-version": "4.4",
					"runtime-version": "4.4",
					"PageCacheMisses": 0,
					"EstimatedRows": 1.0,
					"planner": "COST",
					"Rows": 1,
					"PageCacheHits": 0,
					"identifiers": [
						"TotalEmissionsByAll"
					],
					"children": [
						{
							"operatorType": "EagerAggregation@neo4j",
							"Details": "sum(compliancerecord_pcfComplianceRecord.score * cache[orderline.quantity]) AS TotalEmissionsByAll",
							"Memory": 64,
							"PageCacheMisses": 0,
							"EstimatedRows": 1.0,
							"DbHits": 1149154,
							"Rows": 1,
							"PageCacheHits": 0,
							"identifiers": [
								"TotalEmissionsByAll"
							],
							"children": [
								{
									"operatorType": "OptionalExpand(All)@neo4j",
									"Details": "(orderline)-[anon_0:ORDERLINE_PCFCOMPLIANCERECORD]->(compliancerecord_pcfComplianceRecord) WHERE compliancerecord_pcfComplianceRecord:ComplianceRecord",
									"PageCacheMisses": 0,
									"EstimatedRows": 8565.241875,
									"DbHits": 3438667,
									"Rows": 1152073,
									"PageCacheHits": 0,
									"identifiers": [
										"orderline",
										"anon_0",
										"compliancerecord_pcfComplianceRecord"
									],
									"children": [
										{
											"operatorType": "Filter@neo4j",
											"Details": "orderline.type = $autostring_1",
											"PageCacheMisses": 0,
											"EstimatedRows": 8565.241875,
											"DbHits": 1140359,
											"Rows": 1140359,
											"PageCacheHits": 0,
											"identifiers": [
												"orderline"
											],
											"children": [
												{
													"operatorType": "CacheProperties@neo4j",
													"Details": "cache[orderline.quantity]",
													"PageCacheMisses": 0,
													"EstimatedRows": 1259.7910761687024,
													"DbHits": 1140359,
													"Rows": 1140359,
													"PageCacheHits": 0,
													"identifiers": [
														"orderline"
													],
													"children": [
														{
															"operatorType": "NodeIndexSeek@neo4j",
															"Details": "BTREE INDEX orderline:OrderLine(tenantId, plannedShipDate) WHERE tenantId = $autostring_0 AND plannedShipDate >= $autostring_3 AND plannedShipDate < $autostring_2",
															"PageCacheMisses": 0,
															"EstimatedRows": 1259.7910761687024,
															"DbHits": 1140360,
															"Rows": 1140359,
															"PageCacheHits": 0,
															"identifiers": [
																"orderline"
															],
															"children": []
														}
													]
												}
											]
										}
									]
								}
							]
						}
					]
				}
			}
		}
	],
	"errors": []
}

@debatosh.tripathy

a. What version of Neo4j? it appears maybe? a 4.4.x release? Any reason not to use a more current release?

b. if you dont include USING INDEX orderline:OrderLine(tenantId, plannedShipDate) is the index thus not automatically pulled in by the planner?

c. 12-14 seconds to return response. Is this from a Neo4j cold start or if you run the query multiple times does it always take 12-14 seconds

d. how have you configured conf/neo4j.conf and dbms.memory.pagecache.size and dbms.memory.heap.init_size and dbms.memory.heap.max_size? How much total RAM is on the instance

Hi @dana_canzano

a) We are using Neo4j version 4.4.42 and have plans to upgrade to 5.x latest version by end of this year.

b) Yes it does. Even if USING INDEX orderline:OrderLine(tenantId, plannedShipDate) is not explicitly used in the cypher, the same index is still internally used, and I have verified by running the cypher with PROFILE command.

c) Yes it does take 12-14 seconds even if we ran it multiple times. The same query may take even more time if we restart the database due to cold start for neo4j.

d) Here are the details:

dbms.mode=SINGLE
dbms.memory.pagecache.size=81G
dbms.memory.heap.max_size=105G
dbms.memory.heap.initial_size=105G

@debatosh.tripathy

d) Here are the details:

dbms.mode=SINGLE
dbms.memory.pagecache.size=81G
dbms.memory.heap.max_size=105G
dbms.memory.heap.initial_size=105G
  • how much total RAM is on the machine. How did you determine heap should be set to 105G?

  • how large is the database? what is the output of

du -hc data/databases/

and specifically the last line

Hi @dana_canzano these setting were not changed from quite some time. Here are the CPU & Memory details.

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests           Limits
  --------           --------           ------
  cpu                45275m (81%)       55100m (98%)
  memory             232099346Ki (99%)  236206368Ki (101%)
  ephemeral-storage  5Mi (0%)           105Mi (0%)
  hugepages-1Gi      0 (0%)             0 (0%)
  hugepages-2Mi      0 (0%)             0 (0%)

I will post the output for du -hc data/databases/ in some time.

Here is the disk usage (du -hc) output:

560K	/var/lib/neo4j/data/databases/neo4j/corrupted-neostore.transaction.db
4.0K	/var/lib/neo4j/data/databases/neo4j/schema/index/fulltext-1.0/188/188.tx
8.0K	/var/lib/neo4j/data/databases/neo4j/schema/index/fulltext-1.0/188/1
32K	/var/lib/neo4j/data/databases/neo4j/schema/index/fulltext-1.0/188
36K	/var/lib/neo4j/data/databases/neo4j/schema/index/fulltext-1.0
97M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/3
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/117
1.6G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/42
9.8G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/29
2.0G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/141
11M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/85
30M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/189
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/27
4.8G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/201
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/98
129M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/24
4.0M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/102
385M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/77
23M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/180
513M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/173
1.5G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/119
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/178
161M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/192
244K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/123
23M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/154
1.6G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/75
25M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/155
161M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/140
385M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/82
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/19
9.8M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/8
396K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/153
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/11
193M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/67
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/198
385M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/78
129M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/81
161M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/45
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/179
236K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/114
97M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/193
449M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/200
97M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/71
65M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/33
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/162
2.1M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/207
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/68
385M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/14
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/116
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/20
385M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/203
5.0M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/133
8.3M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/105
27M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/40
1.1M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/108
257M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/2
161M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/44
5.2M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/15
2.4G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/121
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/113
1.6G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/72
3.9M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/184
3.4M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/208
225M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/57
84K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/164
129M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/62
17M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/129
65M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/167
65M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/41
1012K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/128
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/197
6.8G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/74
288K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/152
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/97
268K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/104
577M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/172
129M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/70
129M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/18
97M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/166
4.4M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/186
257M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/95
225M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/139
4.5M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/170
1.2G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/34
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/169
5.3M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/36
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/86
2.0G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/76
240K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/107
161M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/148
577M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/176
31M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/130
268K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/26
257M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/56
11G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/175
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/161
1.1M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/157
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/25
22M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/125
97M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/47
449M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/30
97M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/10
737M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/136
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/100
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/89
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/163
180K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/61
929M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/43
412K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/205
161M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/145
572K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/204
129M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/53
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/99
1.3G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/48
321M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/150
641M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/177
31M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/190
129M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/144
180K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/110
225M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/54
1.6G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/143
161M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/158
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/35
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/101
288K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/124
1.5G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/69
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/91
1.6G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/6
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/93
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/92
97M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/126
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/134
65M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/46
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/28
14G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/64
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/111
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/196
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/22
8.8G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/1
321M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/5
1.4M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/109
225M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/49
1.1G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/142
833M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/66
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/137
161M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/159
1.3G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/194
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/63
92K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/183
316K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/115
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/160
2.4G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/118
4.1M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/185
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/106
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/16
129M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/50
97M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/38
18M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/58
1.2G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/4
1.3G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/151
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/112
2.0G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/187
2.5M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/206
1.2G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/13
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/138
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/80
97M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/94
84K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/195
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/21
1.8G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/37
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/135
312K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/60
65M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/39
3.1M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/171
1.5M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/156
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/52
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/87
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/12
4.4G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/120
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/90
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/51
1.5G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/84
65M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/127
97M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/55
4.5M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/7
92K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/182
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/168
68K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/165
272K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/59
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/88
161M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/31
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/23
13G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/73
129M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/83
18M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/181
4.1M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/103
164K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/131
300K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/17
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/96
52K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/9
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/147
385M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/191
15G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/199
44K	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/146
161M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/32
4.0G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/174
161M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/149
321M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/79
5.2M	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0/132
138G	/var/lib/neo4j/data/databases/neo4j/schema/index/native-btree-1.0
138G	/var/lib/neo4j/data/databases/neo4j/schema/index
138G	/var/lib/neo4j/data/databases/neo4j/schema
334G	/var/lib/neo4j/data/databases/neo4j
52K	/var/lib/neo4j/data/databases/system/schema/index/native-btree-1.0/3
52K	/var/lib/neo4j/data/databases/system/schema/index/native-btree-1.0/6
52K	/var/lib/neo4j/data/databases/system/schema/index/native-btree-1.0/1
160K	/var/lib/neo4j/data/databases/system/schema/index/native-btree-1.0
164K	/var/lib/neo4j/data/databases/system/schema/index
168K	/var/lib/neo4j/data/databases/system/schema
1.1M	/var/lib/neo4j/data/databases/system
334G	/var/lib/neo4j/data/databases
334G	total

@debatosh.tripathy

give the last line of du -hc data/databases reports 334G for the entire database to be in RAM you would defined dbms.memory.pagecache.size=334G +/-. As you currently have it defined as 81G then at most 25% of the database is in RAM, i.e. 81G/334G. And so more than likely most of your queries are getting data from the filesystem as opposed to RAM.

Further given your update indicating

Allocated resources:
  (Total limits may be over 100 percent, i.e., overcommitted.)
  Resource           Requests           Limits
  --------           --------           ------
  cpu                45275m (81%)       55100m (98%)
  memory             232099346Ki (99%)  236206368Ki (101%)

if i read this correctly then 232099346Ki is 230GB. So even if heap was at 0, which I am not recommending, the most you could allocate is 230 GB to dbms.memory.pagecache.size.

How did you come to setting min/max heap to 105G? That seems rather large but maybe you have details as to what requires it to be so large?

A couple of dumb questions:

  • are there other nodes/relationships in your graph?
  • what about the total size of your database?

So the neo4j db is deployed as a k8 pod, we have 230 gb on a high ram node, but the pod is limited to 220 gb.

So assuming the page cache and the heap is fully used, we will use 186 gb in memory which leaves around 34 gb for os memory usage.

We had seen issues where the db pod was killed due to memory limit exceed when the os memory usage was limited to 15 gbs alone.

with regards to why the size is at 105 gb, there is no history available on that as of now.

So there are around 63 types of nodes which could be created and many relations which could be created between each all under the relationship type called as RELATES_TO.

We already had a recommendation from the forum to change from the generic (RELATES_TO) to specific relations (RELATIONSHIP NAME) and we do see some minor improvement in the performance (~1 to 3 seconds being shaved off).

With regards to the size and counts, we do a have a global timeout implemented using dbms.transaction.timeout at 30 seconds (since we do display results form neo4j into UI), so trying to query the total nodes/relations might not be possible at the moment (during business days/hours).

With regards to the size, we did estimate it using the file system to be around 334 gb, but exact figures are added in the comment previously in this thread.

@debatosh.tripathy @vijeya.nidhi.b1

a bit odd that dbms.memory.pagecache.size is 81G and is smaller than total heap which is set to 105G.

and if you are encountering out of memory then more than likely you have transactions which need to be batched, i.e. do not updated 100 million nodes in one pass rather do this in batches of 500k for example.

heap of 105G is certainly not typical. and given your comment of

why the size is at 105 gb, there is no history available on that as of now.

isnt great.

be mindful that just in memory traversal you do 21 to 44 GB/s ... just moving through RAM (without counting disk fetches if needed) you are spending a significant portion of time "moving"

Hi @dana_canzano , @joshcornejo

The database was created 5 years back, now it has grown significantly, and we would appreciate, if you could tell us how we could fine-tune these settings in terms of sizes to be allocated.

Current DB size: 334G
Max capacity configured: 500G

Could you please also explain why the below observation of yours is a problem and what we could do to fix it.

a bit odd that dbms.memory.pagecache.size is 81G and is smaller than total heap which is set to 105G.

be mindful that just in memory traversal you do 21 to 44 GB/s ... just moving through RAM (without counting disk fetches if needed) you are spending a significant portion of time "moving"

My comment regarding memory speed is that if you have huge amounts of data - even in the cache, "moving to process" could take some time (no idea what's your server's memory speed, but if you have 210 GB used and 21GB/s ... that's 10 seconds moving across memory)

I'm sure Dana will come up with more detail. but here is the link to the configuration documentation:

@debatosh.tripathy @vijeya.nidhi.b1

the link to doc that @joshcornejo describes is a good 1st start as it describes how memory can be carved up for Neo4j use.

But specifically heap is generally used as and intermediate scratch are for storing results.

For example if one runs match (n:Person) return n order by n.age; then this finds all :Person nodes and then returns results order by age. The ordering is performed in heap. Further if one run match (n:Person) return n.age,count(*); then similarly it needs to group the Person nodes by age and return the count. This grouping is occuring in heap. And match (n:Person) return n.age,collect(n.name); its going to use heap to group by age and then collect names.
Now as to why you would have a heap defined at 105G, i.e. 1/3 of the database size is a bit unexpected. Are you returning results that are 1/3 of the entire database?

I don't think we are fetching 1/3 of the entire database. Do you recommend an ideal size, or could you please let us know how we can determine that number in terms of GB for the heap?

@debatosh.tripathy @vijeya.nidhi.b1

Do you recommend an ideal size ....

the other thing is heap is also somewhat related to query concurrency. i.e. one user may have a query which utilizes 500mb of heap but if you have 100 users with 100 concurrent queries these 100 queries in total may use 25G at a given time.

the doc previously referenced and

includes detail that indicates

Initial memory recommendation

Use the neo4j-admin server memory-recommendation command to get an initial recommendation for how to distribute a certain amount of memory. The values may need to be adjusted to cater for each specific use case

Has this been considered. Note neo4j-admin server memory-recommendation is a 'recommendation and starting point.

1 Like