VM stop-the-world pause for simple LOAD csv query for creating a graph

Neo4j Graph Platform Operations
, ,
26

Thank you @dana_canzano for helping out!

to answer 1, I'm pasting some more logs from debug.log I see (basically first VM stop the world error repeated-> conection time out then it stops running current query and go to my next cypher query in the cypher-shell.

2022-02-17 01:45:48.077+0000 INFO [o.n.k.a.p.GlobalProcedures] starting batching from `call apoc.load.jdbc($azure_url, ​

"select distinct  m.measurement_id, m.visit_occurrence_id, m.measurement_concept_id, v.visit_concept_id, v.visit_start_date, v.visit_end_date​

from omop.measurement m​

join omop.visit_occurrence v ​

on v.visit_occurrence_id = m.visit_occurrence_id")​

yield row​

` operation using iteration `​

match (m:Measurement {measurement_concept_id: row.measurement_concept_id})​

match (v:VisitOccurrence {visit_concept_id:row.visit_concept_id})​

merge (m)-[r:ASSOCIATED_DURING_VISIT {visit_occurrence_id:row.visit_occurrence_id, ​

visit_start_date:row.visit_start_date, ​

visit_end_date:row.visit_end_date,​

measurement_id:row.measurement_id}]->(v)​

` in separate thread​

2022-02-17 01:46:44.330+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=276, gcTime=307, gcCount=1}​

2022-02-17 01:47:21.996+0000 ERROR [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection: [id: 0xafcb7c11, L:/172.17.0.2:7687 - R:/10.120.14.85:60148] readAddress(..) failed: Connection timed out​

io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection timed out​

2022-02-17 01:47:21.997+0000 ERROR [o.n.b.t.p.HouseKeeper] Fatal error occurred when handling a client connection: [id: 0x6f3b153e, L:/172.17.0.2:7687 - R:/10.120.14.85:60149] readAddress(..) failed: Connection timed out​

io.netty.channel.unix.Errors$NativeIoException: readAddress(..) failed: Connection timed out​

2022-02-17 01:48:46.708+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=578, gcTime=0, gcCount=0}​

for 2, when I ran again neo4j-admin memrec after loading most of the nodes and edges I got super high pagecache recommendation - more than 450GB, is this expected?

# Based on the above, the following memory settings are recommended:
dbms.memory.heap.initial_size=31g
dbms.memory.heap.max_size=31g
dbms.memory.pagecache.size=459300m