Hello,
I have to have an Unique Identifier on one of the node is a 38 character string, eg - "37300097819223399953200236708848436046".
Is there anyway to optimize this kind of Unique Identifier. I have an index on this. This fails to load the 9M rows.
I have around 9M records which I am loading into graph with below command -
CALL apoc.periodic.iterate("
UNWIND ['activity.csv'] AS file CALL apoc.load.csv('file:///dcm/' + file, {header:true, sep:',', quoteChar:'"', nullValues:['']}) YIELD
map as row WHERE row.ecid is not null return row
","
WITH row
MERGE (ec:ec_id {ec_id:row.ecid})
ON CREATE SET ec+={create_dt:datetime()}
", {batchSize: 10000, iterateList: true, parallel: false})
Community Edition Server -
Server Size - 16 Core 128GB
Server is running on 70% CPU when loading data
Memory Configs -
dbms.memory.heap.initial_size=31g
dbms.memory.heap.max_size=31g
dbms.memory.pagecache.size=73g
dbms.tx_state.max_off_heap_memory=8g
I am seeing below logs in debug.log and database becomes inaccessible.
2020-05-29 04:34:54.697+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5933, gcTime=5972, gcCount=2}
2020-05-29 04:35:01.850+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5951, gcTime=5963, gcCount=2}
2020-05-29 04:35:08.773+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5910, gcTime=5916, gcCount=2}
2020-05-29 04:35:15.831+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=6021, gcTime=6102, gcCount=2}
2020-05-29 04:35:22.785+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=6131, gcTime=6173, gcCount=2}
2020-05-29 04:35:29.434+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5847, gcTime=5935, gcCount=3}
2020-05-29 04:35:36.352+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=6082, gcTime=6097, gcCount=2}
2020-05-29 04:35:42.847+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5680, gcTime=5773, gcCount=2}
2020-05-29 04:35:49.551+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=6101, gcTime=6196, gcCount=2}
2020-05-29 04:35:56.490+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=6137, gcTime=6181, gcCount=2}
2020-05-29 04:36:03.019+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5927, gcTime=5974, gcCount=3}
2020-05-29 04:36:09.480+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5960, gcTime=6023, gcCount=2}
2020-05-29 04:36:15.959+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5978, gcTime=6049, gcCount=3}
2020-05-29 04:36:22.346+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5885, gcTime=5933, gcCount=2}
2020-05-29 04:36:28.693+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5945, gcTime=5987, gcCount=2}
2020-05-29 04:36:35.028+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5807, gcTime=5881, gcCount=2}
2020-05-29 04:36:41.493+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=6063, gcTime=6132, gcCount=3}
2020-05-29 04:36:47.641+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5747, gcTime=0, gcCount=0}
2020-05-29 04:36:53.606+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5764, gcTime=5796, gcCount=2}
2020-05-29 04:36:59.946+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=6039, gcTime=6083, gcCount=3}
2020-05-29 04:37:05.837+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5590, gcTime=5618, gcCount=2}
2020-05-29 04:37:11.823+0000 WARN [o.n.k.i.c.VmPauseMonitorComponent] Detected VM stop-the-world pause: {pauseTime=5669, gcTime=5759, gcCount=2}