We are indexing a property of a node and the server is doing something (I see increments in the physical disk size of the index) but it is running for several days (more than a week) and it doesn't seem near completion. For me several days of indexing in a powerful machine denotes that something is wrong but I don't know what, so asking help here in order to diagnose the problem.
- The number of nodes is 999x10^6 , a billion.
- The property is a string hash of size 64 completely random characters
- We are using Neo4J Enterprise Causal Cluster created from Google Cloud Marketplace template. The cluster is just 3 core members:
- The leader has 32 CPU cores and 128Gb of RAM.
- Followers have 4 CPU cores and 128Gb of RAM
- Memory is configured as:
dbms.memory.heap.iniitial_size=30600m dbms.memory.heap.max_size=30600m dbms.memory.pagecache.size=74800m
as was recommended by neo4j-admin memrec
- Memory usage for the java process in Leader is reported as 64%
- CPU usage is just at 10% consistently
- Index size keeps growing although slowly
neo4j-enterprise-causal-cluster-1-core-vm-1:/var/lib/neo4j/data/databases/graph.db$ while true; do echo "$(date -Iseconds) $(du -ck schema/index/native-btree-1.0/* | grep total)" ; sleep 60; done 2019-04-18T07:06:07+00:00 145382404 total 2019-04-18T07:07:07+00:00 145384212 total 2019-04-18T07:08:07+00:00 145386312 total 2019-04-18T07:09:07+00:00 145388496 total 2019-04-18T07:10:07+00:00 145390420 total 2019-04-18T07:11:07+00:00 145392748 total
- There is only one index being created at the moment and it has been running for 6 days.
I don't know if these numbers are "normal" for the problem's size, but I would appreciate any help on trying to diagnose if this is expected or if I should tweak some parameter or check any log to discover who us causing this slow down.