Newbie question: unexpected cypher-shell memory/heap usage

cypher-shell

(Alec Muffett) #1

I have a reasonably large AWS Ubuntu instance with 60+Gb of RAM.

I have configured CE with:

dbms.memory.heap.initial_size=20g
dbms.memory.heap.max_size=20g
dbms.memory.pagecache.size=20g

I am running a complex/large query under cypher-shell --format plain --fail-fast and I am surprised that the cypher-shell process is running out of memory... so I am wondering whether the query is running partially/wholly within the cypher-shell client process, rather than the server.

Behaviour: according to top:

  • neo4j database process = 43Gb (fits above configuration) and eventually using minimal CPU
  • cypher-shell process (running as me) = 20Gb (surprise) and eating 1200% CPU (surprise) and tending to run out of heap unless I tweak the query to be conservative, etc.

So... am I correct that cypher-shell is doing part/all of the query work and thereby suffering from resource limit constraints that I had presumed would be handled by the (more generously resourced) main daemon?


(Michael Hunger) #2

Cypher shell is not streaming it materializes all results.

I have a PR open for cypher-shell which has not been merged yet to add streaming.

Use your own code to consume the results in a streaming manner, in any language you want.