Neo4j crashes when do batching but cannot locate the reason

Neo4j Graph Platform
,
1

Hi guys,

My neo4j crashes when I'm using java API to do batching. The data I have to update is already stored on the server as file. My server is dedicated to neo4j, and I'm just using Java API to call APOC procedures to read those files, after dozens of batch, the program crashes with org.neo4j.driver.v1.exceptions.ServiceUnavailableException: Connection to the database terminated. This can happen due to network instabilities, or due to restarts of the database

It's unlikely due to my java program, because when I logged on my Neo4j server and ran bin/neo4j status, it showed that neo4j is not running. However, when I tried to locate the problem, I didn't get much useful information from log.

The cypher query I'm using is like:

CALL apoc.periodic.iterate(
                            "CALL apoc.load.json('file:///neo4j/update_type3/updatePropertiesJson41.txt') yield value as row", 
                            "match (n:Entity{eid:row.eid}) CALL apoc.create.setProperties(n, keys(row.properties), [k in keys(row.properties)|row.properties[k]]) yield node return 'node'",{batchSize:1000, parallel:true})

My neo4j.log just showed the batching information, there's no exception, error etc.:

image
image1237×738 88 KB

Please notice that the failure of some transactions is expected behavior because I have duplicate eid in my update files, which leads to batch failure when parallel is true

My debug.log showed similar result as neo4j.log, no exception, error etc. as well:

image
image1239×702 79 KB

I'm using 3.2.3 enterprise edition.

I suppose the reason for the crash may be the huge workload, but I didn't make it to locate the reason. Any workaround or suggestion will be well appreciated.

2

When you say "crashes" do you mean the Neo4J server aborts and leaves no trace?
Or is it still there but not responding to further requests?
What happens when you do the queries via the webinterface on :7474 instead?

3

Hi Paul,

Thanks for respond. The word "crashes" means that the neo4j service is totally down, and not accessible via Java API or Web UI.

Any hints?

4

Is the neo4j process still there when you look with ps ?
Eg: ps -ef |grep neo ?

Perhaps /var/log/syslog contains more hints?

5

Again thanks for your respond. I ended up solving this problem by increasing the memory configuration in the neo4j.conf, didn't know why the database crashed without any trace.

1 Like
6

Howdy! Can I ask how you were able to get the Neo server back up after it failed? I am running into a similar issue.