Neo4j stopping automatically and restarting automatically

Neo4j Graph Platform Operations
1

Hi everyone...
I'm running my neo4j community edition 3.5.5 version with 8gb ram in aws instance.Initially for few months it ran very fine and got results in millis of time,but now a days it's getting stopping automatically and starting automatically.Sometimes it's not at all starting for hours,even we started it manually also.Can anyone please help me with this.
Thank you.

I'm getting the below logs.

tail -100f /var/log/neo4j/neo4j.log
2019-07-29 13:17:52.570+0000 WARN The client is unauthorized due to authentication failure.
2019-09-04 05:33:52.328+0000 WARN The client is unauthorized due to authentication failure.
2019-09-04 05:33:56.236+0000 WARN The client is unauthorized due to authentication failure.
2019-09-04 05:33:57.454+0000 WARN The client is unauthorized due to authentication failure.
2019-09-04 05:39:21.779+0000 WARN The client is unauthorized due to authentication failure.
2019-09-04 06:48:20.119+0000 WARN The client is unauthorized due to authentication failure.
2019-09-04 14:30:38.256+0000 WARN The client is unauthorized due to authentication failure.
2019-09-04 17:13:46.963+0000 WARN The client is unauthorized due to authentication failure.
2019-09-04 18:39:15.905+0000 WARN The client is unauthorized due to authentication failure.
2019-09-05 05:56:00.350+0000 WARN The client is unauthorized due to authentication failure.
2019-09-05 11:36:48.268+0000 WARN The client is unauthorized due to authentication failure.
2019-09-05 11:40:30.361+0000 WARN The client is unauthorized due to authentication failure.
2019-09-06 16:12:39.181+0000 WARN The client is unauthorized due to authentication failure.
2019-09-09 04:47:16.335+0000 WARN The client is unauthorized due to authentication failure.
2019-09-09 09:35:40.851+0000 WARN The client is unauthorized due to authentication failure.
2019-09-09 16:16:09.109+0000 WARN The client is unauthorized due to authentication failure.
2019-09-10 05:18:48.279+0000 WARN The client is unauthorized due to authentication failure.
2019-09-10 12:53:07.026+0000 WARN The client is unauthorized due to authentication failure.
2019-09-10 14:26:39.366+0000 WARN The client is unauthorized due to authentication failure.
2019-09-11 04:45:16.942+0000 WARN The client is unauthorized due to authentication failure.
2019-09-11 10:00:30.476+0000 WARN The client is unauthorized due to authentication failure.
2019-09-11 11:42:52.046+0000 WARN The client is unauthorized due to authentication failure.
2019-09-11 12:29:14.341+0000 WARN The client is unauthorized due to authentication failure.
2019-09-11 15:25:41.425+0000 WARN The client is unauthorized due to authentication failure.
2019-09-14 07:08:53.391+0000 WARN The client is unauthorized due to authentication failure.
2019-09-15 10:42:04.931+0000 WARN The client is unauthorized due to authentication failure.
2019-09-18 04:58:56.789+0000 WARN The client is unauthorized due to authentication failure.
2019-09-18 07:01:57.698+0000 WARN The client is unauthorized due to authentication failure.
2019-09-19 04:55:47.986+0000 WARN The client is unauthorized due to authentication failure.
2019-09-20 05:59:28.116+0000 WARN The client is unauthorized due to authentication failure.
2019-09-20 06:40:01.734+0000 WARN The client is unauthorized due to authentication failure.
2019-09-20 10:10:04.689+0000 WARN The client is unauthorized due to authentication failure.
2019-09-20 10:26:35.811+0000 WARN The client is unauthorized due to authentication failure.
2019-09-20 10:33:05.379+0000 WARN The client is unauthorized due to authentication failure.
2019-09-20 12:58:18.059+0000 WARN The client is unauthorized due to authentication failure.
2019-09-25 14:28:27.793+0000 WARN The client is unauthorized due to authentication failure.
2019-10-17 15:18:14.652+0000 INFO Transaction with id 2683388 has been automatically rolled back due to transaction timeout.
2019-10-17 15:21:44.651+0000 INFO Transaction with id 2683569 has been automatically rolled back due to transaction timeout.
2019-10-17 15:26:14.651+0000 INFO Transaction with id 2683870 has been automatically rolled back due to transaction timeout.
2019-10-17 15:28:14.651+0000 INFO Transaction with id 2684019 has been automatically rolled back due to transaction timeout.
2019-10-17 15:39:14.652+0000 INFO Transaction with id 2684490 has been automatically rolled back due to transaction timeout.
2019-10-17 15:42:44.651+0000 INFO Transaction with id 2684594 has been automatically rolled back due to transaction timeout.
2019-10-17 15:43:44.651+0000 INFO Transaction with id 2684624 has been automatically rolled back due to transaction timeout.
2019-10-17 15:44:44.651+0000 INFO Transaction with id 2684653 has been automatically rolled back due to transaction timeout.
2019-10-17 15:45:44.651+0000 INFO Transaction with id 2684668 has been automatically rolled back due to transaction timeout.
2019-10-17 15:50:14.671+0000 INFO Transaction with id 2685019 has been automatically rolled back due to transaction timeout.
2019-10-17 15:55:14.652+0000 INFO Transaction with id 2685205 has been automatically rolled back due to transaction timeout.
2019-10-17 16:14:14.651+0000 INFO Transaction with id 2685953 has been automatically rolled back due to transaction timeout.
2019-10-17 16:17:14.651+0000 INFO Transaction with id 2686064 has been automatically rolled back due to transaction timeout.
2019-10-17 16:34:14.652+0000 INFO Transaction with id 2686957 has been automatically rolled back due to transaction timeout.
2019-10-17 16:34:44.651+0000 INFO Transaction with id 2686992 has been automatically rolled back due to transaction timeout.
2019-10-17 17:02:14.651+0000 INFO Transaction with id 2688614 has been automatically rolled back due to transaction timeout.
nohup: ignoring input
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006e5400000, 3670016000, 0) failed; error='Cannot allocate memory' (errno=12)

There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 3670016000 bytes for committing reserved memory.
An error report file with more information is saved as:
/home/ubuntu/hs_err_pid8965.log
nohup: ignoring input
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006e5400000, 3670016000, 0) failed; error='Cannot allocate memory' (errno=12)

There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 3670016000 bytes for committing reserved memory.
An error report file with more information is saved as:
/home/ubuntu/hs_err_pid9050.log
nohup: ignoring input
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006e5400000, 3670016000, 0) failed; error='Cannot allocate memory' (errno=12)

There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 3670016000 bytes for committing reserved memory.
An error report file with more information is saved as:
/home/ubuntu/hs_err_pid9116.log
nohup: ignoring input
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006e5400000, 3670016000, 0) failed; error='Cannot allocate memory' (errno=12)

There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 3670016000 bytes for committing reserved memory.
An error report file with more information is saved as:
/home/ubuntu/hs_err_pid9172.log
nohup: ignoring input
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006e5400000, 3670016000, 0) failed; error='Cannot allocate memory' (errno=12)

There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 3670016000 bytes for committing reserved memory.
An error report file with more information is saved as:
/home/ubuntu/hs_err_pid9362.log
nohup: ignoring input
OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000006e5400000, 3670016000, 0) failed; error='Cannot allocate memory' (errno=12)

There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 3670016000 bytes for committing reserved memory.
An error report file with more information is saved as:
/home/ubuntu/hs_err_pid9437.log
2019-10-17 17:14:44.651+0000 INFO Transaction with id 2689294 has been automatically rolled back due to transaction timeout.
2019-10-17 17:30:44.652+0000 INFO Transaction with id 2690165 has been automatically rolled back due to transaction timeout.
2019-10-17 17:33:14.651+0000 INFO Transaction with id 2690345 has been automatically rolled back due to transaction timeout.
2019-10-17 17:33:14.652+0000 INFO Transaction with id 2690292 has been automatically rolled back due to transaction timeout.
2019-10-17 17:33:14.652+0000 INFO Transaction with id 2690390 has been automatically rolled back due to transaction timeout.
2019-10-17 17:35:44.651+0000 INFO Transaction with id 2690502 has been automatically rolled back due to transaction timeout.

2

What's your setting for JVM heap and pagecache?

3

@stefan.armbruster
it's
memory.heap.initial_size=3500m
memory.heap.max_size=3500m
memory.pagecache.size=2000m

4

Settings do sound reasonable. Are there any other memory intensive processes running on that machine competing for memory?
Maybe reduce heap size to 2.5G leaving up more for the system.

The error message indicates that you're running short in memory on OS level.

1 Like
5

Thank you @stefan.armbruster for your reply,i'll try and update you soon.
thank you so much

1 Like
6

Нещодавно я теж задумався про те щоб підняти neo4j базу даних на сервісах amazon aws.
recently I also thought to set up now for g database in Amazon AWS cloud.

This issue caused my interest because I don't know the bottleneck if using neo4j in Amazon AWS.

let me know how this issue was fixing successfully and how are you we've been setting up neo4j in Amazon AWS cloud I want to share this experience in my blog for my mentees and for my followers.