Hello, I installed neo4j community version using helm charts on my kube cluster. I've been facing an issue where my pod restarts every day around 21:40 UTC. There is nothing in the neo4j logs -- all it shows it that it gets a request for bolt server shutdown, and then restarts back in a few seconds, so it's a gracefull shutdown and restart.
kubectl describe pod <my_pod> shows Exit Code: 137 and Reason: Error. There's nothing in the events as well.
- I've done helm releases on different times, different clusters with different storage and different chart version, and still all of them restart at around same time (mentioned above) daily.
- I've removed readiness and liveness probes. Didn't help.
- I tried increasing resources --
cpus: 8, memory: 80Gi. Didn't help. - I monitored the pod stats -- there's a sudden peak in memory around the same time, but it's still way below than the request resources.
- I checked dmesg, and couldn't find anything there as well.
- I even did an idle helm release - with 0 data and 0 requests -- this was also affected by all above problems which is very weird.
- There are other applications like Kafka, open search etc as well running on the cluster, and they don't show same issue, so I'm pretty sure it's not cluster related.
Would really appreciate help on how to debug this further and resolve this issue.