- Neo4j version: 4.2.1
- Operating system: Ubuntu 20.04
- API/Driver: (customi…zed stored procedure)
After incurring about 2G of incremental memory usage (about 4000+ runs of my customized stored procedure), the db will experience out-of-memory error despite plenty memory available. Restarting the db results in cleaning up of indices.
root@479b6f0ffe68:/var/lib/neo4j# head -100 hs_err_pid6.log
**There is insufficient memory for the Java Runtime Environment to continue.
Native memory allocation (mmap) failed to map 65536 bytes for committing reserved memory.**
\# Possible reasons:
\## The system is out of physical RAM or swap space
\# The process is running with CompressedOops enabled, and the Java Heap may be blocking the growth of the native heap
\# Possible solutions:
\# Reduce memory load on the system
\# Increase physical memory or swap space
\# Check if swap backing store is full
\# Decrease Java heap size (-Xmx/-Xms)
\# Decrease number of Java threads
\# Decrease Java thread stack sizes (-Xss)
\# Set larger code cache with -XX:ReservedCodeCacheSize=
\# This output file may be truncated or incomplete.
\#
\# Out of Memory Error (os_linux.cpp:2913), pid=6, tid=335
\#
\# JRE version: OpenJDK Runtime Environment 18.9 (11.0.9.1+1) (build 11.0.9.1+1)
\# Java VM: OpenJDK 64-Bit Server VM 18.9 (11.0.9.1+1, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
\# Core dump will be written. Default location: Core dumps may be processed with "/usr/share/apport/apport %p %s %c %d %P %E" (or dumping to /var/lib/neo4j/core.6)
\#
--------------- S U M M A R Y ------------
Command Line: -Xms31g -Xmx31g -Xlog:gc*,safepoint,age*=trace:file=/logs/gc.log::filecount=5,filesize=20m -XX:+UseG1GC -XX:-OmitStackTraceInFastThrow -XX:+AlwaysPreTouch -XX:+UnlockExperimentalVMOptions -XX:+TrustFinalNonStaticFields -XX:+DisableExplicitGC -XX:MaxInlineLevel=15 -Djdk.nio.maxCachedBufferSize=262144 -Dio.netty.tryReflectionSetAccessible=true -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -Djdk.tls.ephemeralDHKeySize=2048 -Djdk.tls.rejectClientInitiatedRenegotiation=true -XX:FlightRecorderOptions=stackdepth=256 -XX:+UnlockDiagnosticVMOptions -XX:+DebugNonSafepoints -XX:ReservedCodeCacheSize=2048m -XX:NativeMemoryTracking=detail -Dfile.encoding=UTF-8 com.neo4j.server.enterprise.EnterpriseEntryPoint --home-dir=/var/lib/neo4j --config-dir=/var/lib/neo4j/conf
Host: AMD EPYC 7571, 32 cores, 249G, Debian GNU/Linux 10 (buster)
Time: Mon Nov 30 15:08:12 2020 UTC elapsed time: 3603.235905 seconds (0d 1h 0m 3s)
--------------- T H R E A D ---------------
Current thread (0x00007f3ac27ce000): JavaThread "C2 CompilerThread0" daemon [_thread_in_vm, id=335, stack(0x00007f39e45c1000,0x00007f39e46c2000)]
Current CompileTask:
C2:3603235 29280 4 org.neo4j.kernel.impl.api.transaction.monitor.TransactionMonitor::run (20 bytes)
Stack: [0x00007f39e45c1000,0x00007f39e46c2000], sp=0x00007f39e46be5e0, free space=1013k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V [libjvm.so+0xdeec5b] VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x23b
V [libjvm.so+0xdefe75] VMError::report_and_die(Thread*, char const*, int, unsigned long, VMErrorType, char const*, __va_list_tag*)+0x45
V [libjvm.so+0x677ada] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xea
V [libjvm.so+0xbe8355] os::Linux::commit_memory_impl(char*, unsigned long, bool)+0x115
V [libjvm.so+0xbe8579] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0x29
V [libjvm.so+0xbd9f8a] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x2a
V [libjvm.so+0xde83b6] commit_expanded(char*, unsigned long, unsigned long, bool, bool)+0x26
V [libjvm.so+0xde9299] VirtualSpace::expand_by(unsigned long, bool)+0x1b9
V [libjvm.so+0x8046cd] CodeHeap::expand_by(unsigned long)+0xad
V [libjvm.so+0x5d72a1] CodeCache::allocate(int, int, int)+0xb1
V [libjvm.so+0xba8ac8] nmethod::new_nmethod(methodHandle const&, int, int, CodeOffsets*, int, DebugInformationRecorder*, Dependencies*, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, int, _jobject*, _jobject*)+0x168
V [libjvm.so+0x568b62] ciEnv::register_method(ciMethod*, int, CodeOffsets*, int, CodeBuffer*, int, OopMapSet*, ExceptionHandlerTable*, ImplicitExceptionTable*, AbstractCompiler*, bool, bool, RTMState)+0x572
V [libjvm.so+0x618c34] Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool, DirectiveSet*)+0x1424
V [libjvm.so+0x538d3c] C2Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0x14c
V [libjvm.so+0x62025f] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x3ff
V [libjvm.so+0x6220b0] CompileBroker::compiler_thread_loop()+0x650
V [libjvm.so+0xd8c671] JavaThread::thread_main_inner()+0x111
V [libjvm.so+0xd8735b] Thread::call_run()+0x6b
V [libjvm.so+0xbe6be8] thread_native_entry(Thread*)+0x128
In debug.log, the top of the trace stack looks like this, but I assume the actual failure position is arbitrary due to out-of-memory:
2020-12-03 23:54:35.044+0000 DEBUG [i.n.u.ResourceLeakDetectorFactory] Loaded default ResourceLeakDetector: io.netty.util.Resour
ceLeakDetector@65a129ea
2020-12-03 23:54:36.396+0000 DEBUG [o.n.c.i.EnterpriseCompilerFactory] [btc/1974efb1] Index not found for indexPropertyExistsSel
ectivity
org.neo4j.internal.kernel.api.exceptions.schema.IndexNotFoundKernelException: No index was found
at org.neo4j.kernel.impl.newapi.AllStoreHolder.assertValidIndex(AllStoreHolder.java:1052) ~[neo4j-kernel-4.2.1.jar:4.2.1
]
at org.neo4j.kernel.impl.newapi.AllStoreHolder.indexSize(AllStoreHolder.java:715) ~[neo4j-kernel-4.2.1.jar:4.2.1]
at org.neo4j.cypher.internal.spi.TransactionBoundGraphStatistics$BaseTransactionBoundGraphStatistics.indexPropertyExists
Selectivity(TransactionBoundGraphStatistics.scala:91) ~[neo4j-cypher-4.2.1.jar:4.2.1]
at org.neo4j.cypher.internal.planner.spi.MinimumGraphStatistics.indexPropertyExistsSelectivity(MinimumGraphStatistics.sc
ala:57) ~[neo4j-cypher-planner-spi-4.2.1.jar:4.2.1]
at org.neo4j.cypher.internal.planner.spi.InstrumentedGraphStatistics.indexPropertyExistsSelectivity(InstrumentedGraphSta
tistics.scala:98) ~[neo4j-cypher-planner-spi-4.2.1.jar:4.2.1]
at org.neo4j.cypher.internal.compiler.planner.logical.cardinality.ExpressionSelectivityCalculator.$anonfun$calculateSele
ctivityForValueRangeSeekable$2(ExpressionSelectivityCalculator.scala:242) ~[neo4j-cypher-planner-4.2.1.jar:4.2.1]
at scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:245) ~[scala-library-2.12.10.jar:?]
: