Out of memory when invoking OGM session loadAll

Drivers & Stacks Spring Data Neo4j & Neo4j-OGM
1

Hi,

A java Neo4J app that has been working perfectly ok for months is suddenly failing with an unexpected exception Java heap space java.lang.OutOfMemoryError. The exception occurs when restoring Java POJOs via the OGM Session loadAll method.

public class CheckEntitiesFactory implements PersistCheckEntities {

private static final int DEPTH_ALL = -1;
private static final int DEPTH_SINGLE_ENTITY = 1;
    ...
private final Session mySession;
private final Class<T> mySupportedType;
    ...
public Iterable<T> findAll() 
{
	if (useable)
	{
		return mySession.loadAll(getEntityType(), DEPTH_ALL);			
	}
	else
	{
		return null;
	}
}
    .....
private Class<T> getEntityType()
{
	return mySupportedType;
}
    ...

}

I have run ne04j-admin memrec and implemented the recommendations. Had no effect.

this could not have happened at a worse time as i am meant to be demo-ing the app this week!.

any suggestions would be really welcome

PS to give a little more context : this problem is occuring when i am trying to load 10 nodes and 5 relationships.
Previously (when the app was happily working) the following debug output would occur ...

INFO: Direct driver instance 802243390 created for server address localhost:7687
23:59:42.228 [main] DEBUG org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction - No current transaction, starting a new one
23:59:42.243 [main] DEBUG org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction - Native transaction: org.neo4j.driver.internal.InternalTransaction@31fa1761
23:59:42.310 [main] DEBUG org.neo4j.ogm.drivers.bolt.request.BoltRequest - Request: MATCH (n:CheckEntity) WITH n MATCH p=(n)-[*0..]-(m) RETURN p with params {}
23:59:42.844 [main] DEBUG org.neo4j.ogm.metadata.MetaData - looking for concrete class to resolve label: CheckEntity
23:59:42.844 [main] DEBUG org.neo4j.ogm.metadata.MetaData - label is on an abstract class. Looking for a single concrete subclass...
23:59:42.845 [main] DEBUG org.neo4j.ogm.metadata.MetaData - More than one class subclasses ClassInfo{className='checkEntities.entities.CheckEntity', neo4jName='CheckEntity'}
23:59:42.845 [main] DEBUG org.neo4j.ogm.metadata.MetaData - no implementing class or concrete subclass found!
23:59:42.845 [main] DEBUG org.neo4j.ogm.metadata.MetaData - looking for concrete class to resolve label: CheckGroup
23:59:42.845 [main] DEBUG org.neo4j.ogm.metadata.MetaData - concrete class found: ClassInfo{className='checkEntities.entities.CheckGroup', neo4jName='CheckGroup'}. comparing with what's already been found previously...
23:59:42.845 [main] DEBUG org.neo4j.ogm.metadata.MetaData - CheckGroup resolving class: ClassInfo{className='checkEntities.entities.CheckGroup', neo4jName='CheckGroup'}
23:59:42.859 [main] DEBUG org.neo4j.ogm.metadata.MetaData - looking for concrete class to resolve label: Check
23:59:42.859 [main] DEBUG org.neo4j.ogm.metadata.MetaData - concrete class found: ClassInfo{className='checkEntities.entities.Check', neo4jName='Check'}. comparing with what's already been found previously...
23:59:42.859 [main] DEBUG org.neo4j.ogm.metadata.MetaData - Check resolving class: ClassInfo{className='checkEntities.entities.Check', neo4jName='Check'}
23:59:42.859 [main] DEBUG org.neo4j.ogm.metadata.MetaData - looking for concrete class to resolve label: CheckEntity
23:59:42.859 [main] DEBUG org.neo4j.ogm.metadata.MetaData - label is on an abstract class. Looking for a single concrete subclass...
23:59:42.859 [main] DEBUG org.neo4j.ogm.metadata.MetaData - More than one class subclasses ClassInfo{className='checkEntities.entities.CheckEntity', neo4jName='CheckEntity'}
23:59:42.859 [main] DEBUG org.neo4j.ogm.metadata.MetaData - no implementing class or concrete subclass found!
23:59:42.859 [main] DEBUG org.neo4j.ogm.metadata.MetaData - looking for concrete class to resolve label: CheckParent
etc, etc

Now all that is output is...
INFO: Direct driver instance 701166114 created for server address localhost:7687
14:10:00.386 [main] DEBUG org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction - No current transaction, starting a new one
14:10:00.416 [main] DEBUG org.neo4j.ogm.drivers.bolt.transaction.BoltTransaction - Native transaction: org.neo4j.driver.internal.InternalTransaction@6548bb7d
14:10:00.579 [main] DEBUG org.neo4j.ogm.drivers.bolt.request.BoltRequest - Request: MATCH (n:CheckEntity) WITH n MATCH p=(n)-[*0..]-(m) RETURN p with params {}
After approximately 20 minutes with no further debug o/p the out of memory exception occurs

cheers

R

2

I have identified the cause of the out of Memory issue when invoking the loadAll method on an OGM Session object.

But the cause raises further questions that I would be interested to hear the thoughts of others.

In a nutshell I only want to use the OGM layer for a portion of the graph.

My graph can be thought of as having 2 major areas sparsely interconnected. Diagrammatically one could imagine it as follows
OGM managed node Non OGM managed node

image
image635×626 30 KB

As long as no ‘red’ relationships exist the OGM restore (via loadAll) works perfectly ok. As soon as there is an instance of the ‘red’ relationship within the graph the out of memory exception occurs.

For performance, complexity and security reasons I am loathe to bring all of the nodes under OGM control.

Using @Transient and a Dummy version of the ‘Orange’ root label declared to OGM I can possibly fool the OGM but it is not elegant and could have unintended side effect.

Maybe others have alternative strategies – I would be interested to hear.

3

Great analysis that nails the point. Given that you define an unlimited depth for the loadAll Neo4j-OGM falls back to the path querying instead of using its knowledge about the domain model (here esp. the relationships) to create the query.
What happens then is that not only the whole reachable graph gets fetched but also that Neo4j-OGM tries to eagerly map anything that gets returned.
I have no solution for this conceptual problem with the mixture of "I want to load everything" and the amount of data the application will suddenly have to process.